Development in the Cloud

I’ve recently made an effort to stop using local virtual machines. This has not been by choice, but rather because OS X has become extremely unstable as of late with VirtualBox and seems to show similar behavior with VMWare. Rather than trying to find a version of VirtualBox that is more stable, I’m making an effort to develop on cloud servers instead.

First off, to aid in the transition, I’ve started using Emacs in a terminal exclusively. While I miss some aspects of GUI Emacs, such as viewing PDFs and images, it generally hasn’t been a huge change. I’ve had to do some fiddling as well with my $TERM in order to make sure Emacs picks up a value that provides a readable color setting.

Another thing I started doing was getting more familiar with byobu and tmux. As Emacs does most of my window management for me, my use is relatively limited. That said, it is nice to keep my actually terminal (iTerm2) tabs to a minimum and use consistent key bindings. It also makes keeping an IRC bouncer less of a requirement because my client is up all the time.

The one thing I haven’t done yet is to provision a new dev machine automatically. The dev machine I’m on now has been updated manually. I started using a Vagrantfile to configure a local VM that would get everything configured, but as is clear by my opening paragraph, frequent crashes made that less than ideal. I’m hoping to try and containerize some processes I run in order to make a Vagrantfile that can spin up a cloud server reasonably simple.

What makes all this possible is Emacs. It runs well in a terminal and makes switching between local and remote development reasonably painless. The biggest pain is the integrations with my desktop, aka my local web browser. When developing locally, I can automatically open links with key bindings. While I’m sure I could figure something out with iTerm2 to make this happen, I’m going to avoid wasting my time and just click the link.

If you don’t use Emacs, I can’t recommend tmux enough for “window” management. I can see how someone using vim could become very proficient with minimal setup.

Modern Development

My MacbookPro crashed with a gray screen 4 times yesterday and it gave me time to think about what sort of environment to expect when developing.

The first thing is that you can forget about developing locally. Running OS X or Windows means your operating system is nothing more than an inconvenient integration point that lets you use Office and video conferencing software. Even if you use Linux, you’ll still have some level of indirection as you separate your dev environment from your OS. At the very least, it will be language specific like virtualenv in Python. At most you’ll be running VirtualBox / Vagrant with Docker falling somewhere in between.

Seeing as you can’t really develop locally, that means you probably don’t have decent integration into an IDE. While I can already hear the Java folks about to tell me about remote debugging, let me define “decent”. Decent integration into and IDE means running tests and code quickly. So, even if you can step through remote code, it is going to be slow. The same goes for developing for iOS or Android. You have a VM in the mix and it is going to be slow. When developing server software, you’re probably running a Vagrant instance and sharing a folder. Again, this gets slow and you break most slick debugging bits your editor / IDE might have provided.

So, when given the choice, I imagine most developers choose speed over integration. You can generally get something “good enough” working with a terminal and maybe some grep to iterate quickly on code. That means you work in a shell or deal with copying code over to some machine. In both cases, it’s kludgey to say the least.

For example, in my case, I’ve started ssh’ing into a server and working there in Emacs. Fortunately, Emacs is reasonably feature-full in a terminal. That said, there are still integration issues. The key bindings I’ve used to integrate with non-code have been lost. Copy and paste becomes tricky when you have Emacs or tmux open on a server with split screens. Hopefully, your terminal is reasonably smart where it can help finding links and passing through mouse events, but that can be a painful process to configure.

OK. I’m venting. It could be worse.

That said, I don’t see a major shift anytime soon. I’ve gone ahead and tried to change my expectations. I’ll need to shell into servers to develop code. It is important to do a better job learning bash and developing a decent shell work flow. Configuring tmux / screen / byobu is a good investment. Part of me can appreciate the lean and mean text interface 24/7, but at the same time, I do hope that we eventually will expect a richer environment for development than a terminal.

Ansible and Version Control

I’ve become reasonbly comfortable with both Chef and Ansible. Both have pros and cons, but there is one aspect of Ansible that I think deserves mention is how it can work with version control thanks to its lack of a central server and through defining its operations via YAML.

No Central Server

In Chef, there is the chef server that keeps the actual scripts for different roles, recipes, etc. It also maintains the list of nodes / clients and environments available. The good thing about this design is that you have a single source of truth for all aspects of the process. The downside, is that the central server must be updated outside of version control. This presents the situation where version 1.1 of some recipe introduces some bug and you may need to cut a 1.2 that is the same as 1.0 in order to fix it.

Another downside is that if a node goes down or doesn’t get cleaned up properly, it will still exist on the chef server. Other recipes may still think the node is up even though it has become unavailable.

Ansible, at its core, runs commands via SSH. The management of nodes happens in the inventory and is dependent on file listing or a dynamic module. The result is that everything Ansible needs to work is the local machine. While it is not automatic, using a dynamic inventory, Ansible can examine the infrastructure at run time and act accordingly.

If you are not using a dynamic inventory, you can add hosts in your invetory files and just commit them like any other change! From here you can see when nodes come up and go down in your change history.

YAML Roles

Ansible defines its actions via playbooks defined as YAML. You can also add your own modules if need be in the same repo. What’s more, if you find a helpful role or library in Ansible Galaxy, installing the library downloads its file directly into your code tree, ready to be committed. This vendoring makes things like version pins unnecessary. Instead, you simply checkout the changeset or version tag and everything should be good to go.

To compare this with Chef, you can maintain roles, environments, etc. as JSON and sync them with the central Chef server using Kitchen. The problem with this tactic is that a new commit in version control may or may not be an update to the resource on the chef server. You can get around this limitation with things like commit hooks that automatically sync the repo with the chef server, but that is not always feasible. For example, if you mistakenly update a role with an incorrect version pin and your servers are updating on a cadence, then that change will get rolled out automatically.

Again, there are known ways around these sorts of issues, but the point being is that it is harder to maintain everything via version control, which I’d argue is beneficial.

I’m a firm believer that keeping your config management code in version control is the best way to manange your systems. There are ways to make central server based systems effectively gated via version control, but it is not always obvious, and it is certainly not built in. Ansible, by running code from the local machine and repository code, makes it simple to keep the repository as the single source of truth.

More Virtual Machine Development

I’ve written before developing on OS X using virtual machines. That was ~6 months ago and it seemed time for an update.

Docker is Better

One thing that has improved is Docker. Docker Compose now works on OS X. Docker Machine also has become more stable, which allows a boot2docker like experience on any “remote” VM where “remote” includes things like cloud servers as well as Vagrant VMs.

In terms of development, Docker Compose seems really nice in theory. You can define a suite of services that get started and linked together without having to manually switch configs all over the place. Eventually, Docker Compose will also be supported on things like Docker Swarm, which means, in theory at least, that you could deploy an entire stack.

In practice though, the linking is not as robust as one would like. If a container doesn’t start, the link doesn’t get added to the host machine’s /etc/hosts file. It’s also unclear how portable this is across different distros. While an /etc/hosts file is pretty standard, it still must be used by the OS using whatever resolution system is in place. I’ve gotten the impression that things are moving towards injecting DNS in the mix in order to avoid lower level changes, but time will tell.

While it hasn’t worked for me, I’m still planning to keep trying docker compose as it seems like the best solution to starting up a suite of microservices for development.

Docker and Emacs !?!

Emacs has a package called Prodigy that lets you manage processes. I also played around with running my dev environment in a docker container where I just used Prodigy to run all the services I need as subprocesses of the Emacs process. It is like a poorman’s emacs init system. Alas, this wasn’t much better than working on a cloud server. While it is clever and portable, it still is like ssh’ing into a box to work, which get frustrating over time.

Vagrant and rdo

A while back I released a tool called rdo that lets you remotely execute commands like they were local. This has become my primary entry point into using Vagrant and avoiding having to use a terminal to access my dev environment. I also integrated this into my xe tool to make things more cohesive, but overall, it is easier to just use rdo.

Even with a reasonably efficient way to make VM development feel local, it’s still a little painful. The pain comes in the slowness of always working through the VM. Vagrant shares (by way of VirtualBox shared folders) are slow. I’m looking at using NFS instead, which I understand should be faster, so we’ll see. The fastest method is to use rsync. This makes sense because it basically copies everything over to the VM, making things run as “native” speeds. My understanding is that this is a one way operation, so that doesn’t work well if you want to run a command and have the output piped to a file so you can use it locally (ie dosomething | grep “foo” > result_to_email.txt).

Cloud Server

I also spent some time developing on a cloud server. Basically, I set up an Ubuntu box in our cloud and ssh’d in to work. From a purely development standpoint, this worked pretty well. I used byobu to keep my emacs and shell sessions around after logging out or getting disconnected. Everything worked really fast as far as running tests, starting processes and even IRC. The downside was the integration into my desktop.

When I work in a local Emacs, I have key bindings and tools that help me work. I use Emacs for IRC, therefore, I have an easy key binding to open URLs. This doesn’t work on a cloud server. What’s more, because I’m using Emacs, any fancy URL recognition iTerm2 often gets screwed up where I can’t click it, and in some cases, I can’t even copy it. Another thing that was problematic was that since the cloud server used a hypervisor, I couldn’t run VirtualBox. This meant no Chef Kitchen, so I was forced to head back to OS X for doing any devops work. Due to firewall restrictions, I also couldn’t access my work email on my cloud server.

Lastly, since the machine was on a public cloud, running any process was usually available to the public! In theory, these are just dev processes that don’t matter. In practice though, spinning up a jenkins instance to test jobs in our real environment, was dangerous to run openly.

The result was that I had to make a rather hard switch between pure dev work and devops work when using a cloud server. I also had to be very aware of the security implications of using my cloud server to access important networks.

Sidenote: Emacs in the Terminal

Emacs through a terminal, while very usable, has some nits related to key bindings. There are far fewer key bindings available in a terminal due to the shell having to provide actual characters to the process. While I tried to work around these nits, the result was more complex key bindings that I had no hopes of establishing as habit. One example is C-= (C == ctrl). In reStructuredText mode, hitting C-= will allow cycling through headings and automatically add the right text decoration. Outside rst-mode, I typically have C-= binded to expand region. Expand region lets me hit C-= repeated to semantically select more of the current section. For example, I could expand to a word, sentence, paragraph, section, and then document. While these seem like trivial issues, they are frustrating to overcome as they have become ingrained in how I work. This frustration is made more bitter by the fact that the hoops I’ve jumped through are simply due to using OS X rather than Linux for development.

Future Plans

I’m not going to give up just yet and install Linux on my Macbook Pro. I’m doing some work with rdo to allow using docker in addition to Vagrant. I’m also confident that the docker compose and network issues will be solved relatively soon. In the meantime, I’m looking at ways to make rdo faster by reusing SSH connections and seeing if there are any ways to speed up shared folders in Vagrant (NFS, Rsync).

Config Files

In Chef, you can use attributes to set up values that you can later use in your recipes. I suspect a majority of these attributes end up in config files. For example, at work, we have added pretty much every value necessary in our config files. The result is that we duplicate our configuration (more or less) as a Ruby hash that gets serialized using ERB templates into the necessary config, which, in this case, is a conf file format. The other thing that happens is that we also set many of these values via environment variables using withenv, which describes this data as YAML.

Essentially, we go from YAML to Ruby to a template to a config file in a known parsable format. The problem is that each time you transition between data formats there is a chance for mistakes. As we all know, humans can make mistakes writing config files. It is worth considering how we could improve the situation.

I imagine a tool that accepts YAML and spits out different config file formats. YAML seems like a good option because it is arguably ubiquitous and provides data structures that programmers like. The tool to spit out a config file would use consistent patterns to output formats like conf files and plugins would need to be written for common tools like databases, web servers, queues, etc.

For example:

$ ymlconf --format rsyslog rsyslog_conf.yml > /etc/rsyslog.conf

I’m sure there would be some weirdness for some apps archaic and silly configuration file formats, but I’d hope that 1) the people using these apps understand the archaic file format well enough that 2) translating it to a YAML format wouldn’t be that terrible. For apps that do understand simple conf files, or even better, YAML or JSON or environment variables, things can be a matter of simply writing the files.

What’s more, YAML is resonably programmatic. If you have a list of nodes that need to repeat the same sections over a list of IPs you get when you start up cloud servers, it is trivial to do in a chef recipe. Rather than adding it to a data structure, only to decompose and pass that data to a template, you just append them to a list in a data structure read from YAML.

After using withenv, I think this is another way to greatly reduce the cognitive mapping that is required to constantly go from some driving data (like environment variables) to configuration management system data structures (chef attributes) that are passed to a template languages in order to write config files. Instead it would simply be a matter of running some command and pass it the path or YAML as stdin and be done with it.

Getting Paid

Cory wrote up some thoughts on funding OSS that was inspired by the recent decision to stop feature development on hypothesis. There is a problem in the FOSS community where developers get frustrated that the time they invest has a very low return. While I understand this feeling, it is also worthwhile to consider some benefits to investing in open source software that makes it worth your time.

I think most FOSS developers would agree the benefit of working on FOSS is recognition. It feels good to know others are using your software. Beyond simply being recognized, FOSS also are often given respect from other programmers and technical community at large. It is this respect that can be used to

So, what do you get, right now, from working on (successful) Open Source software?

While many developers would like to get money, the reality is you get something less tangible, a reputation as a good developer. You can use your reputation as an open source leader to negotiate for the things you want. Often you can work from home, negotiate for more vacation time, better pay and time to work on your project, all thanks to your reputation. Not to mention, you often can choose to work for well respected organizations doing work you find reasonably interesting.

Companies should support FOSS developers for the code they have graciously offered for free. At the same time, we as developers should realize that it is our responsibility to capitalize on our contributions, even when they may not be directly justifiable to a business. If a company hired a well known Emacs (or Vim!) developer, even though the company may uses Python, the company may still be able to offer time to work on the FOSS code, more money and/or sponsoring going to conferences. These types of expenses are easy to account for on a balance sheet when compared to giving someone money for non-specific work on a project.

Hopefully, in the future new methods of directly supporting FOSS developers come to light, but in the meantime, lets see what we can do today. Ask your manager about having X number of hours to work on your open source project. Request sponsorship to a conference. If they refuse, look for another job and include your project work as part of the package. A new opportunity is a great means of letting your employer know your skills are valuable and your terms for staying include working on your FOSS work.

For companies, support developers to work on FOSS! Even if someone doesn’t work on something directly associated with your current code base or business, that person has proven themselves as a good developer, with the real world experience being available in the open. Similarly, if your organization is strugging to keep or acquire good talent, offering someone 4-8 hours a week to work on a project they already contribute to is a relatively cheap benefit in order to hire someone that is a proven successful developer. What’s more, that person is likely to have a network of other great devs that you can dip into.

Again, I understand why developers are frustrated that they spend time on FOSS with seemingly very little to gain. But, rather than sit by and wait for someone to offer to pay you money for your time, communicate your frustrations to your employer and try to use some of your reputation to get what you want. If your current employer refuses to listen, it is probably time to consider other options. Companies that having difficulties attracting or keping talent should offer FOSS support as part of the benefits package.

Finally, for some developers, it is a good idea to take a step back and consider why you write software. As a musician, it is cliché to say I don’t do it for the money. The reason being is that the music industry is notorious for not paying anything with a long line of willing musicians to work for nothing. While we as software developers do make a good living writing software, there is something to be said for enjoying our independent time writing code. Taking a hobby you enjoy and turning into a business often removes much of what makes that hobby fun. It no longer is yours to do with as you want. If you write FOSS code for fun, then you are under no obligations, other than your own desires. Programming is fun, so regardless of whether 1 or 1 million people use your code, recognize why you started writing the code in the first place.

Heat vs. Ansible

At work we’re using Ansible to start up servers and configure them to run Chef. While I’m sure we could do everything with Ansible or Chef, our goal is to use the right tool for each specific job. Ansible does a pretty decent job starting up infrastructure, while Chef works better maintaining infrastructure once it has been created.

With that in mind, as we’ve developed our Ansible plays and created some tooling, there is a sense that Heat could be a good option to do the same things we do with Ansible.

Before getting too involved comparing these two tools, lets set the scope. The goal is to spin up some infrastructure. It is easy enough in either tool to run some commands to setup chef or what have you. The harder part is how to spin everything up in such a way that you can confirm everything is truly “up” and configured correctly. For example, say you wanted to spin up a load balancer, some application nodes, and some database nodes. You need to be sure when you get a message that everything is “done” that:

  1. The load balancer is accepting traffic with the correct DNS hostname.
  2. There are X number of app server nodes that all can be accessed by ssh and any other ports services might be running on.
  3. There are X number database nodes that are accessed via the proxy using a private network.

I’m not going to provide examples on how to spin this infrastructure up in each tool, but rather discuss what each tool does well and not so well.


Anisble, for the most part, connects to servers via ssh and runs commands. It has a bunch of handy modules for doing this, but in a nutshell, that is what Ansible does. Since all Ansible really does is run commands on a host, it might not be clear how you’d automate your infrastructure. The key is the inventory.

Ansible uses a concept of an inventory that contains the set of hosts that exist. This inventory can be dynamic as well such that “plays” create and add hosts to the inventory. A commone pattern we use is to create some set of hosts and pass that list on to subsequent plays that do other tasks like configure chef.

What’s Good

The nice aspect of Ansible is that you have an extremely granular process of starting up hosts. You can run checks within the plays to ensure that nothing continues if there is a failure. You can “rescue” failed tasks as well in order to clean up broken resources. It is also really simple to understand what is happening. You know Ansible is simply running some code on a host (sometime localhost!) via ssh. This makes it easy to reproduce and perform the same tasks manually.

What’s Bad

The difficulty in using Ansible is that you are responsible for everything. There is no free lunch and the definition of what you want your infrastructure to look like is completely up to you. This is OK when you’re talking about a few servers that all look the same. But, when you’d need 50 small 512 mem machines along with 10 big compute machines using some shared block storage, 10 memcache nodes with tons of ram, a load balancer and ensure this infrastructure runs in 3 different data centers, then it starts to hurt. While there is a dynamic inventory to maintain your infrastructure, it is not well integrated as a concept in Ansible. The process often involves using a template language in YAML to correctly access your infrastructure, which is less than ideal.


I’m sure ansible gurus have answers to my complaints. No doubt, using tower could be one. Unfortunately, I haven’t had the opportunity to use tower and since it isn’t free, we haven’t considered it for our relatively limited use case.


Heat comes from Cloud Formation Templates from AWS. The idea is to define what you’d like your infrastructure to look like and pass that definition to your orchestration system. The orchestration system will take the template, establish a plan of attack and start performing the operations necessary. The end result is that everything gets created and linked together as requested and you’re done!

At Rackspace, we have a product called Cloud Orchestration that is responsible for making your template a reality.

What’s Good

Heat lets you define a template that outlines precisely what you want your infrastructure to look like. Just to provide a simple example, here is a template I wrote to spin up a Fleet cluster.

heat_template_version: '2015-04-30'
description: "This is a Heat template to deploy Linux servers running fleet and etcd"

    type: 'OS::Heat::ResourceGroup'
      count: 3
        type: 'OS::Nova::Server'
          flavor: '512MB Standard Instance'
          image: 'CoreOS (Stable)'
          config_drive: true
          user_data: |
                  initial-advertise-peer-urls: http://$private_ipv4:2380
                  advertise-client-urls: http://$public_ipv4:2379
                  listen-peer-urls: http://$private_ipv4:2380,http://$private_ipv4:7001
                  public-ip: $private_ipv4

                  - name: etcd2.service
                    command: start

                  - name: fleet.service
                    command: start

    value: { get_attr: [fleet_servers, accessIPv4] }

Heat templates allow a bunch of features to make this more programmable such that you pass in arguments where necessary. For example, I might make count a parameter in order to spin up 1 server when testing and more in production.

What we do currently in Ansible is to pass environment variables to our plays that end up as configuration options for creating our dynamic inventory. We use the withenv to make this more declarative by writing this in YAML. Here is an example:

  - MDNS:          1
  - POOL_MGR:      1
  - CENTRAL:       1
  - API:           1
  - DB:            3
  - QUEUE:         3

As you can see, the process defining this sort of infrastructure is slowly becoming closer to Heat templates.

Another benefit of using Heat is that you are not responsible for implementing every single step of the process. Heat provides semantics for naming a group of servers in such a way that they can be reused. If you create 5 hosts for some pool that need to be added to a load balancer, that is easy peasy with Heat. What’s more, the orchestration system can act with a deeper knowledge of the underlying system. It can perform retries as needed with no manual intervention.

Heat also provides makes it easy to use cloud-init. While this doesn’t provide the same flexibility as an Ansible play, it is an easy way to get a node configured after it boots.

What’s Bad

Heat templates are still just templates. The result is that if you are trying to do complex tasks, get ready to write a bunch of YAML that is not easy to look at. Heat also doesn’t provide a ton of granularity. If one step fails, where failure is defined by the orchestration system and the heat template, the entire stack must be thrown away.

Heat is really meant to spin up or teardown a stack. If you have a stack that has 5 servers and you want to add 5 more, updating that stack with your template will teardown the entire stack and rebuild it from scratch.


I’m thankfully wrong here! Heat should recognize that you are only adding/removing servers and assuming you aren’t changing other details, it will just add or remove the machines. The one caveat here is if there are dependencies on that server. I’m not clear on what a “dependency” in this case means, but for my basic use case of adding more nodes in the typical case (ie more REST API nodes) should work just fine.

Conclusions and Closing Thoughts

Heat, currently, is a great tool to spin up and tear down a complex stack. While it seems frustrating that updates do not consider the state of the stack, it does promote a better deployment design where infrastructure is an orthogonal concern to how apps are actually run.

Also, Heat at Rackspace, supports autoscaling, which handles the most common use case of adding / removing nodes from a cluster.

From the user perspective, decoupling your infrastructure from your application deployments works well when you run containers and use a tool like Fleet to automatically start your app on the available hosts in a cluster. When a host goes away, Fleet is responsible for running the lost processes on the nodes still available in the cluster.

With that in mind, if your applications haven’t been developed to run on containers and that isn’t part of your CI/CD pipeline, Ansible is a great option. Ansible is simple to understand and has a vibrant ecosystem. There are definitely mismatches when it comes to infrastructure, but nothing is ever perfect. For example, I do think the dynamic inventory ends up a little bit cleaner than the machine semantics I’ve seen in chef.

Finally, there is no reason you can’t use both! In my Heat template example, you notice that there is an outputs section. That can be used to create your own dynamic inventory system so you could get the benefits of setup/teardown with Heat, while doing your initial machine configuration with Ansible rather than fitting it all into a cloud-init script.

I hope this overview helps. Both Heat and Ansible are excellent tools for managing infrastructure. The big thing to rememeber is that there no free lunch when it comes to spinning up infrastructure. It is a good idea to consider it as separate process from managing software. For example, it is tempting to try and install and configure your app via a cloud-init script or immediately after spinning up a node in ansible. Step one should be to get your infrastructure up and tested before moving on to configuring software. By keeping the concerns separate, you’ll find the tools like, heat and ansible, become more reliable while staying simple.

Vendoring Dependencies

All software has to deal with dependencies. It is a fact of life. No program can execute without some supporting tools. These tools, more often than not, are made available by some sort of dependency management system.

There are many paths to including dependencies in your software. A popular use case, especially in uncompiled languages like Python or Ruby, is to resolve dependencies when installing the program. In Python, pip install will look at the dependencies a program or library needs, download and install them into the environment.

The best argument for resolving dependencies during install (or runtime) is that you can automatically apply things like security fixes to dependent libraries across all applications. For example, if there is a libssl issue, you can install the new libssl, restart your processes and the new processes should automatically be using the newly installed library.

The biggest problem with this pattern is that it’s easy to have version conflicts. For example, if a software declares it needs version 1.2 of a dependency and some other library requires 1.1, the dependencies are in conflict. These conflicts are resolved by establishing some sort of sandboxed environment where each application can use the necessary dependencies in isolation. Unfortunately, by creating a sandboxed environment, you often disconnect the ability for a package to inherit system wide libraries!

The better solution is to vendor dependencies with your program. By packaging the necessary libraries you eliminate the potential for conflicts. The negative is that you also eliminate the potential for automatically inheriting some library fixes, but in reality, this relationship is not black and white.

To make this more concrete lets look at an example. Say we have a package called “supersecret” that can sign and decrypt messages. It uses libcrypto for doing the complicated work in C. Our “supersecret” package installs a command line utility ss that uses the click library. The current version of click is 4.x but we wrote this when 3.x was released. Lets assume as well that we use some feature that breaks our code if we’re using 4.x.

We’ll install it with pip

$ pip install supersecret

When this gets installed, it will use the system level shared libcryto library. But, we’ve vendored our click dependency.

The benefit here is that we’ve eliminated the opportunity to conflict with other packages, while still inheriting beneficial updates from the lower level system.

The arguments against this pattern is that keeping these dependencies up to date can be difficult. I’d argue that this is incorrect when you consider automated testing via a continuous integration server. For example, if we simply have click in our dependency list via or requirements.txt we can assume our test suite will be run from scratch, downloading the latest version and revealing broken dependencies. While this requires tests that cover your library usage, that is a good practice regardless.

To see a good example of how this vendoring pattern works in practice, take a look at the Go <> language. Go has explicitly made the decision to push dependency resolution to happen at build time. The result is that go binaries can be copied to a machine and run without any other requirements.

One thing that would make vendoring even safer is a standard means of providing information about what libraries are versioned. For example, if you do use libssl, having a way to communicate that dependency is vendored would allow an operator recognize what applications may need to be updated when certain issues arise. That in mind, as we’ve seen above, many critical components in languages such as Python or Ruby make it trivial utilize the system level dependencies that typically are considered when discussions arise regarding rotted code due to vendoring.

Vendoring is far from a panacea, but it does put the onus on the software author to take responsibility for dependencies. It also promotes working software over purity from the user’s perspective. Even if you are releasing services where you are the operator, managing your dependencies when you are working on the code will greatly simplify the rest of the build/release process.

Small Functions without an IDE

I’ve been reading Clean Code for a book club at work. So far, it is really a great book as it provides attributes and techniques for understanding what clean code really looks like. My only complaint, which is probably the wrong word, is that the suggestions are based on using a more verbose language such as Java that you use with an IDE.

As I said, this really isn’t a complaint so much as the author mentions how things like renaming things and creating very small functions are painless thanks to the functionality of the IDE. In dynamically typed languages like Python, the same level of introspection doesn’t generally exist for tooling available.

As an aside, function calls in Python can be expensive, so extracting a single function into many, much smaller functions does have the potential to slow things down. I saw this impact on a data export tool that needed to perform a suite of operations on each role. It had started as one huge function and I refactored it into a class with a ton of smaller methods. Unfortunately, this did slow things down somewhat, but considering the domain and expense of maintaining the single, huge function, the slowdown was worth it.

Performance aside, I’d argue that it is definitely better to try to keep functions small and use more when writing any code. The question then, is how do you manage the code base when you can’t reliably jump to function references automatically?

I certainly don’t have all the answers, but here are some things I’ve noticed that seem to help.

Use a Single File

While your editor might support refactoring tools, it most certainly has the ability to search. When you need to pop around to different functions, keep the number of files to a minimum so you can easily use search to your advantage.

Use a Flat Namespace

Using a flat namespace goes hand in hand with keeping more functions / methods in a single file. Avoid nesting your modules to make it faster to find files within the code. One thing to note is that the goal here is no to keep a single folder with hundreds of files. The goal is to limit the scope of each folder / module to include the code it will be using.

You can think of this in the same terms as refactoring your classes. If a file has functionality that seems out of place in the module, move it somewhere else. One benefit of using a dynamic language like Python is you don’t have the same one class per file requirements you see in something like Java.

Consistent Naming

Consistent naming is pretty obvious, but it is even more important in a dynamic language. Make absolutely sure you name the same usage of the same objects / classes throughout your code base. If you are diligent in how you name your variables, search and replace can be extremely effective in refactoring.

Write Tests

Another obvious one here, but make sure you write tests. Smaller functions means more functions. More functions should mean more tests. Fortunately, writing the tests are much, much easier.

class TestFoo(object):

    def test_foo_default(self): ...
    def test_foo_bar(self): ...
    def test_foo_bar_and_baz(self): ...
    def test_foo_bar_and_baz_and_kw(self): ...

If you’ve ever written a class like this, adding more functions should make your tests easier to understand as well. A common pattern is to write a test for each path a function can take. You often end up with a class that has tons of oddly named test functions with different variables mocked in order to test the different code paths in isolation. When a function gets refactored into many small functions (or methods) you see something more like this:

class TestFoo(object):

    def setup(self): = Foo()

    def test_foo(self): = Mock()


    def test_bar(self): = Mock()'/path/to/cfg')

In the above example, you can easily mock the functions that should be called and assert the interface the function requires is being met. Since the functions are small, you’re tests end up being easy to isolate and you can test the typically very small bit of functionality that needs to happan in that function.

Use Good Tools

When I first started to program and found out about find ... | xargs grep my life was forever changed. Hopefully your editor supports some integration of search tools. For example, I use Emacs along with projectile, which supports searching with ag. When I use these tools in my editor along side the massive amount of functionality my editor provides, it is a very powerful environment. If you write code in a dynamic language, it is extremely important to take some time to master the tools available.


I’m sure there are other best practices that help to manage well factored code in dynamic languages. I’ve heard some programmers that feel refactoring code to very small functions is “overkill” in a language like Python, but I’d argue these people are wrong. The cost associated with navigating the code base can be reduced a great deal using good tools and some simple best practices. The benefits of a clean, well tested code base far out weigh the cost of a developer reading the code.


Someone mentioned to me a little while back a disinterest in going to PyCon because it felt directed towards operators more than programmers. Basically, there have become more talks about integrations using Python than discussions regarding language features, libraries or development techniques. I think this trend is natural because Python has proven itself as a main stream language that has solved many common programming problems. Therefore, when people talk about it, it is a matter of how Python was used rather than describing how to apply some programming technique using the language.

With that in mind, it got me thinking about “Operators” and what that means.

Where I work there are two types of operators. The first is the somewhat traditional system administrator. This role is focused on knowledge about the particular system being administered. There is still a good deal of automation work that happens at this level, but it is typically focused on administering a particular suite of applications. For example, managing apache httpd or bind9 via config files and rolling out updates using the specific package manager. There is typically more nuance to this sort of role than can be expressed in a paragraph, so needless to say, these are domain experts that understand the common and extreme corner cases for the applications and systems they administer.

The second type of operator is closer to the operations included devops. These operators are responsible for building the systems that run application software. These folks are responsible for designing the systems and infrastructure to run the custom applications. While more traditional sysadmins use configuration management, these operators master it. Ops must have a huge breadth of knowledge that spans everything. File systems, networking, databases, services, *nix, shell, version control and everything in between are all topics that Ops are familiar with.

As a software developer, we think about abstract designs, while ops makes the abstract concrete.

After working with Ops for a while, I have a huge amount of respect due to the complexity that must be managed. There is no way to simply import cloud and cloud.start(). The tools available to Ops for enacapsulating concepts is rudimentary by necessity. The configuration management tools are still very new and the terminology hasn’t coalesced towards design patterns due to the fact that everyone’s starting point is different. Ops is where linux distros, databases, load balancers, firewalls, user management and apps come together to actually have working products.

It is this complexity that makes DevOps such an interesting place for software development. Amidst the myriad of programs and systems, there needs to be established concepts that can be reused as best practices, and eventually, as programs. Just as C revolutionized programming by allowing a way to build for different architectures, DevOps is creating the language, frameworks, and concepts to deploy large scale systems.

The current state of the art is using configuration manangement / orchestration tools to configure a system. While in many ways this is very high level, I’d argue that it is closer to assembly in the grand scheme of things. There is still room to encapsulate these tools and provide higher level abstractions that simplify and make safe the processes of working with systems.