For almost two decades now, I’ve spun up servers, downloaded software or created USB keys to do SysAdmin magic on servers. This was Systems Engineering and at first I loved it. I did this for setting up IRC bots, IRC servers, Apache web servers, and finally running Secure Shell boxes for friends to run their own stuff on it. It’s hard to recall every configuration decision that I made at any given point over that timeline, and I’m sure I forgot a lot of great things along the way too. However, it started to become very cumbersome due to the number of tasks that needed to be repeated. It was either build-time tasks, or operational tasks which includes troubleshooting. I consider troubleshooting steps to be a set of operational tasks.

In the last 5 years, I had the privilege of being a part of what has been a tremendous evolution in terms of working with systems.  We’re calling this the DevOps Movement, and my exposure to DevOps itself was due to a tool called Puppet. A company I worked with had pioneered the use of Puppet to manage thousands of purpose-built security appliance nodes and it wasn’t love at first sight because this was something really different from the write-your-own-shell-scripts or copy and paste from text files way of building things I had put into practice. It was completely different from burning an image of a system and cloning it across appliances that are basically the same.

Today, I think I nailed this configuration management thing. So here are 3 ways to make it work for you: 

Start building the smart way.

When I first started building systems, it wasn’t always the most exciting task. It required hours of concentration to parse READMEs, Configuration files, and Shell scripts. One way or another with enough time and patience, things would work as expected and you’d hope to repeat those steps again if necessary.  I could have scripted this, and have in many cases, but it did nothing for me if I wanted it to work on FreeBSD systems or  RPM-based Linux distributions.  The scripts I wrote focused a lot on APT-based systems.

Now, it took some time for that Puppet thing to hit me, I just didn’t get it back in 2009. I could use Puppet, and for me it was useful for managing files, especially configuration files. I was copying files over to managed systems, and that was amazing because it didn’t happen with SSH. I was still missing the obvious, the DEB packages I was installing by hand during system installation, that’s part of configuration too!

There was a learning curve but the effort required to continue doing things in the ‘Idiot’ way was larger than the effort required to learn the ‘Smart’ way. Once I was able to understand the basic concepts and RTFM, it was a no-brainer to start defining all the individual packages needed to create a basic LAMP server.

Stop repeating yourself.

In early 2011 I had started to freelance and I began to offer services to individuals and companies that needed help cranking out servers, and custom configurations for their customers.  We’re talking 20 to 30 websites, applications, or production-replicas for development in a week.  This was unheard of for a single freelancer, and I was not able to hire some crack team of engineers to handle this, so out of necessity I needed to find a solution to this problem of production on a consistent basis.

It was right then and there when it became obvious, I had already built LAMP servers within Puppet’s configuration management and deployed it successfully.  It dawned on me, that I could consistently build servers with Apache 2.2 and PHP5 FPM 5.3 and had developed very stable performance optimized Apache2 and PHP configurations, (0.3s Page Loads for WordPress on tiny 1GB RAM servers) by building the ‘Smart’ way.

This ability to produce new servers and services in a ‘cookie cutter’ like fashion enabled me to iterate on configurations, to throw out what didn’t work and keep what did, to experiment and add things that would improve my server stacks and allow me to push these improvements to all the other “cookies” that were cut before. The best part of it was, I didn’t need to do these updates manually – the configuration management system handled it for me.

Don’t repeat yourself.  Use the configuration management system to build your systems, and to implement your changes, bug fixes, tests, and tweaks. It will help you considerably and shape your ability to think about problem solving at scale.

Use the community.

I was not the most experienced with Apache 2.2 and PHP5.5 FPM configurations back in 2011, but I followed some great guides from Biapy to setup Apache 2.2, PHP 5.3, MySQL 5.3 and APC. This had provided me with a very solid basis for creating a stack that worked for me and for my customers.

If it hadn’t been for the work Biapy had put into creating those HOWTO guides to setting up Apache2 with PHP 5.3 PFM using libapache2-mod-fastcgi and the trial and errors they won’t through to get to a stable configuration,  I doubt I would’ve been able to satisfy as many clients as I was able to.

The community had allowed me to stand on the shoulders of giants and look further than I had before and it will allow you to do the same. It will start you off in the right path, and get you to your destination in the shortest possible way. It won’t always be easy but it would be worth it.

As much as I like Puppet, it won’t do the whole job for you. You will need to develop skills, and you will need to adapt the way in which you work to fit within a new model. The journey is worth it.