At work, I have invested a lot of time in two tools that have made configuration and deployment as close to a painless process as I think is possible.
Puppet (available from Puppet Labs) is an amazing configuration tool that I have been working with for over a year. Since my place of work is cloud based, I need to spin up dozens of virtual machines that need to be identically configured automatically. Puppet allows you to achieve consistency over time as machines are configured by runs into a known state.
Now I have greater than 100 nodes and I want to perform some action on them to collect data or perform some action on each node in real time. SSH loops are fine for a couple of machine with a static list, but I have many nodes spread out in different locations and I am not a patient individual. Mcollective makes it possible to run massively parallel jobs across my infrastructure in seconds as opposed to minutes.
The use case that got me started was a co-worker says “We just got a call from customer XYZ and they say there is a problem. Quick – do something”. Because all of the nodes are in puppet and my configurations are in source control, I can immediately be sure of the state of my system configurations. I could log into monitoring and check each host and wait for some information, but instead I just run my mcollective check that goes out to each box and performs all monitoring checks in real time to see if there is some failure*. Within 30 seconds, I am confident that I can rule out the main two causes of trouble – configuration drift and network/host level issues and concentrate on the application itself. In the past this might have taken 10s of minutes to ascertain system state and it was most likely the culprit as to the current outage.
When I’m asked why you need both Puppet and Mcollective, I use the following shopping analogy to explain the relationship:
Puppet is the weekly shopping trip where you buy necessities and follow a list to ensure you have everything you need for a well stocked pantry of basic ingredients and what you need for dinner.
Mcollective is the quick run to the store to pick up a wine to compliment dinner.
The food is great, but the wine puts it over the top and the wine, while certainly nice by itself, lacks the foundation of a good meal.
Mcollective now handles deploys of software, monitoring checks, audits and many other functions on my company’s infrastructure when immediate action is required and is itself installed and configured by Puppet. It does require a significant upfront investment in time and a change in the way you think about processing requests, but is, in my opinion, necessary to grow your infrastructure and be responsive to business needs.
* for speed reference on part of my company’s infrastructure I can run approximately 1736 monitoring checks over 129 hosts in the following time
Finished processing 129 / 129 hosts in 3411.43 ms