Due to the holidays, I’ve had to add a large number of new nodes to our infrastructure. This started putting too much CPU and memory load on the puppet master. Instead of moving to a larger instance, I looked to spread out to multiple boxes.
This presented the problem of how the ops team could run tests against their own environments, how to handle the revocation and issuance of certs and keeping the manifests on the backends in sync.
Using nginx as a software load balancer solved all of these issues.
After talking with an ex-collegue ( I owe you some ramen eric0 ) I took a closer look at the URL paths being requested by the puppet clients.
Certificate requests start with /production/certificate so get routed to the puppet instance that only serves up certificates.
10.10.0.235 - - [14/Nov/2011:20:02:03 +0000] "GET /production/certificate/machine123.example.com HTTP/1.1" 404 60 "-" "-"
Each ops team member has their own environment for testing and the URLs start with the environment name
10.170.25.2 - - [14/Nov/2011:17:24:02 +0000] "GET /chris/file_metadata/modules/unixbase/fixes/file.conf HTTP/1.1" 200 330 "-" "-"
Everything else gets routed to a group of puppet backend servers.
The full nginx.conf file is available from GitHub.
Configurations are tested on the ops dev server then checked into a git repo that is pulled by all of the puppet backend servers.