Sync Puppet Certs between EC2 regions

In the past I have used nginx to route all cert requests to a single cert server. This worked fine when I had limited my puppet infrastructure to a single EC2 region. However, I recently decided to have puppet masters on separate coasts.

Keeping the certs in sync requires a two-way sync, so I ruled out just rsyncing files around. I tried deploying various drbd+clustered_file_system solutions, and while the tests worked within a region, I could not get them working well through the NAT of the two regions.

A helpful IRC regular (semiosis, thanks!) suggested unison. The issue was that a cron job might be too slow and I may run into issues performing the unison sync. There’s a very useful program that monitors file systems for changes and performs actions based on inode level changes called incron. So the obvious solution was to monitor the filesystem for changes then force a unison sync.

The final solution looks like this:

Install Unison

apt-get install unison

make sure ssh works

unison -batch -auto /etc/puppet/ssl/ca/signed \
ssh://puppet@OTHERPUPPETHOST//etc/puppet/sslca/signed

write simplescript on each host

#!/bin/bash
 
/usr/bin/unison -batch -auto /etc/puppet/ssl/ca/signed \
ssh://puppet@OTHERPUPPETHOST//etc/puppet/ssl/ca/signed > /tmp/sync.log

Set the right mode

 chmod +x /bin/puppet_cert_sync

add a crontab entry to make sure it stays kosher

31 * * * * /bin/puppet_cert_sync

install incron

sudo apt-get install incron

configure it to allow user puppet

echo "puppet" >> /etc/incron.allow

add the incrontab entry

export EDITOR=vi
incrontab -e
 
/etc/puppet/ssl/ca/signed IN_CLOSE_WRITE,IN_CREATE,IN_DELETE /bin/puppet_cert_sync

Then test on one host by running

watch -n 1 ls /etc/puppet/ssl/ca/signed/testhost.pem

And on the other host run

sudo puppetca --clean testhost

Caveats: This may not work in an environment with a many new certs being created very close to each other in both environments. It is also not as highly performant as a clustered file system, but seems to work well in my use case. In addition, the default puppet ssl directory is different, so adjust as necessary.

• Tweet
• Pin It