In the past I have used nginx to route all cert requests to a single cert server. This worked fine when I had limited my puppet infrastructure to a single EC2 region. However, I recently decided to have puppet masters on separate coasts.
Keeping the certs in sync requires a two-way sync, so I ruled out just rsyncing files around. I tried deploying various drbd+clustered_file_system solutions, and while the tests worked within a region, I could not get them working well through the NAT of the two regions.
A helpful IRC regular (semiosis, thanks!) suggested unison. The issue was that a cron job might be too slow and I may run into issues performing the unison sync. There’s a very useful program that monitors file systems for changes and performs actions based on inode level changes called incron. So the obvious solution was to monitor the filesystem for changes then force a unison sync.
The final solution looks like this:
Install Unison
apt-get install unison
make sure ssh works
unison -batch -auto /etc/puppet/ssl/ca/signed \ ssh://puppet@OTHERPUPPETHOST//etc/puppet/sslca/signed
write simplescript on each host
#!/bin/bash /usr/bin/unison -batch -auto /etc/puppet/ssl/ca/signed \ ssh://puppet@OTHERPUPPETHOST//etc/puppet/ssl/ca/signed > /tmp/sync.log
Set the right mode
chmod +x /bin/puppet_cert_sync
add a crontab entry to make sure it stays kosher
31 * * * * /bin/puppet_cert_sync
install incron
sudo apt-get install incron
configure it to allow user puppet
echo "puppet" >> /etc/incron.allow
add the incrontab entry
export EDITOR=vi incrontab -e /etc/puppet/ssl/ca/signed IN_CLOSE_WRITE,IN_CREATE,IN_DELETE /bin/puppet_cert_sync
Then test on one host by running
watch -n 1 ls /etc/puppet/ssl/ca/signed/testhost.pem
And on the other host run
sudo puppetca --clean testhost
Caveats: This may not work in an environment with a many new certs being created very close to each other in both environments. It is also not as highly performant as a clustered file system, but seems to work well in my use case. In addition, the default puppet ssl directory is different, so adjust as necessary.