Prevent certificate collisions due to servers going up and down with same hostname

[Iván del Castillo Zamora]

Hi!

We have a setup with a puppetmaster CA and several servers (AWS instances) which are spawned depending on the workload. On a daily basis from 50 to 100 instances can be spawned and shutdown (not at the same time), and what occurs is that a new server can have the IP and hostname . When a new certificate is created due to a new instance, this goes down after a while and if right after that a new instance with this just released IP (an IP 1.2.3.4 sets the hostname ip-1-2-3-4 in AWS, for example) is spawned, we get the usual SSL error as the private key has changed (a new one was generated in the last instance).
I have tried a quite dirty solution which involved a task running almost continuously which took every certificate from the SSL folder in the puppetmaster, and as the hostname(certname) includes the IP(just replace - with .), the script checked every IP against the whole list of IPs we have up at that moment, but in the end we are facing some race conditions due to timings so it just worked fine for a while.

It seems that we need a solution that is in sync with the state of the server when it boots up and it is shut down. Not all instances involved in this are located in a "Auto Scaling Group", so a solution I checked related to send notifications to a SNS queue sadly would not work for us.

We though of a solution which involved creating a new certificate, which should be stored in disk and add the directive certname in puppet.conf so every server presents the same certificate with the same private key and cert. We are already using autosign and as the puppetserver is only on the local network and firewalled it should not be a security issue to share the same certificate among our servers. We tested it manually, but we are afraid we will face another issue we did not foresee as it happened with the task I mentioned before.

Has anyone tried any of these solutions or are using a different approach?

Thanks a lot!

[Eric Sorenson]

Hi Iván - I think there are a couple of approaches that could work for you.

1 - you could continue to provision as you do today, but include a step in the shut-down that cleans a certificate. It is possible to add a rule to the auth.conf file that permits access to puppet's HTTPS endpoints which allows a node to delete its own certificate.  There's a blog post about it here:

http://www.nightbluefruit.com/blog/2015/02/allowing-puppet-agents-manage-their-own-certificates/

But I would suggest doing something a little nicer with the auth.conf rules, like this for /etc/puppetlabs/puppetserver/conf.d/auth.conf

        {

            "allow" : "$1",

            "match-request" : {

                "method" : "delete",

                "path" : "/puppet-ca/v1/certificate_status/([^/]+)$",

                "query-params" : {},

                "type" : "path"

            },

            "name" : "nodes deleting their own certs",

            "sort-order" : 500

        }

2 - You can indeed re-use the same cert and key for all your nodes. I have used this setup in production and it works pretty well but it is not a common best practice.  I have a write-up of how to do it here: https://gist.github.com/ahpook/1182243  but it is probably a bit out of date now.  

I would suggest going to option #1 but either could work for you.  hope this helps!

--eric0