Beware breaking updates on puppetserver v2.5.0 with external CA, downgrade ALSO destructive

[Ben West]

The configuration for running Open Source puppetserver with an external CA changed in v2.4 -> v2.5, explained in more detail here:
https://docs.puppet.com/puppetserver/latest/bootstrap_upgrade_notes.html#cacfg

If you happen to run yum upgrade (presumably similar results with apt-get update), the package's upgrade process for v2.4 -> v2.5 will actually delete any existing copy of /etc/puppetlabs/puppetserver/bootstrap.cfg. Which is reasonable.

HOWEVER, if you try to downgrade puppetserver to roll back, e.g. "yum downgrade puppetserver-2.4.0-1.el7," the package downgrade process will overwrite /etc/puppetlabs/puppet/ssl/crl.pem and break your Puppetserver's SSL. Which isn't particularly reasonable.

Options for fixing are A) restore crl.pem from backup, B) restore crl.pem from the CA's ca_crl.pem file (if it is also a puppetserver), or C) to regenerate all of your puppet SSL certs.

Possible to add mention this downgrade pitfall in the Puppetserver v2.5.0 release notes?
https://docs.puppet.com/puppetserver/2.5/release_notes.html


[Jeremy Barlow]

On Thursday, August 18, 2016 at 9:53:37 AM UTC-7, Ben West wrote:

The configuration for running Open Source puppetserver with an external CA changed in v2.4 -> v2.5, explained in more detail here:
https://docs.puppet.com/puppetserver/latest/bootstrap_upgrade_notes.html#cacfg

If you happen to run yum upgrade (presumably similar results with apt-get update), the package's upgrade process for v2.4 -> v2.5 will actually delete any existing copy of /etc/puppetlabs/puppetserver/bootstrap.cfg. Which is reasonable.

If you have made modifications to the bootstrap.cfg file before the upgrade, I think (at least on systems which use yum) that the upgrade will additionally store a backup copy of the file to /etc/puppetlabs/puppetserver/bootstrap.cfg.rpmsave.
 

HOWEVER, if you try to downgrade puppetserver to roll back, e.g. "yum downgrade puppetserver-2.4.0-1.el7," the package downgrade process will overwrite /etc/puppetlabs/puppet/ssl/crl.pem and break your Puppetserver's SSL. Which isn't particularly reasonable.

I don't think it's the "yum downgrade" itself which does this.  The puppetserver service, when running in a standard (non-external CA) setup has logic in it which, at startup, will try to copy over the file configured for the hostcrl setting with the file configured for the cacrl setting.  It does this to ensure that the CRL file used by puppetserver's web server reflects the latest updates that have may have been done to the cacrl file since the last startup.  This file synchronization does not occur in the "external CA" configuration, though, since the cacrl file is not used in that case.  My guess is that after downgrade, the original bootstrap.cfg file from the puppetserver-2.4.0 package was reinstalled, re-enabling the standard CA service and, therefore, the CRL file synchronization logic.

I think you might have been able to avoid having the CRL file overwritten if you had copied the file at /etc/puppetlabs/puppetserver/bootstrap.cfg.rpmsave back to /etc/puppetlabs/puppetserver/bootstrap.cfg before the downgrade.  In that case, I think the disabled CA service would continue to be used when the puppetserver-2.4.0 service had been restarted.  Does that make sense?

 

Options for fixing are A) restore crl.pem from backup, B) restore crl.pem from the CA's ca_crl.pem file (if it is also a puppetserver), or C) to regenerate all of your puppet SSL certs.

Possible to add mention this downgrade pitfall in the Puppetserver v2.5.0 release notes?
https://docs.puppet.com/puppetserver/2.5/release_notes.html


Yeah, I think we could add some more detail to the release notes about this case.  I definitely think the best way to manage through these issues is to ensure that the disabled (external) CA service continues to be used before upgrade, after upgrade, and (if applicable) after downgrade.  The release notes do try to cover the best way to prepare for keeping the disabled CA service in place on upgrade - putting the ca.cfg file in place, as you mentioned.  Maybe a little more detail on the same for a downgrade could be helpful.

[Ben West]

At rate any, yes, that running "yum update puppetserver" followed by "yum downgrade puppetserver-2.4.0-1.el7" without any other precautions may hose the CRL seems like it deserves mention.

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/2eHcuhJejKA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/83160dc2-453f-4b76-b8ef-5384687f8a51%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Ben West

m...@benwest.name