Clean Puppet 3 to Puppet 4 migration

[Peter Berghold]

About three years ago (4 years ago?) I deployed a Puppet infrastructure for my company and department based on FOSS Puppet 3.7.   Given that's been deprecated of course I'm very much looking to migrate to Puppet 4.   Besides for about three months I worked for another company and got spoiled by Puppet 4. 

So I'm back at my old digs and assessing what it will take to go from P3 to P4.   Here are some of the things I have to think about

1) the migration needs to be fully automated both on the server end and the client end. 

2) no really.... this is going to be done by operations personnel who have a low threshold of fright for Puppet in spite of my best efforts to desensitize them.

3) There is as penetrable firewall between me (developer) and the Puppet infrastructure servers and clients so I cannot personally intervene

Anybody out there been through this pain and have any suggestions and pointers on how to make this happen with minimal "breakage?" 

[Michael Watters]

Been through a similar upgrade myself.  The first step would be to spin up a new puppet master running Puppet Server.  You can copy over the SSL dir from your old/current master to avoid SSL errors on the agents.  For testing you'll want to make sure your manifests work correctly using the future parser, that can be enabled in the puppet.conf file on the agents.  Setting up a separate environment for testing is also recommended.

Puppet Server will work with 3.7 clients so once you have a manifest that compiles correctly you can just point the agents to the new master's IP.

[Peter Souter]

One of the useful tools for an upgrade situation is the Catalog Preview tool. It requires Puppet 3.8, which should be a much smaller jump for you (3.7 -> 3.8) to begin with. It'll probably help soothe worries for the ops team by allowing a level of testing and certainty before the big jump.

It's pretty rad, because it will compile catalogs in the Puppet 3 format, and compare them with catalogs compiled with "future parser" turned on, which was the Puppet 4 compatibility mode avaliable for Puppet 3 for helping to upgrade.

Setup steps are here: https://forge.puppet.com/puppetlabs/catalog_preview

Another tool that's used is Githubs Octocatalog-diff https://github.com/github/octocatalog-diff, which is similar to catalog-preview

<Personal opinion>I generally recommend a cutover to a completely new infra, rather than trying to upgrade in place. This is more philosophical than technical, as it's probably easier if you have the server resources to create a new shiny Puppet 4 infra, then slowly just point old servers to the new shiny servers in batches, picking the easiest or least breaking servers first (eg. lab, dev, canary servers for testing) rather than trying to big-bang upgrade everything at once and worrying about rollback. But I know this can be a harder battle with change windows and getting everyone aligned.</Personal opinion>

In terms of automation, the new Bolt tool would be perfect for you here, as you can use bolt to run the commands needed to move servers to the new infra. 

We have a bootstrap task avaliable, but right now it's PE only (it uses a particular script we host to bootstrap Puppet, open-source support is coming in the future) but it wouldn't be too hard to just use bolt to either run a script (Such as my https://github.com/petems/puppet-install-shell) or write your own organisational specific task to do the steps you need, and rollback if something goes wrong.

I'd also recommend the following talks and blogposts about Puppet 3->4 upgrades:

• Rob Nelsons talk about AT&T's upgrade from PuppetConf 2016 (https://www.youtube.com/watch?v=FWnj0xQOZN8)
  
• Nacho Barrientos talk about CERN's Puppet 4 update (https://puppetconf17.sched.com/event/B4wI/a-not-so-bumpy-road-to-puppet4-at-cern-nacho-barrientos-cern Vids not out yet, soon!)
  
• Skroutz's blogpost about their upgrade (https://engineering.skroutz.gr//blog/upgrading-puppet3-to-puppet4/
  
• Githubs blogpost about their Puppet 4 Upgrade: https://puppet.com/blog/upgrading-to-puppet-4-at-github
  

There's also the #upgraders room on the Puppet Slack channel if you have other questions :) Feel free to ping me there with questions, if I'm around I might be able to help (@petems)