Upgrading Puppet from 2.7 to 4

[christg76]

Hi, I'm fairly new to Puppet and have been given the project of upgrading an existing Puppet 2.7 site (Puppetmaster with Apache/Passenger, and MySQL for exported resources, with hundreds of clients), possibly to Puppet 4.
Just wanted to ask around if anybody has got experience with such an upgrade. Reading the docs it says that skipping major versions isn't recommended, so I guess I'd have to first upgrade to 3.x? Are there anymore upgrade docs for 2.7 to 3.x available?

Many thanks,
Chris

[Martin Alfke]

Hi Chris,

On 06 May 2016, at 15:46, christg76 <chri...@gmail.com> wrote:

> Hi, I'm fairly new to Puppet and have been given the project of upgrading an existing Puppet 2.7 site (Puppetmaster with Apache/Passenger, and MySQL for exported resources, with hundreds of clients), possibly to Puppet 4.

“fairly new to Puppet” and migration from 2.x to 4 does not fit well.
Go and teach yourself lots of stuff or attend a training or make use of the self paced training program at learn.puppet.com
Search for talks on upgrading to Puppet 4 (there have been plenty ones during the past 15 months).

> Just wanted to ask around if anybody has got experience with such an upgrade. Reading the docs it says that skipping major versions isn't recommended, so I guess I'd have to first upgrade to 3.x? Are there anymore upgrade docs for 2.7 to 3.x available?

In general:
upgrade master first, then the agents.
You can skip some minor versions. I am unsure whether you really can skip all.
Read the existing Puppet code and get yourself an idea on what is deprecated.

In your project: I would not go for upgrading from MySQL/Apache-Passenger. I would target a complete setup from scratch and migrate system via system.
This will allow you to build good Puppet 4 code from scratch and to not deal with old broken code which will make yourself banging your head against a wall.

hth,
Martin

>
> Many thanks,
> Chris
>
> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/92886404-4a00-45c6-92ef-eb336f5f1211%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Rob Nelson]

I have not done this, but I suspect a stairstep upgrade of 2.7 -> 3.0 -> 3.8 w/future parser -> 4.latest would be the easiest. You *might* be able to skip to 3.8, technically, but I suspect your code would blow up in strange ways that would be less obvious than the additional step.

Puppetconf 2015 talks that may be of specific interest to upgrades:
https://www.youtube.com/watch?v=LahcjL8UY-0&list=PLV86BgbREluUDlJW_jAqnWPj0THx7eXBA&index=12
https://www.youtube.com/watch?v=biqYDOJNTB4&index=14&list=PLV86BgbREluUDlJW_jAqnWPj0THx7eXBA
https://www.youtube.com/watch?v=D7rbUlNWOg4&list=PLV86BgbREluUDlJW_jAqnWPj0THx7eXBA&index=23

There are a ton of other talks about Puppet 4 in the playlist, though maybe not dealing specifically with upgrades.

Rob Nelson
rnel...@gmail.com

[Ramin K]

On 5/6/16 7:22 AM, Rob Nelson wrote:
> I have not done this, but I suspect a stairstep upgrade of 2.7 -> 3.0 ->
> 3.8 w/future parser -> 4.latest would be the easiest. You *might* be
> able to skip to 3.8, technically, but I suspect your code would blow up
> in strange ways that would be less obvious than the additional step.

Depending on your OS I'd lean towards this upgrade progression.

2.7.25 -> 3.2.4 (new master, migrate clients) -> 3.4.3 (optional) ->
3.8.7 -> 3.8.7 (future parser) -> 4.x (new master, migrate clients)

People tend to forget, but moving off 2.7 will likely change from Ruby
1.8.7 to 1.9.3+ which can cause templates to be evaluated differently.

Ramin

[Andrew Grimberg]

On 05/06/2016 06:59 AM, Martin Alfke wrote:
> Hi Chris,
>
> On 06 May 2016, at 15:46, christg76 <chri...@gmail.com> wrote:
>
>> Hi, I'm fairly new to Puppet and have been given the project of
>> upgrading an existing Puppet 2.7 site (Puppetmaster with
>> Apache/Passenger, and MySQL for exported resources, with hundreds
>> of clients), possibly to Puppet 4.
>
> “fairly new to Puppet” and migration from 2.x to 4 does not fit
> well. Go and teach yourself lots of stuff or attend a training or
> make use of the self paced training program at learn.puppet.com
> Search for talks on upgrading to Puppet 4 (there have been plenty
> ones during the past 15 months).

What Martin says here can't be stressed enough.

>> Just wanted to ask around if anybody has got experience with such
>> an upgrade. Reading the docs it says that skipping major versions
>> isn't recommended, so I guess I'd have to first upgrade to 3.x? Are
>> there anymore upgrade docs for 2.7 to 3.x available?
>
> In general: upgrade master first, then the agents. You can skip some
> minor versions. I am unsure whether you really can skip all. Read the
> existing Puppet code and get yourself an idea on what is deprecated.
>
> In your project: I would not go for upgrading from
> MySQL/Apache-Passenger. I would target a complete setup from scratch
> and migrate system via system. This will allow you to build good
> Puppet 4 code from scratch and to not deal with old broken code which
> will make yourself banging your head against a wall.

My team has been working on a transition (_not_ upgrade but migration!)
from 2.x to 4 since the middle of last year.

We stood up a completely separate puppet infra for our new Puppet4
configuration and are moving systems to it as we rebuild / bring up new
ones. There's two reasons for this:

1) Our old puppet architecture was a monolithic design which is really
hard to work with

2) Our Puppet 4 design is roles, profiles and hiera based. Additionally,
we don't accept modules into our infra that aren't released to the forge
with the exception of our roles and profiles modules themselves since
those are the internal business logic / stitching.

Because of those two requirements we're essentially green-fielding our
rearchitecting into Puppet4 which is allowing us to fix a lot of
outstanding technical debt issues. It also means that we've been
regularly building new modules that are clean, generic, and public
usable and getting them released to the Forge instead of the home grown
monstrosities we had previously or sending patches to upstream modules
that we use that need some tweaks to work better for our particular use
cases.

-Andy-

[Henrik Lindberg]

You have already received great advice on two strategies:

* Ramin K - points out the good versions to move to before jumping to
the next level. Each step requires addressing something fairly major
changing.

* Andrew Grimberg - greenfield approach on a 4.x to also do major
refactoring / cleanup.

I would just like to add a couple of considerations. With newer puppet
versions you also get the opportunity to use newer versions of modules
from the forge + better tooling. Your code base may contain solutions /
work-arounds that are now no longer needed (fixed in modules, better /
cleaner ways of doing things). Thus if you follow the longer path of
upgrading several times; you may need to learn / and re-learn how to do
things.

The chain of good versions (as pointed out by Ramin K) can also be made
shorter depending on how your operation is set up, how much the
operation depends on things like dynamic environments, how much custom
code you have to migrate, custom facts, types and providers that you
have implemented in house. Is your code in git or not? What are the
processes around your installation, what is it integrated with?

Also, worth considering is how different your nodes are and who controls
their configuration. Worst case it is like migrating multiple puppet
installations (different groups having written different things
different ways).

If things are really messy, what Andrew Grimberg suggests sounds best.
As you are learning (and do what Martin Alfke said; attend training),
set up a canary and use 4.x and apply best practices. Then you know what
the target is. Then "slice the elephant" and migrate nodes over
gradually. Now you "only" have to figure out what "old messy things"
intended and express then once in 4.x rather than risking changing a lot
of code several times and learning best practices as they were 3 years
ago and then what they were 2 years ago...

A very concrete example - if your in-house code base is fairly clean /
small someone else has already done the heavy lifting to get to 4.x as
most popular modules supports it.


Best of luck,
- henrik

--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

[christg76]

Thanks to everyone for the comments! I think I first need to do some preliminary testing in order to assess the quality of the code and to see what the real challenges are, and to ultimately decide on a strategy, ie upgrade vs transition/migration.
Ramin K: OS of the Master is Debian Wheezy. Have you actually done the upgrade?
Andrew Grimberg: This approach in fact sounds as if its the best way, considering also what Henrik says below about new versions/modules/tooling/practices. But it sounds like a massive amount of work, particularly since we do not have any unit tests or similar in place. Did you have any unit tests in place with the old code, or if not are you implementing them with the new code?

Chris

[Andrew Grimberg]

Chris,

Our old code base had no unit testing built in. That was one of the that
I, as our puppet architect, require of all the modules that we use. The
only exception (and this is mostly due to lack of some internal tooling)
is our roles / profiles modules.

If you're interested in our puppet layout I can't give you a link to our
current configuration (as much as I want and we'll eventually have it
available) but I _can_ give you a link to all the repositories I use for
my private servers and personal puppet lab :)

I have a from nothing to fully running article that I really need to
write utilizing these at some point. That being said here's the design
what you can see in how I (and by extension the company I work for) utilize:

Master puppet repo: https://github.com/tykeal/puppetserver-main
hiera: https://github.com/tykeal/puppetserver-hiera
role module: https://github.com/tykeal/puppetserver-mod-role
profile module: https://github.com/tykeal/puppetserver-mod-profile
local_fw module: https://github.com/tykeal/puppetserver-mod-local_fw.git

Yes, that really is the hiera module that I use. I'm not concerned about
having it visible because I use the hiera-eyaml-gpg backend and all the
truly secure information is GPG encrypted, anything not encrypted isn't
all that useful. Yes, it would help someone map how my environment goes
together but you would have to be able to get in first and if you can do
that a lot of stuff ends up in the clear anyway ;)

The Puppetfile in the master repo details every single module I
presently have in my testing environments. You'll notice most are just
pinned to Forge versions, some are pinned to github checkouts for
various reasons. I'll also note that in some cases, there are modules
that are being installed that I don't actually use (gentoo/portage for
instance), they're only there so I don't get the annoying warning
displays of unmet dependencies if I do a 'puppet module list' on the
puppet master ;)

The profile, role and local_fw modules are not pinned to a specific
checkout because it is expected that the puppet master should always be
at the lastest version of those and it gets to that state after every
puppet run as there is a post run job defined for the puppetmaster
itself that forces an r10k run to happen. I had initially tried it as a
pre-puppet run but always ran into some weird issues with it that way.

-Andy-

[Eric Sorenson]

Hi Chris, in addition to all the practical advice for moving through the upgrade cycles that you have gotten already, I would suggest taking a step back and analyzing your existing code-base to understand what the actual business function of the existing code is for the nodes. You can spend a ton of time doing things like adjusting syntax and eliminating deprecation warnings on a module, only to find out that the only host that is using that module was turned off a year ago and nobody noticed! 

So it might help to just draw out with a whiteboard or sticky notes what the existing mapping of puppet code to groups of machines looks like, what you think it SHOULD look like, and talk it over with your management/team to make sure the new setup is going the right direction. It's possible that you can save yourself a ton of work, plus be able to build a really good plan of the most valuable places to spend time. 

--eric0