Advice on Puppet update to 4

[Stack Kororā]

Greetings,

First, a short intro. :-)

Last year I invested quite a bit of time building out a puppet infrastructure. I have +150  Scientific Linux 6 and RHEL 6 servers running puppet 3.7.1. It has been working really well concidering that I basically haven't touched it since last October....

The issues I see are things like:
* I never got around to moving off of WEBrick for the webserver. So the box takes a beating (it is always under a lot of load with that many clients).
* I haven't gotten around to updating past 3.7.1....that is kinda a big one...
* A lot of the poorly written modules _I_ wrote a year ago can now be easily replaced with much better versions from the forge.

Basically, things are pretty much my fault for not properly caring for it. :-D

But! It is working /really/ well. I don't have many changes except the addition/decomission of servers. I keep meaning to spend time, but it hasn't happened....until last weekend when the puppetmaster server went off line...I have the server functioning again, but the motherboard is *not* long for this world.

Since I am rebuilding anyway and I _know_ that our infrastructure is about to grow another 100 servers in the next few months, I figured I should look into upgrading to puppet 4.1. I have read through the changelog (didn't understand a lot, but I recognize there are a lot of changes I will have to make). If I am willing to take the time to migrate to puppet-forge modules anyway (plus hopefully fix any bad code I can't replace), wouldn't this be a good time to switch puppet versions too?

Questions!

1a)
I am debating as to whether I need to build multiple puppet masters or not. Is the following documentation still valid for the 4.x series of puppet?
https://docs.puppetlabs.com/guides/scaling_multiple_masters.html

I have a lot of load, a lot of puppet clients, and I wouldn't mind the reliability of multiple systems. But I am not sure if it is worth the effort or not. Are there good metrics for when to load balance or not?

I never got around to setting up puppet dashboard, but I would like to. Any load-balanced solution I would prefer to be one that worked with dashboard (unless there is a better recomendation out there??).

1b)
I have also been debating moving the puppetdb (that takes a ton of resources on the system) to a second server and just letting the first server be the puppetmaster/ca/ect. Thoughts on that? Any easier? Harder? I figure it would be a lot easier to configure the balance, but I am not sure what the consequences are or if it is even a good idea at all.

2)
I read in the changelog that the older clients should work with the backward compatibility, but is it worth even trying? Or should I just push out the updated puppet client wherever I can before adding the server into my new build? (not hard to do in my environment)

3)
Any got-cha!'s that I should be aware of? Any suggestions to make this process smoother? Any recommendations for a big jump (more like complete replacement) like this?

Thanks!
~Stack~

[jcbollinger]

On Monday, July 20, 2015 at 6:28:09 PM UTC-5, Stack Kororā wrote:

Greetings,

First, a short intro. :-)

Last year I invested quite a bit of time building out a puppet infrastructure. I have +150  Scientific Linux 6 and RHEL 6 servers running puppet 3.7.1. It has been working really well concidering that I basically haven't touched it since last October....

The issues I see are things like:
* I never got around to moving off of WEBrick for the webserver. So the box takes a beating (it is always under a lot of load with that many clients).
* I haven't gotten around to updating past 3.7.1....that is kinda a big one...
* A lot of the poorly written modules _I_ wrote a year ago can now be easily replaced with much better versions from the forge.

Basically, things are pretty much my fault for not properly caring for it. :-D

But! It is working /really/ well. I don't have many changes except the addition/decomission of servers. I keep meaning to spend time, but it hasn't happened....until last weekend when the puppetmaster server went off line...I have the server functioning again, but the motherboard is *not* long for this world.

Since I am rebuilding anyway and I _know_ that our infrastructure is about to grow another 100 servers in the next few months, I figured I should look into upgrading to puppet 4.1. I have read through the changelog (didn't understand a lot, but I recognize there are a lot of changes I will have to make). If I am willing to take the time to migrate to puppet-forge modules anyway (plus hopefully fix any bad code I can't replace), wouldn't this be a good time to switch puppet versions too?

Sure it would, if you can in fact accommodate that.

 

Questions!

1a)
I am debating as to whether I need to build multiple puppet masters or not. Is the following documentation still valid for the 4.x series of puppet?
https://docs.puppetlabs.com/guides/scaling_multiple_masters.html

It does say in a big callout box, "This document is specific to open source Puppet, versions 2.7 through 3.2".  The information within looks reasonably accurate as far as it goes, and it probably applies more or less to OS Puppet 4.  That page doesn't talk about puppetserver, however, which you should definitely be looking at as an alternative to both Webrick and Passenger.

 

I have a lot of load, a lot of puppet clients, and I wouldn't mind the reliability of multiple systems. But I am not sure if it is worth the effort or not. Are there good metrics for when to load balance or not?

You should consider balancing across multiple servers if one cannot manage the load by itself.  Catalog request timeouts, file service delays, and constantly running at or near CPU or RAM capacity are possible signs.  150 clients all checking in at the default 30-minute interval is far more than Webrick can usually handle.  On the other hand, Webrick is single-threaded, and you cannot easily buy a computer these days that does not sport multiple CPU cores.  Before opting to load-balance, you should look at ways to make one server shoulder more load by engaging more cores, giving it more RAM, or otherwise making more resources available to it.  This is where Passenger or puppetserver comes in.

 

I never got around to setting up puppet dashboard, but I would like to. Any load-balanced solution I would prefer to be one that worked with dashboard (unless there is a better recomendation out there??).

1b)
I have also been debating moving the puppetdb (that takes a ton of resources on the system) to a second server and just letting the first server be the puppetmaster/ca/ect. Thoughts on that? Any easier? Harder? I figure it would be a lot easier to configure the balance, but I am not sure what the consequences are or if it is even a good idea at all.

In a load-balancing scenario with puppetdb involved, it is essential that the masters all have the same logical view of puppetdb data.  It makes sense to designate a separate machine for that role instead of making one of the masters serve it, and it makes sense to reduce the load on your master by splitting out puppetdb to its own machine even before you opt for load balancing.
 

2)
I read in the changelog that the older clients should work with the backward compatibility, but is it worth even trying? Or should I just push out the updated puppet client wherever I can before adding the server into my new build? (not hard to do in my environment)

It's worth trying if upgrading the clients when you upgrade the master would present a challenge.  Even if you plan to update everything in one fell swoop, it may be to your advantage to rely on the newer master serving older clients at least for the duration of the upgrade.
 

3)
Any got-cha!'s that I should be aware of? Any suggestions to make this process smoother? Any recommendations for a big jump (more like complete replacement) like this?

I'd strongly advise you to set up a separate test network on which you can test the upgrade itself and validate all the code you intend to run in the upgraded environment.  Puppet 4 is not 100% backward compatible with Puppet 3 manifests (which is a different question from whether it is backward-compatible with Puppet 3 clients).


John

[Martijn]

Op dinsdag 21 juli 2015 01:28:09 UTC+2 schreef Stack Kororā:

If I am willing to take the time to migrate to puppet-forge modules anyway (plus hopefully fix any bad code I can't replace), wouldn't this be a good time to switch puppet versions too?

I would definitely think so, but one thing that I ran into while considering an update to Puppet 4 is that many Forge modules are not yet compatible with Puppet 4. This situation is of course improving and you can actively help make that happen for any modules that you intend to use, but if you're not that experienced with Puppet code yet it's something to be aware of.

Regards, Martijn

[Felix Frank]

On 07/21/2015 04:14 PM, jcbollinger wrote:

2)
I read in the changelog that the older clients should work with the backward compatibility, but is it worth even trying? Or should I just push out the updated puppet client wherever I can before adding the server into my new build? (not hard to do in my environment)

It's worth trying if upgrading the clients when you upgrade the master would present a challenge.  Even if you plan to update everything in one fell swoop, it may be to your advantage to rely on the newer master serving older clients at least for the duration of the upgrade.

Hi,

I don't think that Puppet 3 agents can work with the Puppet 4 master. There were incompatible changes to the catalog format.

A Puppet 4 upgrade needs to encompass your whole infrastructure. As such, I'd advise to try and not throw it in with other changes.

Cheers,
Felix

[Stack Kororā]

On Tuesday, July 21, 2015 at 9:14:04 AM UTC-5, jcbollinger wrote:

 

Questions!

1a)
I am debating as to whether I need to build multiple puppet masters or not. Is the following documentation still valid for the 4.x series of puppet?
https://docs.puppetlabs.com/guides/scaling_multiple_masters.html

It does say in a big callout box, "This document is specific to open source Puppet, versions 2.7 through 3.2".  The information within looks reasonably accurate as far as it goes, and it probably applies more or less to OS Puppet 4.  That page doesn't talk about puppetserver, however, which you should definitely be looking at as an alternative to both Webrick and Passenger.

The differences between puppetserver and webbrick/Passenger were actually the big concerns I had for that documentation. I did see that warning, but I couldn't find an equivelent page for 4 so I figured it was best to ask. :-)

 

I have a lot of load, a lot of puppet clients, and I wouldn't mind the reliability of multiple systems. But I am not sure if it is worth the effort or not. Are there good metrics for when to load balance or not?

You should consider balancing across multiple servers if one cannot manage the load by itself.  Catalog request timeouts, file service delays, and constantly running at or near CPU or RAM capacity are possible signs.  150 clients all checking in at the default 30-minute interval is far more than Webrick can usually handle.  On the other hand, Webrick is single-threaded, and you cannot easily buy a computer these days that does not sport multiple CPU cores.  Before opting to load-balance, you should look at ways to make one server shoulder more load by engaging more cores, giving it more RAM, or otherwise making more resources available to it.  This is where Passenger or puppetserver comes in.

The old hardware was a 2.2Ghz 4 core with 4GB of memory. The new hardware is a 3Ghz 8 core with 8GB of memory. I read a few forum posts about puppetserver doing much better at multi-threading, but I haven't found any hard metrics on it. I decided to build out a fresh install of Scientific Linux 6 with Puppet 4 today and start seeing how many of my modules will move over and break. To make it simple to get working, I am doing it all on one box, but I am thinking about moving the puppetDB to a second box after I verify my setup works with a few clients. :-)
 

 
1b)
I have also been debating moving the puppetdb (that takes a ton of resources on the system) to a second server and just letting the first server be the puppetmaster/ca/ect. Thoughts on that? Any easier? Harder? I figure it would be a lot easier to configure the balance, but I am not sure what the consequences are or if it is even a good idea at all.

In a load-balancing scenario with puppetdb involved, it is essential that the masters all have the same logical view of puppetdb data.  It makes sense to designate a separate machine for that role instead of making one of the masters serve it, and it makes sense to reduce the load on your master by splitting out puppetdb to its own machine even before you opt for load balancing.
 

Awesome! Thanks for the confirmation. I will pursue that route first and see what kind of load/need I have for multiple puppet masters.

3)
Any got-cha!'s that I should be aware of? Any suggestions to make this process smoother? Any recommendations for a big jump (more like complete replacement) like this?

 

I'd strongly advise you to set up a separate test network on which you can test the upgrade itself and validate all the code you intend to run in the upgraded environment.  Puppet 4 is not 100% backward compatible with Puppet 3 manifests (which is a different question from whether it is backward-compatible with Puppet 3 clients).

I think I am just going to move forward with my proposed-new-build with 4 and a few clients. If everything works, then I will handle the servers in chunks (test the webservers with one node, work out the kinks, move another to verify, then move all of the other webservers before testing other groups).

Thanks for the advice!

[Stack Kororā]

Thanks for heads up. I totally didn't catch that until I read your post. I went and looked at a few of the modules I had planned to use and saw that they were not all 4 compatible.

[Stack Kororā]

Thanks for the advice. I am going to test for a bit on 4 and if all is well, I will update my whole infrastructure in groups upgrading the client as I go. Fingers crossed the old puppet master server hangs on that long! :-)

[Stack Kororā]

One last update on my progress.

I have 4 installed on Scientific Linux with the only client being itself at the moment. The current plan is to either find replacement modules or move over as many modules as I can. If I can get the bulk of the modules and a few important ones, then I will be very happy. I have new hardware that is significantly better and I am really concidering moving puppetdb to a second node. But I will do that when I am done with a bit more module testing.

The only modules I have tested so far are the ones from puppet-labs: concat, stdlib, firewall, and ntp. Strangely enough, I was getting all kinds of errors with the firewall module. I disabled it, ran 'puppet agent -t' again where it configured ntp and had a successful run, then re-enabled it to get an error to report on...and it worked! I am not sure what happened...If it happens again, I will capture more detailed output.

I knew there were a lot of changes, but wowzer. I kept thinking "Huh...that's different. I guess that is what the documentation ment when it said $feature had changed..." :-)

It is also good, because I have been wanting to implement the environments feature for a while, but never got around to it. This is basically forcing me to. :-)

Lastly, I have always had a bit of a rough time with hiera data not picking up at times (see my previous posts for frustrations last year), but a few quick tests worked a LOT better this time. I am guessing it is just the combination of all the updates and improvements (puppet/heira/new hardware/new environment that I haven't managed to mangle in strange ways yet/ect :-D ).

I will be working on this quite a bit this week. I will keep you updated as I go.

Thanks for the help and advice!

[Kevin Corcoran]

Hi,

We added a new feature in Puppet Server 2.1.0 which allows it to be used with Puppet 3 agents (as well as Puppet 4 agents, of course).

https://docs.puppetlabs.com/puppetserver/latest/release_notes.html#whats-new-in-puppet-server-21

- Kevin

[Felix Frank]

On 07/22/2015 11:06 PM, Kevin Corcoran wrote:
> Hi,
>
> We added a new feature in Puppet Server 2.1.0 which allows it to be
> used with Puppet 3 agents (as well as Puppet 4 agents, of course).
>
> https://docs.puppetlabs.com/puppetserver/latest/release_notes.html#whats-new-in-puppet-server-21
>
> - Kevin

Mind. Blown.

No really, that is great news!

Thanks,
Felix

[Stack Kororā]

On Wednesday, July 22, 2015 at 4:06:51 PM UTC-5, Kevin Corcoran wrote:

We added a new feature in Puppet Server 2.1.0 which allows it to be used with Puppet 3 agents (as well as Puppet 4 agents, of course).

https://docs.puppetlabs.com/puppetserver/latest/release_notes.html#whats-new-in-puppet-server-21

Awesome; that is good news! Thanks!


Just as an update to my progress, I found a few modules that work really well to replace some of my not-so-good-but-functional modules. :-)

It also appears to me that puppet dashboard is not yet ready for 4. Anyone have any thoughts on that?

Thanks!

[Felix Frank]

On 07/23/2015 12:45 AM, Stack Kororā wrote:
>
> It also appears to me that puppet dashboard is not yet ready for 4.
> Anyone have any thoughts on that?

Hi,

the dashboard was discontinued, afaik, so it may never be "ready" to
work with Puppet 4+.

If you need a good ENC, look at Foreman. If you need reporting, look at
puppetboard or Puppet Explorer.

Best,
Felix

[Dirk Heinrichs]

Am 24.07.2015 um 12:59 schrieb Felix Frank:

If you need a good ENC, look at Foreman. If you need reporting, look at
puppetboard or Puppet Explorer.

Is any of those ready for Puppet 4? At least Foreman 1.9 (current development version) doesn't seem to be. There is an open ticket, though...

Bye...

    Dirk

--

Dirk Heinrichs, Senior Systems Engineer, Engineering Solutions
Recommind GmbH, Von-Liebig-Straße 1, 53359 Rheinbach
Tel: +49 2226 1596666 (Ansage) 1149
Email: d...@recommind.com
Skype: dirk.heinrichs.recommind
www.recommind.com

[Fabrice Bacchella]

> Le 24 juil. 2015 à 14:10, Dirk Heinrichs <dirk.he...@recommind.com> a écrit :
>
> Am 24.07.2015 um 12:59 schrieb Felix Frank:
>
>> If you need a good ENC, look at Foreman. If you need reporting, look at
>> puppetboard or Puppet Explorer.
>
> Is any of those ready for Puppet 4? At least Foreman 1.9 (current development version) doesn't seem to be. There is an open ticket, though...

Puppetboard not yet.

Puppet Explorer should be, but I still have to found a way to make it works wit puppetdb.

[Fabrice Bacchella]

I know why I failed: puppet explorer is not compatible either with Puppetdb. The APIv4 from puppetdb 2.1 is not the APIv4 from Puppet 3 (cough cough) :
http://docs.puppetlabs.com/puppetdb/2.3/api/query/v4/metrics.html
is now moved to
http://docs.puppetlabs.com/puppetdb/3.0/api/metrics/v1/mbeans.html

[Byron Miller]

Foreman doesn't support puppet 4.x yet. Lots of open tickets :)

[Wyatt Alt]

Fabrice,

Correct. The differences are listed here:
http://docs.puppetlabs.com/puppetdb/3.0/api/query/v4/upgrading-from-v3.html
. Anything missing from that list is a docs bug that we'll be glad to fix.

Wyatt