Re: [Puppet Users] Scalability and performance

[Dan Bode]

On Tue, Oct 9, 2012 at 4:56 PM, Robjon <rober...@gmx.us> wrote:

Hi guys,

I am pretty new to this space, playing around with a few tools.
I am trying to read up on how I would scale Puppet (or other tools) up in my installation, and came across this blog post comparing Puppet and CFEngine: http://www.blogcompiler.com/2012/09/30/scalability-of-cfengine-and-puppet-2/

The numbers presented here are pretty extreme: CFEngine agents running 166 times faster than Puppet agents in a small installation

The results of that paper are not very realistic. The benchmark is based on doing nothing but running echo commands.  Since cfengine is written in C (or C++) there is not question that it will perform many actions faster than Puppet, but saying that it is 100X faster or whatever is disingenuous (unless you can manage your infrastructure with nothing but echo commands).

I would be more interested to see comparisons based on real admin tasks like managing packages or services.

 

- and the difference is increasing?
Also, it seems to be the case that Puppet is more centralized which results in everything slowing down: "as the master gets more loaded, all the Puppet agents run slower".

it is possible to either run puppet with or without a master. If you want more centralized control, use a master, if you need something that scales to the extreme, run puppet without a master using puppet apply (which is must more similar to how cfengine works)

 

Is this correct? Could some of you with more experience please comment on this?

Thanks.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/5LcBoBBaZGQJ.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[Luke Bigum]

On Wednesday, October 10, 2012 7:44:48 AM UTC+1, Dan Bode wrote:

On Tue, Oct 9, 2012 at 4:56 PM, Robjon <rober...@gmx.us> wrote:

Hi guys,

I am pretty new to this space, playing around with a few tools.
I am trying to read up on how I would scale Puppet (or other tools) up in my installation, and came across this blog post comparing Puppet and CFEngine: http://www.blogcompiler.com/2012/09/30/scalability-of-cfengine-and-puppet-2/

The numbers presented here are pretty extreme: CFEngine agents running 166 times faster than Puppet agents in a small installation

The results of that paper are not very realistic. The benchmark is based on doing nothing but running echo commands.  Since cfengine is written in C (or C++) there is not question that it will perform many actions faster than Puppet, but saying that it is 100X faster or whatever is disingenuous (unless you can manage your infrastructure with nothing but echo commands).

I would be more interested to see comparisons based on real admin tasks like managing packages or services.

As Dan said, the test case is rather biased. However the raw numbers are believable: CFEngine is "faster". If performance is your be-all and end-all, or you are paying per CPU cycle, then CFEngine is hard to argue against., but I wouldn't discount Puppet just yet. How you scale Puppet depends a lot on how you use it. If you have very computationally expensive manifests (lots of resources) then your Master needs more power (or more Masters). If you describe your site in more detail then I think a lot of people here would be happy to describe their architecture or give recommendations.

Also this blog post is only talking about performance and no other considerations like the tools' communities, existing modules/examples and the language itself. A trial you might like to do yourself is how to do the same thing in both languages and evaluate this (the language's) scalability.

[Trevor Vaughan]

Also, be aware that (unless this changed in 3.0), Exec statements and
Service statements in Puppet stack into some sort of queue.

Moving all Execs to simple native types will increase your performance
dramatically and there are good examples of how to do this in the
Puppet Labs stdlib module.

I posted about this before but I'm not sure if there was a resolution.

Trevor

--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvau...@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

[jcbollinger]

On Wednesday, October 10, 2012 6:48:20 AM UTC-5, Trevor Vaughan wrote:

Also, be aware that (unless this changed in 3.0), Exec statements and
Service statements in Puppet stack into some sort of queue.

Moving all Execs to simple native types will increase your performance
dramatically and there are good examples of how to do this in the
Puppet Labs stdlib module.

I posted about this before but I'm not sure if there was a resolution.

As far as I know, the Puppet agent is single-threaded and serializes application of all resources of all types.  I understood that to be an invariant of the execution model (i.e. intentional).  Is that not / no longer true?


John

[Trevor Vaughan]

It's a very good question and, yes, as far as I know, it is single threaded.

However, testing demonstrates a direct correlation between service and
exec statements and examining the code indicates that they use the
same routines to call out to the shell.

Also, I found that the translation of a brain dead simple exec could,
in most cases, change it from a >=1s runtime to a <=0.01s runtime.

Why this happens.....I'll let the PL guys answer....because I stopped
going down that rabbit hole when I figured out how.

The bad news is that services are just slow on the base OS so no help
there if you're managing a lot of them.

Trevor

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/puppet-users/-/KbSPhSZtFFkJ.

[ad]

Hi,

I've never used CFEngine and don't have input on those benchmarks other than they seem silly. But, fwiw...

- scaling Puppet masters seems to be straighforward and well documented (there are chapters in books on it)

- Performance in Puppet 3 is much better. Check out this presentation from PuppetConf - http://www.youtube.com/watch?v=8h2PbdFi0OU

- The key bottleneck of a Puppet Master is cpu usage when compiling catalogs. How long catalogs take to compile varies widely depending on your usage. I currently manage over 600 Windows nodes on a single, small, ec2 instance (one core) that has no problem keeping up. Most of the nodes run every 4 hours and we use splay settings to randomize the run intervals. We also currently have very small catalogs for these nodes (unlike our Linux server nodes handled by a separate master) that compile extremely fast.

- Puppet ships with a rarely mentioned load testing tool. Google this and try it out. See how it will handle your volume. Eventually how big of a server/cluster you need becomes a simple math problem.

- Also, really, how important is paying more for a bigger server to you when considering usability/features/community/momentum/etc differences?

Regards,

Adam

On Tuesday, October 9, 2012 6:56:55 PM UTC-5, Robjon wrote:

Hi guys,

I am pretty new to this space, playing around with a few tools.
I am trying to read up on how I would scale Puppet (or other tools) up in my installation, and came across this blog post comparing Puppet and CFEngine: http://www.blogcompiler.com/2012/09/30/scalability-of-cfengine-and-puppet-2/

The numbers presented here are pretty extreme: CFEngine agents running 166 times faster than Puppet agents in a small installation - and the difference is increasing?

Also, it seems to be the case that Puppet is more centralized which results in everything slowing down: "as the master gets more loaded, all the Puppet agents run slower".