Need help optimizing our Puppet module

[Harrison Ripps]

Hey all--

Over at OpenShift we are big fans of Puppet. We've developed our own module, which is central to our OpenShift Origin installer and to our Vagrant-based development environment. As the author of the installer and the current project lead for OpenShift Origin, I really want to promote these tools across our entire team, ops and developers.

The challenge that I am facing is that right now, our Puppet module seems pretty slow. I am not a Puppet expert, but I'm willing to bet that someone with decent Puppet skills would look at our module and chuckle inwardly about the obvious performance potholes that we're driving through. Hopefully, after that, you'd be willing to follow up on this thread and let me know what we can do to get the lead out.

So, if you have a few cycles, please have a look at https://github.com/openshift/puppet-openshift_origin and let me know how you think we can make it better and faster.

Thanks!

Harrison

[Pete Brown]

Hi Harrison,

Here are some initial thoughts after a quick glance at your code.

First question I would have would be why are you using ensure_resource
to do one thing at a time?
Like install one package or one service etc.
I love wrappers but it seems like overkill to use it do do something
the native resource will do itself.
The native resource is likely much faster.

If you were using ensure_resource to say create 1000 users then sure go for it.
Yeah it make things if they don't exist but to me it's sloppy coding
and a waste of resources (pun intended).

Second question is why do you have a define inside a class?
It will probably have no effect on the speed but it's not good
practice and makes the code harder to read.

Hope that helps.

Pete.

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/40bd9c96-b63e-4752-a487-661591a35d02%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Denmat]

Hi,

And further to that I also did a quick skim and found you use augeas for a few resources that can probably be better executed with simpler resource types (I'm looking at you yumrepo). Augeas is costly from my understanding.

Also to get a good view of where the costs are in your manifest you can use the practically undocumented '--evaltrace'. It will show how long it takes to apply a resource on the host.

Cheers
Den

> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAJ8DPF7UJZKM6RGxnx-PFcVg9eNnQ%3DJxXeNaFM%2BO4jRH5JOKxQ%40mail.gmail.com.

[jcbollinger]

On Monday, April 7, 2014 5:06:25 PM UTC-5, denmat wrote:

Hi,

And further to that I also did a quick skim and found you use augeas for a few resources that can probably be better executed with simpler resource types (I'm looking at you yumrepo). Augeas is costly from my understanding.

Also to get a good view of where the costs are in your manifest you can use the practically undocumented '--evaltrace'. It will show how long it takes to apply a resource on the host.

Excellent advice.  Also, the module is BIG.  It manages a lot of pieces, so to some extent it can be expected to take some time for that reason alone.  That's especially true on the first run because the module manages a lot of Packages, and installing packages is comparatively expensive.

Additionally, if you are managing any very large files, or recursively managing and directory trees containing many files, then computing the File checksums may be imposing a substantial cost on you.

And following on from Pete, I have rarely seen such a consistent and expansive collection of coding practices that I deplore.  Specifically,

• the ensure_resource() function should never be used.  ever.
  
• the resource-like form of class declarations should generally be avoided, and it should always be avoided when no parameters are declared
  

Neither of those is likely to impose much performance penalty in themselves, but they do make your code harder to maintain and use.


John

[Harrison Ripps]

Hey all--

Thanks for this great feedback; much appreciated. I'll take a shot at making some of these changes and see if any other speed benefit opportunities materialize in the process. I'll post my GitHub pull requests here in case you're interested in commenting on the changes.

Cheers,

Harrison

[Henrik Lindberg]

On 2014-07-04 22:03, Harrison Ripps wrote:
> Hey all--

> Over at OpenShift <https://www.openshift.com/> we are big fans of

> Puppet. We've developed our own module

> <https://github.com/openshift/puppet-openshift_origin>, which is central

> to our OpenShift Origin installer and to our Vagrant-based development
> environment. As the author of the installer and the current project lead
> for OpenShift Origin, I really want to promote these tools across our
> entire team, ops and developers.
>
> The challenge that I am facing is that right now, our Puppet module
> seems pretty slow.

I wonder what it is that you have measured to be slow? Is it the master
side catalog production, or the application of the catalog on the agent?
or both?

On the master side, you can use rspec-puppet to test the catalog, and at
the same time benchmark how long it takes to generate the catalog.
This is very useful when trying to optimize since you a) want to ensure
you get the catalog you want, b) you need to measure it, and c) you
don't want to actually run it against an agent each time.

Just 2c more...

- henrik

[Harrison Ripps]

For those who are interested, here is the PR that I am planning to merge based on your feedback. I teamed up with a Puppet superstar at my office to pull this together. Any other comments that you have before I merge this are certainly welcome!

https://github.com/openshift/puppet-openshift_origin/pull/192

Thanks again for the reviews; much appreciated.

--Harrison