Puppet Master Profiles

[Britt Gresham]

Hello,

We are interested in collecting puppet profile data from the puppet master in the real world. This data will be used to help us find a direction to go to increase the throughput of the puppet master master.

I've created a script that will create a yaml file that will have various system facts and profile data that you will need to generate from a master.

To generate this data follow the steps listed below.

1. Download the profiler script from https://github.com/demophoon/puppet-master-profiler
2. Run your puppet master with the `--debug`, `--profile`, and `--logdest=<path_to_a_blank_file>` flags
3. Wait for a few agents to check-in. You can tail the log file to see agents checking in to the master.
4. From the repository run `profile.rb <path_to_log_file>`. The log file will the the same path as the one specified above.
5. Answer the questions on screen and email the result file to me at br...@puppetlabs.com.

If you would like to do so send the apache log files for the time that the puppet master was running as well, it will help us see how the agent acts during the profile. We realize that some may not be comfortable doing this so it is completely optional.

If you have any questions or concerns please feel free to ask them here!

Thank you,

Britt Gresham

[Andy Parker]

On Mon, Sep 22, 2014 at 10:04 AM, Britt Gresham <br...@puppetlabs.com> wrote:

Hello,

We are interested in collecting puppet profile data from the puppet master in the real world. This data will be used to help us find a direction to go to increase the throughput of the puppet master master.

We haven't gotten any responses to this yet. :( Getting this information will really help in allowing us to target the right places for performance improvements.

 

I've created a script that will create a yaml file that will have various system facts and profile data that you will need to generate from a master.

To generate this data follow the steps listed below.

1. Download the profiler script from https://github.com/demophoon/puppet-master-profiler
2. Run your puppet master with the `--debug`, `--profile`, and `--logdest=<path_to_a_blank_file>` flags
3. Wait for a few agents to check-in. You can tail the log file to see agents checking in to the master.
4. From the repository run `profile.rb <path_to_log_file>`. The log file will the the same path as the one specified above.
5. Answer the questions on screen and email the result file to me at br...@puppetlabs.com.

If you would like to do so send the apache log files for the time that the puppet master was running as well, it will help us see how the agent acts during the profile. We realize that some may not be comfortable doing this so it is completely optional.

If you aren't able to run this in a production environment, that is fine as well. We just need some information from installations that have some load and are configured in the way it is normally used.

 

If you have any questions or concerns please feel free to ask them here!

Thank you,

Britt Gresham

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/5c2ec333-d92a-441b-98a8-8dc62e51795d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Andrew Parker

an...@puppetlabs.com

Freenode: zaphod42

Twitter: @aparker42

Software Developer

Join us at PuppetConf 2014, September 20-24 in San Francisco - www.puppetconf.com 

[Dean Wilson]

On 29 September 2014 18:20, Andy Parker <an...@puppetlabs.com> wrote:
> We haven't gotten any responses to this yet. :( Getting this information
> will really help in allowing us to target the right places for performance
> improvements.

Quite a few of us on this list run puppet in masterless mode so we
might not be the ideal target market. If you put a blog post up about
it I'm sure some of us will be happy to retweet/link to it for wider
viewing.

> If you aren't able to run this in a production environment, that is fine as
> well. We just need some information from installations that have some load
> and are configured in the way it is normally used.

As a side note I did have a look through the script to see if it'd be
something I would run in prod if I had a master and the length of the
code, most of which seems to be vendoring an optparser, is going to
deter quite a few admins from reviewing and running it (IMHO).

Dean
--
Dean Wilson http://www.unixdaemon.net
Profanity is the one language all programmers understand
--- Anon

[Andy Parker]

On Mon, Sep 29, 2014 at 10:27 AM, Dean Wilson <dean....@gmail.com> wrote:

On 29 September 2014 18:20, Andy Parker <an...@puppetlabs.com> wrote:
> We haven't gotten any responses to this yet. :( Getting this information
> will really help in allowing us to target the right places for performance
> improvements.

Quite a few of us on this list run puppet in masterless mode so we
might not be the ideal target market. If you put a blog post up about
it I'm sure some of us will be happy to retweet/link to it for wider
viewing.

Dean, thanks for the feedback. Actually running it on masterless can be useful as well. "puppet apply --profile --logdest /something" should capture a lot of information as well. It won't have some of the things the master ends up doing, but it will contain information about where the catalog compilation is taking time.

Britt, what do you think about a blog post?

 

> If you aren't able to run this in a production environment, that is fine as
> well. We just need some information from installations that have some load
> and are configured in the way it is normally used.

As a side note I did have a look through the script to see if it'd be
something I would run in prod if I had a master and the length of the
code, most of which seems to be vendoring an optparser, is going to
deter quite a few admins from reviewing and running it (IMHO).

That is great feedback!

Britt, I had a similar concern about vendoring the code and I think Dean is right. The script should be made as straightforward, simple, and small as possible so that people can understand what it is doing at a glance.

 

  Dean
--
Dean Wilson               http://www.unixdaemon.net
Profanity is the one language all programmers understand
--- Anon

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CAFbDO0e8HdHQCdUNvh054sx%3D5DFyL14-wwTS0DjZeMGyQwEjFg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Isaac Finnegan]

I’d be happy to do this, but we’re still running super old (2.6) master. Once we get upgraded, I’ll be happy to start capturing.

-Isaac

> --
> You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CAFbDO0e8HdHQCdUNvh054sx%3D5DFyL14-wwTS0DjZeMGyQwEjFg%40mail.gmail.com.

[Britt Gresham]

Britt, I had a similar concern about vendoring the code and I think Dean 

is right. The script should be made as straightforward, simple, and small 

as possible so that people can understand what it is doing at a glance.

 

I can make the changes to the script to remove the opt parser since there isn't many options that the script needs.

Quite a few of us on this list run puppet in masterless mode so we
might not be the ideal target market. If you put a blog post up about
it I'm sure some of us will be happy to retweet/link to it for wider
viewing.

I also like the idea of a blog post, I'll try and get that written up to get a larger audience.

--
Britt Gresham
Associate Software Engineer
br...@puppetlabs.com
Freenode: demophoon
Twitter: @demophoon

[Johan De Wit]

Here also, still running on 2.7, preparing to migrate to the latest 3.7
version, but will take some time.


--
Johan De Wit

Open Source Consultant

Red Hat Certified Engineer (805008667232363)
Puppet Certified Professional 2013/2014 (PCP0000006)
_________________________________________________________

Open-Future Phone +32 (0)2/255 70 70
Zavelstraat 72 Fax +32 (0)2/255 70 71
3071 KORTENBERG Mobile +32 (0)474/42 40 73
BELGIUM http://www.open-future.be
_________________________________________________________


Next Events:
Puppet Introduction Course | http://www.open-future.be/puppet-introduction-course-13th-october
Puppet Practitioner Training | http://www.open-future.be/puppet-practitioner-former-puppet-advanced-training-14-till-16th-october
Linux Training | http://www.open-future.be/linux-training-20-till-24th-october
Puppet Introduction Course | http://www.open-future.be/puppet-introduction-course-10th-november
Puppet Fundamentals Training | http://www.open-future.be/puppet-fundamentals-training-12-till-14th-november
Subscribe to our newsletter | http://eepurl.com/BUG8H

[Britt Gresham]

An update regarding the profile script,

I've updated the script to remove the vendored libraries so it should be more readable. It also has a new home at https://github.com/puppetlabs/puppet-master-profiler

Going forward that repository should be used instead of the old repository.

I've also added in the directions for people running in a masterless setup to give input as well. You all should be seeing a blog post about the data and what our plans are as soon as it is finished.

Thank you,

[Daniele Sluijters]

So I ran the script:

Do you wish to continue and have the script generate a /var/log/puppet/profile.yml file?

[Y/n]: Y

Script canceled by user.

It works with 'y', just not 'Y', which is a bit stupid. I realise that Y just means "I default to y" but the capitalisation shouldn't matter :).