Puppet managing thousands of resources

[Zachary Vida]

Hello, I was wonder if there are any significant impovements in later version of puppet >= 2.6 to catolog compilation and/or application runtimes. 

In an environment I manage we populate many local files (/etc/passwd,/etc/group,/etc/hosts) via ENC. This results in a steady state catalog compilation/apply run times of several minutes and during an inital puppet apply clocking in at 90 minutes. 

[Lowe Schmidt]

How do  you run the puppetmaster? (Also, 2.6 is waaaay EOL)

--

Lowe Schmidt | +46 723 867 157

On 11 January 2017 at 18:58, Zachary Vida <vida...@gmail.com> wrote:

Hello, I was wonder if there are any significant impovements in later version of puppet >= 2.6 to catolog compilation and/or application runtimes. 

In an environment I manage we populate many local files (/etc/passwd,/etc/group,/etc/hosts) via ENC. This results in a steady state catalog compilation/apply run times of several minutes and during an inital puppet apply clocking in at 90 minutes. 

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/d7cc6062-722b-4f8a-9284-27ded5048c34%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Zachary Vida]

Lowe,

I am well aware it is very old. We have a single puppet master and a couple of hundred servers. I hope that answers your question?

On Wednesday, January 11, 2017 at 1:44:26 PM UTC-5, Lowe Schmidt wrote:

How do  you run the puppetmaster? (Also, 2.6 is waaaay EOL)

--

Lowe Schmidt | +46 723 867 157

On 11 January 2017 at 18:58, Zachary Vida <vida...@gmail.com> wrote:

Hello, I was wonder if there are any significant impovements in later version of puppet >= 2.6 to catolog compilation and/or application runtimes. 

In an environment I manage we populate many local files (/etc/passwd,/etc/group,/etc/hosts) via ENC. This results in a steady state catalog compilation/apply run times of several minutes and during an inital puppet apply clocking in at 90 minutes. 

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

[R.I.Pienaar]

Lots of improvements. Http connections are reused and new static catalogs will help. It's different for everyone just how much it helps though. 

---

R.I.Pienaar

On 11 Jan 2017, at 18:58, Zachary Vida <vida...@gmail.com> wrote:

Hello, I was wonder if there are any significant impovements in later version of puppet >= 2.6 to catolog compilation and/or application runtimes. 

In an environment I manage we populate many local files (/etc/passwd,/etc/group,/etc/hosts) via ENC. This results in a steady state catalog compilation/apply run times of several minutes and during an inital puppet apply clocking in at 90 minutes. 

[Trevor Vaughan]

How many resources are in your catalog?

Puppet starts to hit exponential catalog sizes and run times as you approach 10k resources.

Thanks,

Trevor

On Wed, Jan 11, 2017 at 12:58 PM, Zachary Vida <vida...@gmail.com> wrote:

Hello, I was wonder if there are any significant impovements in later version of puppet >= 2.6 to catolog compilation and/or application runtimes. 

In an environment I manage we populate many local files (/etc/passwd,/etc/group,/etc/hosts) via ENC. This results in a steady state catalog compilation/apply run times of several minutes and during an inital puppet apply clocking in at 90 minutes. 

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/d7cc6062-722b-4f8a-9284-27ded5048c34%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Trevor Vaughan
Vice President, Onyx Point, Inc

(410) 541-6699 x788

-- This account not approved for unencrypted proprietary information --

[Zachary Vida]

Exponential? Really or just an expression, but rough estimate is probably about 5k

On Wed, Jan 11, 2017, 2:50 PM Trevor Vaughan <tvau...@onyxpoint.com> wrote:

How many resources are in your catalog?

Puppet starts to hit exponential catalog sizes and run times as you approach 10k resources.

Thanks,

Trevor

On Wed, Jan 11, 2017 at 12:58 PM, Zachary Vida <vida...@gmail.com> wrote:

Hello, I was wonder if there are any significant impovements in later version of puppet >= 2.6 to catolog compilation and/or application runtimes. 

In an environment I manage we populate many local files (/etc/passwd,/etc/group,/etc/hosts) via ENC. This results in a steady state catalog compilation/apply run times of several minutes and during an inital puppet apply clocking in at 90 minutes. 

--

You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/d7cc6062-722b-4f8a-9284-27ded5048c34%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Trevor Vaughan
Vice President, Onyx Point, Inc

(410) 541-6699 x788

-- This account not approved for unencrypted proprietary information --

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/fS1cHkjuWco/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CANs%2BFoWo0EGkWKeLepqP%2BkOmbPN-hEa_cQV9Zq4C_H3caVuKSA%40mail.gmail.com.

[Trevor Vaughan]

By my calculations...really.

I'm around 3k normally right now with no issue but, as you move toward 10k, your agents start to have a bad time.

CC'ing Henrik and Eric since this is an "in the wild" case outside of my theoretical ranting.

It's pretty easy to test using 'puppet apply' and files with resource generators and 'notify' statements.

Thanks,

Trevor

On Wed, Jan 11, 2017 at 2:59 PM, Zachary Vida <vida...@gmail.com> wrote:

Exponential? Really or just an expression, but rough estimate is probably about 5k

On Wed, Jan 11, 2017, 2:50 PM Trevor Vaughan <tvau...@onyxpoint.com> wrote:

How many resources are in your catalog?

Puppet starts to hit exponential catalog sizes and run times as you approach 10k resources.

Thanks,

Trevor

On Wed, Jan 11, 2017 at 12:58 PM, Zachary Vida <vida...@gmail.com> wrote:

Hello, I was wonder if there are any significant impovements in later version of puppet >= 2.6 to catolog compilation and/or application runtimes. 

In an environment I manage we populate many local files (/etc/passwd,/etc/group,/etc/hosts) via ENC. This results in a steady state catalog compilation/apply run times of several minutes and during an inital puppet apply clocking in at 90 minutes. 

--

You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/d7cc6062-722b-4f8a-9284-27ded5048c34%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Trevor Vaughan
Vice President, Onyx Point, Inc

(410) 541-6699 x788

-- This account not approved for unencrypted proprietary information --

--

You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/fS1cHkjuWco/unsubscribe.

To unsubscribe from this group and all its topics, send an email to puppet-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CANs%2BFoWo0EGkWKeLepqP%2BkOmbPN-hEa_cQV9Zq4C_H3caVuKSA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAC0dne-h3Fu%2BxDARb6yzW0OiN6mPwSqa4UR7qjh28oe_gjAP4A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Christopher Wood]

Out of gruesome interest, 5000 resources of what?

Assuming I'm remembering the path correctly, something like this would count it up, modify for your local case (assuming no puppetdb at your place) to search for resource types:

python -m json.tool /var/lib/puppet/client_data/catalog/`hostname -f`.json | grep '"type":' | sort | uniq -c | sort -rn | head

What do you mean by populating local files via ENC? I'm drawing a blank. Or perhaps my mind is recoiling from the notion that you're sending contents of files in via top-level ENC variables.

(For reference catalog sizes around here go from 500-2000 resources with agent funs mostly from 15-90 seconds, 2-7 minutes in the initial run. (Because some departments do actually have sensible reasons to manage things with that granularity.) We're not so big though.)

On Wed, Jan 11, 2017 at 07:59:02PM +0000, Zachary Vida wrote:
> Exponential? Really or just an expression, but rough estimate is probably
> about 5k
>

> On Wed, Jan 11, 2017, 2:50 PM Trevor Vaughan <[1]tvau...@onyxpoint.com>

> wrote:
>
> How many resources are in your catalog?
> Puppet starts to hit exponential catalog sizes and run times as you
> approach 10k resources.
> Thanks,
> Trevor

> On Wed, Jan 11, 2017 at 12:58 PM, Zachary Vida <[2]vida...@gmail.com>

> wrote:
>
> Hello, I was wonder if there are any significant impovements in later
> version of puppet >= 2.6 to catolog compilation and/or application
> runtimes. 
> In an environment I manage we populate many local files
> (/etc/passwd,/etc/group,/etc/hosts) via ENC. This results in a steady
> state catalog compilation/apply run times of several minutes and
> during an inital puppet apply clocking in at 90 minutes. 
>
> --
>
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
>
> To unsubscribe from this group and stop receiving emails from it, send

> an email to [3]puppet-users...@googlegroups.com.

>
> To view this discussion on the web visit

> [4]https://groups.google.com/d/msgid/puppet-users/d7cc6062-722b-4f8a-9284-27ded5048c34%40googlegroups.com.
> For more options, visit [5]https://groups.google.com/d/optout.

>
> --
> Trevor Vaughan
> Vice President, Onyx Point, Inc
> (410) 541-6699 x788
> -- This account not approved for unencrypted proprietary information --
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit

> [6]https://groups.google.com/d/topic/puppet-users/fS1cHkjuWco/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to

> [7]puppet-users...@googlegroups.com.

> To view this discussion on the web visit

> [8]https://groups.google.com/d/msgid/puppet-users/CANs%2BFoWo0EGkWKeLepqP%2BkOmbPN-hEa_cQV9Zq4C_H3caVuKSA%40mail.gmail.com.
> For more options, visit [9]https://groups.google.com/d/optout.

>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to [10]puppet-users...@googlegroups.com.

> To view this discussion on the web visit

> [11]https://groups.google.com/d/msgid/puppet-users/CAC0dne-h3Fu%2BxDARb6yzW0OiN6mPwSqa4UR7qjh28oe_gjAP4A%40mail.gmail.com.
> For more options, visit [12]https://groups.google.com/d/optout.
>
> References
>
> Visible links
> 1. mailto:tvau...@onyxpoint.com
> 2. mailto:vida...@gmail.com
> 3. mailto:puppet-users...@googlegroups.com
> 4. https://groups.google.com/d/msgid/puppet-users/d7cc6062-722b-4f8a-9284-27ded5048c34%40googlegroups.com?utm_medium=email&utm_source=footer
> 5. https://groups.google.com/d/optout
> 6. https://groups.google.com/d/topic/puppet-users/fS1cHkjuWco/unsubscribe
> 7. mailto:puppet-users...@googlegroups.com
> 8. https://groups.google.com/d/msgid/puppet-users/CANs%2BFoWo0EGkWKeLepqP%2BkOmbPN-hEa_cQV9Zq4C_H3caVuKSA%40mail.gmail.com?utm_medium=email&utm_source=footer
> 9. https://groups.google.com/d/optout
> 10. mailto:puppet-users...@googlegroups.com
> 11. https://groups.google.com/d/msgid/puppet-users/CAC0dne-h3Fu%2BxDARb6yzW0OiN6mPwSqa4UR7qjh28oe_gjAP4A%40mail.gmail.com?utm_medium=email&utm_source=footer
> 12. https://groups.google.com/d/optout

[R.I.Pienaar]

----- Original Message -----
> From: "Christopher Wood" <christop...@pobox.com>
> To: "puppet-users" <puppet...@googlegroups.com>
> Sent: Wednesday, 11 January, 2017 22:33:22
> Subject: Re: [Puppet Users] Puppet managing thousands of resources

> Out of gruesome interest, 5000 resources of what?
>
> Assuming I'm remembering the path correctly, something like this would count it
> up, modify for your local case (assuming no puppetdb at your place) to search
> for resource types:
>
> python -m json.tool /var/lib/puppet/client_data/catalog/`hostname -f`.json |
> grep '"type":' | sort | uniq -c | sort -rn | head

last_run_summary.yaml will show totals already also total time per resource type :)

[Ramin K]

Depends on a number of factors. Without knowing more about the system
this is general advice and numbers.

- Upgrade to Puppet 3.8.x. Should be a fairly simple update in most
environments. Expect file serving and catalog compile to take 50% of the
time with 3.8 masters as compared to 2.7 masters. I don't have numbers
on 2.6 masters. You must have 2.7 agents to talk to 3.x masters.

- Upgrade the Puppet master to a distro running Ruby 1.9.3 or better.
Expect catalog compiles to drop to 50% of the 1.8.7 run time with Ruby
1.9.3 or 25% with 2.1. This can be a complex upgrade if you have
templates that use str.each or wrote your own functions.

- Replace source => 'puppet:///modules/sudo/sudoers', with content =>
file('sudo/sudoers'), in as many places as possible. Note do not do this
for binary files, there can be problems with utf8 data. The catalog will
be larger over the wire, but a faster apply since you can eliminate
additional https connections. However I expect you're doing a fair
amount of source => 'puppet:///modules/thing/some_dir_full_of_files/'
and file() can't help you there.

- Use Passenger, tune it appropriately.
https://ask.puppet.com/question/13433/how-should-i-tune-passenger-to-run-puppet/

For the record, I can admit to having "tens of thousands of hosts" and
over half of them have 4k+ file resources.

Ramin

[Christopher Wood]

On Wed, Jan 11, 2017 at 09:40:00PM +0000, R.I.Pienaar wrote:
>
>
> ----- Original Message -----
> > From: "Christopher Wood" <christop...@pobox.com>
> > To: "puppet-users" <puppet...@googlegroups.com>
> > Sent: Wednesday, 11 January, 2017 22:33:22
> > Subject: Re: [Puppet Users] Puppet managing thousands of resources
>
> > Out of gruesome interest, 5000 resources of what?
> >
> > Assuming I'm remembering the path correctly, something like this would count it
> > up, modify for your local case (assuming no puppetdb at your place) to search
> > for resource types:
> >
> > python -m json.tool /var/lib/puppet/client_data/catalog/`hostname -f`.json |
> > grep '"type":' | sort | uniq -c | sort -rn | head
>
> last_run_summary.yaml will show totals already also total time per resource type :)

That's fairly aggregate. Were this issue of lengthy agent run times presented to me I would start out being more interested in the resource types as they might appear in the manifests. I've had success reducing catalog compilation times by optimizing away from stacks of tiny resources (defines, classes, lists of stale ensure=>absent resources).

Here's a contrived example but based on something that happened here.

# super puppetdb querying of all catalogs here
200 File
160 Class
140 Package
70 Customsoftware::Includefile

If they're all files this specific method won't be a useful count but the notion of exploring just what's going on here is what I'm getting at.

> > What do you mean by populating local files via ENC? I'm drawing a blank. Or
> > perhaps my mind is recoiling from the notion that you're sending contents of
> > files in via top-level ENC variables.
> >
> > (For reference catalog sizes around here go from 500-2000 resources with agent
> > funs mostly from 15-90 seconds, 2-7 minutes in the initial run. (Because some
> > departments do actually have sensible reasons to manage things with that
> > granularity.) We're not so big though.)
>

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/1250936490.1005855.1484170800075.JavaMail.zimbra%40devco.net.