can we alter the 30-min run time?

[Jo Rhett]

For a long time it appeared that run cycles were fairly balanced -- a few every 30 seconds over the 30 minute period. Right now I'm seeing more than 100 systems hit in the same minute: 28 and 58 minutes after the hour. Is there some way to alter the spread of these systems back to even out the load?

Or passenger options which could limit the effects of this?

--
Jo Rhett
Net Consonance : consonant endings by net philanthropy, open source and other randomness

[Nigel Kersten]

On Thu, Nov 3, 2011 at 8:36 AM, Jo Rhett <jrh...@netconsonance.com> wrote:

For a long time it appeared that run cycles were fairly balanced -- a few every 30 seconds over the 30 minute period.  Right now I'm seeing more than 100 systems hit in the same minute: 28 and 58 minutes after the hour.  Is there some way to alter the spread of these systems back to even out the load?

Or passenger options which could limit the effects of this?

In your puppet.conf agent block:

    # How often puppet agent applies the client configuration; in seconds.

    # Note that a runinterval of 0 means "run continuously" rather than

    # "never run." If you want puppet agent to never run, you should start

    # it with the `--no-client` option.

    # The default value is '1800'.

    runinterval = 1800

...

    # The maximum time to delay before runs.  Defaults to being the same as the

    # run interval.

    # The default value is '$runinterval'.

    splaylimit = 1800

...

    # Whether to sleep for a pseudo-random (but consistent) amount of time before

    # a run.

    splay = false

 

--
Jo Rhett
Net Consonance : consonant endings by net philanthropy, open source and other randomness

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

--
Nigel Kersten
Product Manager, Puppet Labs

[Randall Hansen]

On Thu, Nov 3, 2011 at 8:38 AM, Nigel Kersten <ni...@puppetlabs.com> wrote:

> In your puppet.conf agent block:

> ...
>     # The maximum time to delay before runs.  Defaults to being the same as the
>     # run interval.
>     # The default value is '$runinterval'.
>     splaylimit = 1800

Having this uncommented as this value is a noop, right?

> ...
>     # Whether to sleep for a pseudo-random (but consistent) amount of time before
>     # a run.
>     splay = false

I think we should document this better; I don't feel any closer to
being able to solve this problem. I would probably set "splay" to
"true" and hope that it worked, but it wouldn't surprise me if I had
to go back and tweak something else. I also can't tell if you're just
telling Jo to read the fine manual, or if you're suggesting
configuration values which will solve the problem.

(Don't anyone take this as a flame on Nigel; he's sitting 6 feet away
from me right now :)

r

[Christopher Wood]

(splay inline)

On Thu, Nov 03, 2011 at 08:38:39AM -0700, Nigel Kersten wrote:
> On Thu, Nov 3, 2011 at 8:36 AM, Jo Rhett <[1]jrh...@netconsonance.com>

On this side, my difficulty in using splay is that it interferes with "puppet kick":

http://projects.puppetlabs.com/issues/1100

I work around this by not using play, instead templating my puppet.conf and making my runinterval different on each host:

----------------------------------------
[agent]
runinterval=<%=

# sixty minutes plus up to sixty minutes
# average runinterval might be ninety minutes

srand MD5.new(fqdn).to_s.hex
num = 3600 + rand(3600)

%>
----------------------------------------

They'll intersect at some point, but I'm not worried if they only do it occasionally.

> --
> Jo Rhett
> Net Consonance : consonant endings by net philanthropy, open source and
> other randomness
>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.

> To post to this group, send email to [2]puppet...@googlegroups.com.

> To unsubscribe from this group, send email to

> [3]puppet-users...@googlegroups.com.

> For more options, visit this group at

> [4]http://groups.google.com/group/puppet-users?hl=en.

>
> --
> Nigel Kersten
> Product Manager, Puppet Labs
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>

> References
>
> Visible links
> 1. mailto:jrh...@netconsonance.com
> 2. mailto:puppet...@googlegroups.com
> 3. mailto:puppet-users%2Bunsu...@googlegroups.com
> 4. http://groups.google.com/group/puppet-users?hl=en

[Nigel Kersten]

On Thu, Nov 3, 2011 at 8:52 AM, Randall Hansen <ran...@puppetlabs.com> wrote:

On Thu, Nov 3, 2011 at 8:38 AM, Nigel Kersten <ni...@puppetlabs.com> wrote:

> In your puppet.conf agent block:

> ...
>     # The maximum time to delay before runs.  Defaults to being the same as the
>     # run interval.
>     # The default value is '$runinterval'.
>     splaylimit = 1800

Having this uncommented as this value is a noop, right?

I was somewhat misleading here.

I ran "puppet agent --genconfig" to get the config, and manually uncommented the value so that it was obvious which bits were comments, and which bits weren't.

 

[Jo Rhett]

Nigel, As you've said, the time chosen for the run cycle will be consistent.  All of these settings are already set -- this isn't a question of how to change how often to run, it's how to affect the chosen runtime?

I've got an awful lot of systems (> 100) which have decided to all roll at 28 and 58 minutes after the hour.  How can I rebalance them?

[Adam Crews]

On Thu, Nov 3, 2011 at 12:41 PM, Jo Rhett <jrh...@netconsonance.com> wrote:
> Nigel, As you've said, the time chosen for the run cycle will be consistent.
>  All of these settings are already set -- this isn't a question of how to
> change how often to run, it's how to affect the chosen runtime?
> I've got an awful lot of systems (> 100) which have decided to all roll at
> 28 and 58 minutes after the hour.  How can I rebalance them?

This is exactly why I use cron instead of the daemon.
In my module that configures puppet I have:

# These are used to set when the puppet_run cron job runs.
$time1 = fqdn_rand(30)
$time2 = $time1 + 30

cron {
"puppet_run":
command => "$common::data::bindir/puppet_run.sh",
user => root,
minute => [$time1, $time2],
ensure => present,
require => File["$common::data::bindir/puppet_run.sh"];
}

The fqdn_rand gives me a consistent random number per host, and so far
this has been much better than letting the daemon try to splay things
on it's own.

[Christopher Wood]

On Thu, Nov 03, 2011 at 12:41:50PM -0700, Jo Rhett wrote:
> Nigel, As you've said, the time chosen for the run cycle will be
> consistent.  All of these settings are already set -- this isn't a
> question of how to change how often to run, it's how to affect the chosen
> runtime?

I thought the chosen run time was selected on the agent. It sounds like you'll have to reconfigure the agent to have it decide differently. So far your options seem to be, from reading this thread:

-run via puppetized fqdn_rand() cron
-use splay (if you don't mind living without puppet kick)
-change the runinterval for each puppet agent

There's possibly other cheap/dirty ways like having a cron job restart the puppet agent at an interval shorter than when they converge, or setting an at job to do it, or similar. Perhaps you could use a cronjob on the puppet master to check logs for run times and kick the groups that are teaming up. And so forth.

> I've got an awful lot of systems (> 100) which have decided to all roll at
> 28 and 58 minutes after the hour.  How can I rebalance them?
> On Nov 3, 2011, at 8:38 AM, Nigel Kersten wrote:
>

> On Thu, Nov 3, 2011 at 8:36 AM, Jo Rhett <[1]jrh...@netconsonance.com>

> To post to this group, send email to [2]puppet...@googlegroups.com.

> To unsubscribe from this group, send email to

> [3]puppet-users...@googlegroups.com.

> For more options, visit this group at

> [4]http://groups.google.com/group/puppet-users?hl=en.

>
> --
> Nigel Kersten
> Product Manager, Puppet Labs
>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.

> To post to this group, send email to [5]puppet...@googlegroups.com.

> To unsubscribe from this group, send email to

> [6]puppet-users...@googlegroups.com.

> For more options, visit this group at

> [7]http://groups.google.com/group/puppet-users?hl=en.

>
> -- 
> Jo Rhett
> Net Consonance : consonant endings by net philanthropy, open source and
> other randomness
>

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>

> References
>
> Visible links
> 1. mailto:jrh...@netconsonance.com
> 2. mailto:puppet...@googlegroups.com
> 3. mailto:puppet-users%2Bunsu...@googlegroups.com
> 4. http://groups.google.com/group/puppet-users?hl=en

> 5. mailto:puppet...@googlegroups.com
> 6. mailto:puppet-users...@googlegroups.com
> 7. http://groups.google.com/group/puppet-users?hl=en

[Ken Barber]

PuppetCommander is another option:

http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/ToolPuppetcommander

It deals with 'splay' globally instead of on a per node basis with
random sleeps.

ken.

[Jo Rhett]

On Nov 3, 2011, at 1:01 PM, Adam Crews wrote:

This is exactly why I use cron instead of the daemon.

we "puppet kick" a lot.

[Jo Rhett]

Why not document how the client decides the runtime, and how to adjust it?

[Nigel Kersten]

On Thu, Nov 3, 2011 at 12:41 PM, Jo Rhett <jrh...@netconsonance.com> wrote:

Nigel, As you've said, the time chosen for the run cycle will be consistent.  All of these settings are already set -- this isn't a question of how to change how often to run, it's how to affect the chosen runtime?

I've got an awful lot of systems (> 100) which have decided to all roll at 28 and 58 minutes after the hour.  How can I rebalance them?

This should be what the splay settings do for you Jo.

Even though those agents all roll at 28/58 minutes past the hour, if you set splay to true, they'll then wait a random amount of time up to "splaylimit" before they *actually* perform the run.

The default setting for "splay" is off, so you won't see this behavior unless you explicitly turn it on.

Does that make more sense? 

[Nigel Kersten]

On Sun, Nov 6, 2011 at 5:57 AM, Piavlo <lolit...@gmail.com> wrote:

 Hi ,

How about an option for throttling number of max puppet clients being active - then puppet client time to run has come it registers at the puppetmaster - and once there is below max allowed active puppet clients running - puppetmaster initiates the client run (puppet client needs to have "listen=true" for that to work). Priorities can also be added easily. This also gives you easy visibility on then running puppet every X minutes becomes a bottleneck.
This ca be probably done with mcollective - but requires a separate daemon process to manage the queue waiting puppet clients.
The advanate of separate daemon is that it can have a view of more than one puppetmaster.
The disadvantage is that puppet clients can still connect directly puppetmaster avoiding the queue altogether.
So it looks like a builtin puppet queue functionality is a better choice.

wdyt?

So you need a queue *somewhere* to do this, and MCollective will do this for you now with the "mco puppetd runall X" agent.

Your point about agents still being able to connect if they want to is a good one though.

I'd love it if you filed this as a feature request though Piavlo, it's important to capture this somewhere outside the mailing lists.

[Piavlo]

 Hi ,

How about an option for throttling number of max puppet clients being active - then puppet client time to run has come it registers at the puppetmaster - and once there is below max allowed active puppet clients running - puppetmaster initiates the client run (puppet client needs to have "listen=true" for that to work). Priorities can also be added easily. This also gives you easy visibility on then running puppet every X minutes becomes a bottleneck.
This ca be probably done with mcollective - but requires a separate daemon process to manage the queue waiting puppet clients.
The advanate of separate daemon is that it can have a view of more than one puppetmaster.
The disadvantage is that puppet clients can still connect directly puppetmaster avoiding the queue altogether.
So it looks like a builtin puppet queue functionality is a better choice.

wdyt?

Alex

[Jo Rhett]

On Nov 5, 2011, at 9:42 AM, Nigel Kersten wrote:

On Thu, Nov 3, 2011 at 12:41 PM, Jo Rhett <jrh...@netconsonance.com> wrote:

Nigel, As you've said, the time chosen for the run cycle will be consistent.  All of these settings are already set -- this isn't a question of how to change how often to run, it's how to affect the chosen runtime?

I've got an awful lot of systems (> 100) which have decided to all roll at 28 and 58 minutes after the hour.  How can I rebalance them?

This should be what the splay settings do for you Jo.

Even though those agents all roll at 28/58 minutes past the hour, if you set splay to true, they'll then wait a random amount of time up to "splaylimit" before they *actually* perform the run.

splay breaks many things, including puppet kick.  It simply won't work until you break splay out to only affect normal running.  I tried enabling it, and found that it simply didn't affect the systems enough.

I am deeply curious as to why you are avoiding the main question here:

   1. How does puppet decide when its 30 minutes is up?  Is this stored in a file somewhere?

   2. How can we change its calculation? (can we edit a file, run a command, …?)

[Christopher Wood]

(Without jumping too hard in the middle of your conversation... inline.)

On Tue, Nov 15, 2011 at 11:30:40AM -0800, Jo Rhett wrote:
> On Nov 5, 2011, at 9:42 AM, Nigel Kersten wrote:
>

> On Thu, Nov 3, 2011 at 12:41 PM, Jo Rhett <[1]jrh...@netconsonance.com>

> wrote:
>
> Nigel, As you've said, the time chosen for the run cycle will be
> consistent.  All of these settings are already set -- this isn't a
> question of how to change how often to run, it's how to affect the
> chosen runtime?
> I've got an awful lot of systems (> 100) which have decided to all
> roll at 28 and 58 minutes after the hour.  How can I rebalance them?
>
> This should be what the splay settings do for you Jo.
> Even though those agents all roll at 28/58 minutes past the hour, if you
> set splay to true, they'll then wait a random amount of time up to
> "splaylimit" before they *actually* perform the run.
>
> splay breaks many things, including puppet kick.  It simply won't work
> until you break splay out to only affect normal running.  I tried enabling
> it, and found that it simply didn't affect the systems enough.
> I am deeply curious as to why you are avoiding the main question here:
>    1. How does puppet decide when its 30 minutes is up?  Is this stored in
> a file somewhere?

Grep shows me (on Debian Stable, puppet 2.6.2) that agent.rb sets up an event loop:

/usr/lib/ruby/1.8/puppet/agent.rb
/usr/lib/ruby/1.8/puppet/external/event-loop/event-loop.rb

>    2. How can we change its calculation? (can we edit a file, run a
> command, �?)

Subclass the event loop or the agent perhaps? (I'm assuming that the several existing techniques for changing agent run times are insufficient for you.)

> [3]puppet...@googlegroups.com.

> To unsubscribe from this group, send email to

> [4]puppet-users...@googlegroups.com.

> For more options, visit this group at

> [5]http://groups.google.com/group/puppet-users?hl=en.

>
> --
> Nigel Kersten
> Product Manager, Puppet Labs
>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To post to this group, send email to

> [6]puppet...@googlegroups.com.

> To unsubscribe from this group, send email to

> [7]puppet-users...@googlegroups.com.

> For more options, visit this group at

> [8]http://groups.google.com/group/puppet-users?hl=en.

>
> -- 
> Jo Rhett
> Net Consonance : consonant endings by net philanthropy, open source
> and other randomness
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.

> To post to this group, send email to [9]puppet...@googlegroups.com.

> To unsubscribe from this group, send email to

> [10]puppet-users...@googlegroups.com.

> For more options, visit this group at

> [11]http://groups.google.com/group/puppet-users?hl=en.

>
> --
> Nigel Kersten
> Product Manager, Puppet Labs
>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.

> To post to this group, send email to [12]puppet...@googlegroups.com.

> To unsubscribe from this group, send email to

> [13]puppet-users...@googlegroups.com.

> For more options, visit this group at

> [14]http://groups.google.com/group/puppet-users?hl=en.

>
> -- 
> Jo Rhett
> Net Consonance : consonant endings by net philanthropy, open source and
> other randomness
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>

> References
>
> Visible links
> 1. mailto:jrh...@netconsonance.com

> 2. mailto:jrh...@netconsonance.com
> 3. mailto:puppet...@googlegroups.com
> 4. mailto:puppet-users%2Bunsu...@googlegroups.com
> 5. http://groups.google.com/group/puppet-users?hl=en
> 6. mailto:puppet...@googlegroups.com
> 7. mailto:puppet-users...@googlegroups.com
> 8. http://groups.google.com/group/puppet-users?hl=en
> 9. mailto:puppet...@googlegroups.com
> 10. mailto:puppet-users%2Bunsu...@googlegroups.com
> 11. http://groups.google.com/group/puppet-users?hl=en
> 12. mailto:puppet...@googlegroups.com
> 13. mailto:puppet-users...@googlegroups.com
> 14. http://groups.google.com/group/puppet-users?hl=en