Enable puppet agent by default

[Marc]

Hello 

I am trying puppet 3.5.1 on a Debian Jessie.

How can I enable Puppet by default on new installations ?

I need to do that, for Debian deployment. I use Debian FAI to install Debian on my workstations. After the deployment, Puppet is launched. I just have to sign them on Puppet Master in order to launch the post install with puppet (AD integration, CUPS configuration...).

So now, I can't do that because the /etc/default/puppet file is not read by the system.

The only solution I see, is to add a cron task @reboot. Am I wrong ?

Thanks for your help

Marc

[Matt Zagrabelny]

Hi!

On Tue, Apr 29, 2014 at 9:13 AM, Marc <marc...@gmail.com> wrote:
> Hello
>
> I am trying puppet 3.5.1 on a Debian Jessie.
>
> How can I enable Puppet by default on new installations ?

It looks like it is.

> I need to do that, for Debian deployment. I use Debian FAI to install Debian
> on my workstations. After the deployment, Puppet is launched. I just have to
> sign them on Puppet Master in order to launch the post install with puppet
> (AD integration, CUPS configuration...).
>
> So now, I can't do that because the /etc/default/puppet file is not read by
> the system.

I don't think there is any need to read /etc/default/puppet on Debian
Jessie. The Jessie puppet package does not install /etc/default/puppet
and the init script doesn't make use of anything in the file either -
thought it does make an attempt to read it:

[ -r /etc/default/puppet ] && . /etc/default/puppet

> The only solution I see, is to add a cron task @reboot. Am I wrong ?

I don't follow what you want to have done. What exactly would you put
in the cron task?

-mz

[Garrett Honeycutt]

Hi,

Recommend that you do not run the agent as a service at all and instead
use cron. You can then have each system randomize when they check in to
prevent thundering herd[1]. I still run the puppet agent by default at
boot time which is handled as you mentioned by cron with @reboot. If you
bring up all your systems at the same time, this could pose an issue for
you.

Here my puppet module[2] that can manage your agent with cron or as a
service.

[1] - http://en.wikipedia.org/wiki/Thundering_herd_problem
[2] - https://github.com/ghoneycutt/puppet-module-puppet

Best regards,
-g

--
Garrett Honeycutt
@learnpuppet
Puppet Training with LearnPuppet.com
Mobile: +1.206.414.8658

[Christopher Wood]

(inline)

Can't hosts already stagger their agent checkin times by using per-host runinterval settings?

At some point, sure, agents may not be the best path forward but I don't see when I'd reach that point.

> Here my puppet module[2] that can manage your agent with cron or as a
> service.
>
> [1] - http://en.wikipedia.org/wiki/Thundering_herd_problem
> [2] - https://github.com/ghoneycutt/puppet-module-puppet
>
> Best regards,
> -g
>
> --
> Garrett Honeycutt
> @learnpuppet
> Puppet Training with LearnPuppet.com
> Mobile: +1.206.414.8658
>

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/535FC0D2.1090807%40garretthoneycutt.com.
> For more options, visit https://groups.google.com/d/optout.

[jcbollinger]

On Tuesday, April 29, 2014 10:15:35 AM UTC-5, Christopher Wood wrote:

Can't hosts already stagger their agent checkin times by using per-host runinterval settings?

No.  Different agents with different runintervals will still all hit the server at nearly the same time when they are started together, and they will do so again periodically thereafter (just not every run).  Moreover, it's nasty to use a policy knob such as runinterval to address a technical issue such as avoiding a thundering herd effect.

Puppet does have the 'splay' and 'splaylimit' configuration settings as a possible solution, however.  If you can accept some variation in the interval between one agent run and the next then those are pretty effective, albeit non-deterministic.

 

At some point, sure, agents may not be the best path forward but I don't see when I'd reach that point.

I'm uncertain whether by "agents" you mean running the agent as a service, or whether you mean using the agent at all (as opposed to using "puppet apply").  Garrett was not suggesting the latter; he was suggesting using cron to schedule runs of the agent in non-daemon mode.  You can also schedule runs of "puppet apply" that way, but that's a whole different ball game.

There is a lot to be said for scheduling the agent via cron.  In addition to possible applications in load leveling, it can make Puppet more resilient.  For example, I was recently working on the provider of a custom type, and I managed to let a broken version escape to some of my systems, where it crashed the agent daemon.  I had to manually restart the daemons on those systems.  If I were launching Puppet from cron then the Puppet runs would still have failed, but the next runs, when a fixed version of the provider was available, would have gone fine without any manual intervention.

In addition, where Puppet manages its own configuration, launching it via cron is much cleaner.  Not necessary, as I understand it, but cleaner and therefore less open to unexpected behavior.  In the past, people have launched Puppet that way also to avoid creeping memory usage, though I think those memory problems are solved in any recent Puppet.

On the other hand, the system service infrastructure is a bit more natural and simpler to use than is cron.  Nevertheless, for my own site I have vague plans to switch from running Puppet as its own service to launching it via cron, at some indefinite point in the future.


John

[José Luis Ledesma]

There is another option, mcollective.

Using cron is somewhat manual, and you have to determine whenever to run puppet to avoid a thundering herd effect.

Mcollective lets you use the

Mco puppet runall <concurrency>

Where the concurrency sets how many agents run at the same time.

It is resielent as cron is, and also you have a central point of puppet agents management, instead of (semi) manually set cron entries.

Regards

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/c0aa9f0c-7730-481e-b026-8ac80df43a5f%40googlegroups.com.

[Garrett Honeycutt]

On 4/30/14, 12:49 PM, José Luis Ledesma wrote:
> There is another option, mcollective.
>
> Using cron is somewhat manual, and you have to determine whenever to run
> puppet to avoid a thundering herd effect.
>
> Mcollective lets you use the
>
> Mco puppet runall <concurrency>
>
> Where the concurrency sets how many agents run at the same time.
>
> It is resielent as cron is, and also you have a central point of puppet
> agents management, instead of (semi) manually set cron entries.
>
> Regards

Hi,

TL;DR: you can totally use cron and not have to think about scheduling
your agents.


MCollective is great for doing ad-hoc runs. If you want puppet to run at
a regular interval, then you would need to run the agent as a service or
with cron or run mcollective with either of those two things which would
achieve the same end.

Using cron is not at all manual and you do not have to determine when to
run your agents to avoid thundering herd. A common way to do this is to
use fqdn_rand(). Here's an example[1] of my implementation of this.

MCollective is not as resilient as cron if your system looses network
connectivity or has a lot of latency such as satellite connections,
though that is likely not an issue for most folks. It does give you a
central point of management for your agents which can be awesome. Some
folks run puppet only in noop mode and only do enforcing runs on demand
with MCollective, though that depends on your change management needs.

[1] -
https://github.com/ghoneycutt/puppet-module-puppet/blob/master/manifests/agent.pp#L110-111

BR,

[Christopher Wood]

(inline)

On Wed, Apr 30, 2014 at 08:21:15AM -0700, jcbollinger wrote:
> On Tuesday, April 29, 2014 10:15:35 AM UTC-5, Christopher Wood wrote:
>
> Can't hosts already stagger their agent checkin times by using per-host
> runinterval settings?
>
> No.  Different agents with different runintervals will still all hit the
> server at nearly the same time when they are started together, and they
> will do so again periodically thereafter (just not every run).  Moreover,
> it's nasty to use a policy knob such as runinterval to address a technical
> issue such as avoiding a thundering herd effect.

In theory the agent runs will intersect and kill the puppetmaster in the timespan around when the lowest common denominator of all the runintervals comes around. In practice if this ever happens to me (hasn't so far) I will shrug and say to wait for the next agent run in less than an hour. Right now my runinterval defaults to (1800 + fqdn_rand(600)), implying that any LCD intersections aren't so frequent yet the agents update pretty frequently.

Out of personal preference I don't make a distinction between technical versus policy issues; that they are both components of the same services.

> Puppet does have the 'splay' and 'splaylimit' configuration settings as a
> possible solution, however.  If you can accept some variation in the
> interval between one agent run and the next then those are pretty
> effective, albeit non-deterministic.

I abandoned splay use when it interfered with 'puppet kick'. I probably wouldn't use it these days because it's not obvious how long the splay will wait and I'm trying to get away from using inferences in preference to literal values. I sometimes don't infer the same results as other people do.

https://groups.google.com/forum/#!topic/puppet-users/EaoiHSd4eEM
http://projects.puppetlabs.com/issues/1100

>  
>
> At some point, sure, agents may not be the best path forward but I don't
> see when I'd reach that point.

(agents as in puppet agent daemons/services)

> I'm uncertain whether by "agents" you mean running the agent as a service,
> or whether you mean using the agent at all (as opposed to using "puppet
> apply").  Garrett was not suggesting the latter; he was suggesting using
> cron to schedule runs of the agent in non-daemon mode.  You can also
> schedule runs of "puppet apply" that way, but that's a whole different
> ball game.

This gets back to, how do I schedule my agent runs to avoid puppetmaster service issues due to load? There are only so many cron slots in a day, manually scheduling that many machines could get boring, and doing it automatically could lead to the same herd issues if I get the algorithm wrong. The answer sounds like it might be a control host kicking off an "mco puppet runall $limit" job via cron but I'm not there yet.

> There is a lot to be said for scheduling the agent via cron.  In addition
> to possible applications in load leveling, it can make Puppet more
> resilient.  For example, I was recently working on the provider of a
> custom type, and I managed to let a broken version escape to some of my
> systems, where it crashed the agent daemon.  I had to manually restart the
> daemons on those systems.  If I were launching Puppet from cron then the
> Puppet runs would still have failed, but the next runs, when a fixed
> version of the provider was available, would have gone fine without any
> manual intervention.

This is where I plug how you should already be killing your own daemons so that you can build service resilience via some form of watcher, and doing dev->stage->prod incremental rollouts so that most of your problems don't happen in production. In my case monit would have brought up puppet on the affected dev hosts, and those hosts would have grabbed the new provider as soon as it was available on the puppetmasters. (Obviously the first time I tried this technique I learned the hard way about monit supervising sshd and init supervising monit.)

# Puppet Name: /usr/local/sbin/monkey
4 * * * * /usr/local/sbin/monkey 10 /bin/ps aux 1

pidstokill.each do |target|
Process.kill('KILL', target)
end

# cat /etc/monit.d/puppet
check process puppet with pidfile /var/run/puppet/agent.pid
start program = "/etc/init.d/puppet restart"
stop program = "/etc/init.d/puppet stop"
if 5 restarts within 5 cycles then alert

> In addition, where Puppet manages its own configuration, launching it via
> cron is much cleaner.  Not necessary, as I understand it, but cleaner and
> therefore less open to unexpected behavior.  In the past, people have
> launched Puppet that way also to avoid creeping memory usage, though I
> think those memory problems are solved in any recent Puppet.
> On the other hand, the system service infrastructure is a bit more natural
> and simpler to use than is cron.  Nevertheless, for my own site I have
> vague plans to switch from running Puppet as its own service to launching
> it via cron, at some indefinite point in the future.
>
> John
>

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to [1]puppet-users...@googlegroups.com.

> To view this discussion on the web visit

> [2]https://groups.google.com/d/msgid/puppet-users/c0aa9f0c-7730-481e-b026-8ac80df43a5f%40googlegroups.com.
> For more options, visit [3]https://groups.google.com/d/optout.
>
> References
>
> Visible links
> 1. mailto:puppet-users...@googlegroups.com
> 2. https://groups.google.com/d/msgid/puppet-users/c0aa9f0c-7730-481e-b026-8ac80df43a5f%40googlegroups.com?utm_medium=email&utm_source=footer
> 3. https://groups.google.com/d/optout

[jcbollinger]

On Thursday, May 1, 2014 9:42:39 AM UTC-5, Christopher Wood wrote:

(inline)

On Wed, Apr 30, 2014 at 08:21:15AM -0700, jcbollinger wrote:
>    On Tuesday, April 29, 2014 10:15:35 AM UTC-5, Christopher Wood wrote:
>
>      Can't hosts already stagger their agent checkin times by using per-host
>      runinterval settings?
>
>    No.  Different agents with different runintervals will still all hit the
>    server at nearly the same time when they are started together, and they
>    will do so again periodically thereafter (just not every run).  Moreover,
>    it's nasty to use a policy knob such as runinterval to address a technical
>    issue such as avoiding a thundering herd effect.

In theory the agent runs will intersect and kill the puppetmaster in the timespan around when the lowest common denominator of all the runintervals comes around.

Yes, subject to a fuzz factor that is correlated with how expensive your catalogs are to compile.  The master is more sensitive to load leveling problems when catalog compilation is expensive, which is exactly the condition in which balancing via a spread of runinterval is least effective.

 

In practice if this ever happens to me (hasn't so far) I will shrug and say to wait for the next agent run in less than an hour. Right now my runinterval defaults to (1800 + fqdn_rand(600)), implying that any LCD intersections aren't so frequent yet the agents update pretty frequently.

Out of personal preference I don't make a distinction between technical versus policy issues; that they are both components of the same services.

>    Puppet does have the 'splay' and 'splaylimit' configuration settings as a
>    possible solution, however.  If you can accept some variation in the
>    interval between one agent run and the next then those are pretty
>    effective, albeit non-deterministic.

I abandoned splay use when it interfered with 'puppet kick'.

That's a fair criticism, where applicable.  Personally, I don't much like 'puppet kick', and I don't configure agents to respond to it.  If I wanted something along those lines then I would set up something more general, such as MCollective, or something that leverages nodes' ordinary features, such as a script around 'ssh -c'.  YMMV.

 

I probably wouldn't use it these days because it's not obvious how long the splay will wait

Well, yes, that's the point of splay.

 

and I'm trying to get away from using inferences in preference to literal values.

Ok.  I have no problem with that.

If you follow that path, however, then doesn't it lead past fixed run intervals, on to scheduling each agent's runs at specific times?  That's even more literal, and it's independent of most timing variables (when was Puppet started, how long does each run take, etc.).

 

I sometimes don't infer the same results as other people do.

https://groups.google.com/forum/#!topic/puppet-users/EaoiHSd4eEM
http://projects.puppetlabs.com/issues/1100

>     
>
>      At some point, sure, agents may not be the best path forward but I don't
>      see when I'd reach that point.

(agents as in puppet agent daemons/services)
 
>    I'm uncertain whether by "agents" you mean running the agent as a service,
>    or whether you mean using the agent at all (as opposed to using "puppet
>    apply").  Garrett was not suggesting the latter; he was suggesting using
>    cron to schedule runs of the agent in non-daemon mode.  You can also
>    schedule runs of "puppet apply" that way, but that's a whole different
>    ball game.

This gets back to, how do I schedule my agent runs to avoid puppetmaster service issues due to load? There are only so many cron slots in a day, manually scheduling that many machines could get boring, and doing it automatically could lead to the same herd issues if I get the algorithm wrong.

You can't be serious.  Why on earth would anyone consider doing it manually?  If you are going to schedule Puppet runs via cron, then you use fqdn_rand() or something similar to set the schedules.  The likelihood of fqdn_rand() producing a problematically non-uniform distribution is minute, and it is anyway equally applicable to setting run intervals.

 

The answer sounds like it might be a control host kicking off an "mco puppet runall $limit" job via cron but I'm not there yet.

That's the general direction in which I might turn for an alternative to 'puppet kick', but I don't like it for scheduling regular agent runs.  For one thing, although it caps the load on the master, it doesn't do a good job of balancing unless you tune it carefully, and keep it in tune.

 

>    There is a lot to be said for scheduling the agent via cron.  In addition
>    to possible applications in load leveling, it can make Puppet more
>    resilient.  For example, I was recently working on the provider of a
>    custom type, and I managed to let a broken version escape to some of my
>    systems, where it crashed the agent daemon.  I had to manually restart the
>    daemons on those systems.  If I were launching Puppet from cron then the
>    Puppet runs would still have failed, but the next runs, when a fixed
>    version of the provider was available, would have gone fine without any
>    manual intervention.

This is where I plug how you should already be killing your own daemons so that you can build service resilience via some form of watcher, and doing dev->stage->prod incremental rollouts so that most of your problems don't happen in production. In my case monit would have brought up puppet on the affected dev hosts, and those hosts would have grabbed the new provider as soon as it was available on the puppetmasters. (Obviously the first time I tried this technique I learned the hard way about monit supervising sshd and init supervising monit.)

That's all good advice, but monit would not be relevant for puppet if I weren't running the agent as a daemon in the first place.  Since I don't want 'puppet kick', I have no reason to do so other than scheduling, which cron can achieve for me just fine, yielding also resiliency benefits comparable to what monit could give me (for Puppet) and other benefits.

John

[Christopher Wood]

I will have to keep that in mind. So far I haven't loaded the puppetmaster hosts too much but that could happen.

> In practice if this ever happens to me (hasn't so far) I will shrug and
> say to wait for the next agent run in less than an hour. Right now my
> runinterval defaults to (1800 + fqdn_rand(600)), implying that any LCD
> intersections aren't so frequent yet the agents update pretty
> frequently.
>
> Out of personal preference I don't make a distinction between technical
> versus policy issues; that they are both components of the same
> services.
>
> >    Puppet does have the 'splay' and 'splaylimit' configuration
> settings as a
> >    possible solution, however.  If you can accept some variation in
> the
> >    interval between one agent run and the next then those are pretty
> >    effective, albeit non-deterministic.
>
> I abandoned splay use when it interfered with 'puppet kick'.
>
> That's a fair criticism, where applicable.  Personally, I don't much like
> 'puppet kick', and I don't configure agents to respond to it.  If I wanted
> something along those lines then I would set up something more general,
> such as MCollective, or something that leverages nodes' ordinary features,
> such as a script around 'ssh -c'.  YMMV.

I don't have any responding to kick either, but I don't really have any reason to revisit splay use what with the open bug and all.

> I probably wouldn't use it these days because it's not obvious how long
> the splay will wait
>
> Well, yes, that's the point of splay.
>
>  
>
> and I'm trying to get away from using inferences in preference to
> literal values.
>
> Ok.  I have no problem with that.
>
> If you follow that path, however, then doesn't it lead past fixed run
> intervals, on to scheduling each agent's runs at specific times?  That's
> even more literal, and it's independent of most timing variables (when was
> Puppet started, how long does each run take, etc.).

If I start running into trouble with the current setup, sure. At scale I wind up with interesting scheduling issues, for example that fqdn_rand doesn't seem to distribute things evenly (just like with round-robin load balancing), and my number of hosts may not fit tidily into cron, and managing schedules for a respectable number of hosts stops being trivial. (Those are of course functions of my skill level.) Consider, with fictional numbers:

for x in `seq 4000 6999`; do echo "notice(fqdn_rand(60, ${x}))"; done >/tmp/xx.pp
puppet apply --color=false /tmp/xx.pp | grep Scope | awk '{print $3}' | sort | uniq -c | sort -rn >/tmp/y
head -5 /tmp/y
tail -5 /tmp/y

There are obviously ways around all those, but they seem to create a lot of work for much the same result. I'm not so concerned about individual agent runs as I am that everything stays in sync over the course of a day.

> I sometimes don't infer the same results as other people do.
>

> [1]https://groups.google.com/forum/#!topic/puppet-users/EaoiHSd4eEM
> [2]http://projects.puppetlabs.com/issues/1100

Not manually as in me with my spreadsheet, manually as in something requiring more supervision and human time. Team A wants their agent runs before 03:00, team B wants them at 09:00, et cetera.

Automatically as in making it happen by itself, for cheap example:

list of hosts from puppetdb
arbitrary time interval (86400 seconds)
desired time between agent runs
algorithm to schedule them given the above
scheduling is recalculated when somebody adds/removes a host

This is where I start furrowing my brow. Perhaps I'm overthinking.

(I know I keep harping on all the hard work, but chasing after minutiae takes its toll. I hear about the Cloudflare/Whatsapp people:result ratio and get jealous.)

Given the chaos monkey I've seen a puppet agent killed and the agent-run child still alive hours later, but I take your point.

> John
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to [3]puppet-users...@googlegroups.com.

> To view this discussion on the web visit

> [4]https://groups.google.com/d/msgid/puppet-users/d8744267-58bc-44a1-b496-7d0f140ee773%40googlegroups.com.
> For more options, visit [5]https://groups.google.com/d/optout.
>
> References
>
> Visible links
> 1. https://groups.google.com/forum/#!topic/puppet-users/EaoiHSd4eEM
> 2. http://projects.puppetlabs.com/issues/1100
> 3. mailto:puppet-users...@googlegroups.com
> 4. https://groups.google.com/d/msgid/puppet-users/d8744267-58bc-44a1-b496-7d0f140ee773%40googlegroups.com?utm_medium=email&utm_source=footer
> 5. https://groups.google.com/d/optout