Puppet can't successfully restart puppetd

[Matthew Burgess]

Hi,

I have a puppet module which is relatively simple; it just defines a
couple of packages (puppet & facter) and a couple of files
(/etc/puppet/puppet.conf, /etc/puppet/namespaceauth.conf and
/etc/puppet/auth.conf) and a service (puppet), which is subscribed to
the puppet package and the 3 config files.

I changed auth.conf and then witnessed all of the puppet agents pick
the apply the change, then got this in the logs:

puppet-agent[6139]: Caught TERM; calling stop

At this point, the puppet agent has obviously died, and I have to ssh
on to the boxes to kick it back off again. Other services seem to
restart fine on a config file change; is there something special about
the puppet service that it issuing the equivalent of 'service puppet
restart' doesn't actually work? I'm using puppet-2.7.17 in case it
matters.

Thanks,

Matt.

[Felix Frank]

On 06/27/2012 10:51 AM, Matthew Burgess wrote:
> is there something special about
> the puppet service that it issuing the equivalent of 'service puppet
> restart' doesn't actually work?

Oh, it works, but think of what you're having puppet do: By calling the
initscript and telling it to restart, it first has to "stop puppet",
i.e., send TERM to the agent process. So the agent is committing suicide
mid-run, taking the invocation of the initscript with it. Now nobody's
there to start puppet up again.

I've worked around this using exec { "echo service puppet restart | at
now+5min" } in the past, but this is quite the hack. Personally I have
found the cron model much more convenient that the long-running agent.

HTH,
Felix

[Matthew Burgess]

On Wed, Jun 27, 2012 at 10:01 AM, Felix Frank
<felix...@alumni.tu-berlin.de> wrote:
> On 06/27/2012 10:51 AM, Matthew Burgess wrote:
>> is there something special about
>> the puppet service that it issuing the equivalent of 'service puppet
>> restart' doesn't actually work?
>
> Oh, it works, but think of what you're having puppet do: By calling the
> initscript and telling it to restart, it first has to "stop puppet",
> i.e., send TERM to the agent process. So the agent is committing suicide
> mid-run, taking the invocation of the initscript with it. Now nobody's
> there to start puppet up again.

Yeah, I did think of that, but I assumed that the 'service puppet
restart' command had been invoked already, i.e. before the puppet
agent had been killed. Obviously, if the restart of the service is
being handled by two separate commands, e.g. 'service puppet stop'
followed by 'service puppet start' then yes, I completely understand
how I'm in the situation I'm now in.

> I've worked around this using exec { "echo service puppet restart | at
> now+5min" } in the past, but this is quite the hack. Personally I have
> found the cron model much more convenient that the long-running agent.

I'm using the long-running agent so that things like 'puppet kick'
work from the master, but that too could be implemented by using
passwordless ssh keys and just sshing from the master to the managed
node and kicking off the agent that way, I guess. I'll give the 'at'
variation a try first though. Thanks for the hint!

Regards,

Matt.

[Martin Alfke]

We redefined the restart command:

restart => '/usr/bin/nohup /etc/init.d/puppet restart &'

- Martin

[Matthew Burgess]

On Wed, Jun 27, 2012 at 10:13 AM, Martin Alfke <tux...@gmail.com> wrote:

> We redefined the restart command:
>
> restart => '/usr/bin/nohup /etc/init.d/puppet restart &'

Nice! Thanks very much, Martin, that works really nicely.

Regards,

Matt.

[Felix Frank]

On 06/27/2012 11:19 AM, Matthew Burgess wrote:
>> We redefined the restart command:
>> >
>> > restart => '/usr/bin/nohup /etc/init.d/puppet restart &'
> Nice! Thanks very much, Martin, that works really nicely.

Yeah, this looks really slick. Thanks for sharing.

[Steve Traylen]

On Wednesday, 27 June 2012 11:31:17 UTC+2, Felix.Frank wrote:

On 06/27/2012 11:19 AM, Matthew Burgess wrote:
>> We redefined the restart command:
>> >
>> > restart => '/usr/bin/nohup /etc/init.d/puppet restart &'
> Nice!  Thanks very much, Martin, that works really nicely.

Before I blindly try this how does the above work... 

I call this midway through a puppet run and it does no harm to the current run but 

loads new configuration for next run. Is that what happens? 

Steve. 

[Martin Alfke]

On 03.07.2012, at 21:31, Steve Traylen wrote:

On Wednesday, 27 June 2012 11:31:17 UTC+2, Felix.Frank wrote:

On 06/27/2012 11:19 AM, Matthew Burgess wrote:
>> We redefined the restart command:
>> >
>> > restart => '/usr/bin/nohup /etc/init.d/puppet restart &'
> Nice!  Thanks very much, Martin, that works really nicely.

Before I blindly try this how does the above work... 

I call this midway through a puppet run and it does no harm to the current run but 

loads new configuration for next run. Is that what happens? 

No.

It will stop the actual puppet run.

Then it will again start puppet daemon and initiate a new run.

(restart is short hand for stop and start).

We do puppet configuration in an early stage.

So we are sure that we use actual puppet configuration when entering the main stage.

- Martin

[Edward Savage]

Late reply, however if you're using Ubuntu 10.04 or greater you can
avoid the nohup:

hasrestart => true;

Edward

[朱文]

thx,

the master is Centos 5.8

the client   is  RedHat 5.4

2012/7/4 Edward Savage <eps...@gmail.com>

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

--
 如有未尽事宜，请及时联系。分机-8752


-----
朱文  PeterZhu

集团技术部  OPS
----------------
办公电话：021-61821155-8753
移动手机：15000654460
办公传真：021-61821150

 
上海市浦东新区陆家嘴环路166号未来资产大厦10楼 安居客集团

[Throwe, Jesse]

Here is how we resolved this problem (although we are now running puppet through cron).  By issuing the HUP puppet will restart, but not until the run has completed.  This is largely for bootstrapping or when the machine environment changes.

If your running puppetd change 'puppet agent' to 'puppetd'.  At least if for the releases you listed, I would move away from running daemonized puppet unless theres a special requirement/reason, and stick it in a cron instead.

== Begin CopyPasta ==

exec { "restart-puppet":

command => "ps -ef | grep 'puppet agent' | grep -v grep | awk {'print \$2'} | xargs -iXX echo kill -HUP XX | sh",

path => "/sbin:/bin:/usr/sbin:/usr/bin",

refreshonly=>true

}

file { "/etc/puppet/puppet.conf":

notify => Exec["restart-puppet"]

}

== End CopyPasta ==

[Felix Frank]

On 07/04/2012 03:58 PM, Throwe, Jesse wrote:
> If your running puppetd change 'puppet agent' to 'puppetd'. At least if
> for the releases you listed, I would move away from running daemonized
> puppet unless theres a special requirement/reason, and stick it in a
> cron instead.

+1

> command => "ps -ef | grep 'puppet agent' | grep -v grep | awk {'print
> \$2'} | xargs -iXX echo kill -HUP XX | sh",

*cringe* I advise the use of pgrep to retrieve pids.

Also, why not pipe to "xargs kill -HUP" directly?

[Throwe, Jesse]

Non homogeneous environment. If I recall we had an issue with a
simpler command like you suggested with old versions of Solaris.

[Felix Frank]

On 07/04/2012 04:18 PM, Throwe, Jesse wrote:
> Non homogeneous environment. If I recall we had an issue with a
> simpler command like you suggested with old versions of Solaris.

Oh, the S-word. Well, that *does* indeed answer to both remarks.
Condolences. ;-)