MCollective/Puppet - one-time run with other options

[Jason Antman]

Hello,

I've recently learned that my plans to use the puppet agent mcollective
plugin to trigger one-time runs against a different environment, with
the daemon running in the background, don't work because of how the
agent plugin works (SIGUSR1 to the daemon if running).

My original intent was as follows:
I have a bunch of puppet nodes dedicated to testing new
manifests/modules. Normally they, like all of my other nodes, run in
daemon mode against the production environment. I have a Jenkins job
that does some static testing of a specific branch of our puppet repo,
and then checks out that branch as an environment on the master and
(should/tries to) run `mco puppet runonce --environment <branchname>`.

So, while I think the "runonce" name is pretty misleading if it can't
handle running when there's a daemon, I understand the limitations
behind it. Now I'm looking for any suggestions on how to achieve what
I'm trying to. I've been able to come up with a few options... if anyone
has other suggestions, or opinions, please pass them along...

1) https://tickets.puppetlabs.com/browse/PUP-1319 (formerly redmine
5153) "Ability to run --onetime in the background while a daemon is
idle" implies that running "--onetime --no-deamonize --foreground" works
fine while there's a backgrounded daemon. So all that would be needed is
a change to the mco puppet plugin to explicitly add an option to allow
this? I manually run "puppet agent -t --environment foo" all the time
while the daemon is running, and it works fine.

2) As suggested by chris spence, I could stop using daemon mode and
start using cron to trigger periodic runs. I've been running in daemon
mode for a loooong time, but I guess with the deprecation of `puppet
kick`, it's no longer needed. The remaining issue is how to spread out
runs of all my nodes to minimize load on the master, and how to run at
boot, which are solvable problems though more complex than "service
puppet enable true". Any suggestions for how to spread out the runs? I'm
using a custom ENC, so I suppose I could do it there as a param for
cron, but I'm open to nicer solutions.

3) Going by R. I.'s blog post from 2010 (yeah a bit dated)
http://www.devco.net/archives/2010/03/17/scheduling_puppet_with_mcollective.php
about puppetcommander.rb, I could use something mco-based to schedule
the runs. Though AFAIK this means running *that* controller either via
cron on the master, or as a daemon somewhere (I assume the former would
be preferable and easier).

Any thoughts/suggestions?

My main concerns are:
1) running 300+ nodes every 30 minutes, with load on the master
distributed as evenly as possible.
2) Jenkins being able to force a given node or list of nodes to run
immediately against an arbitrary environment.

Thanks for any suggestions/feedback,
J Antman

[Jason Antman]

I also forgot the scariest option, which seems apt to break things:
- have mcollective stop the daemon
- mco puppet runonce
- have mcollective start the daemon back up

-jason

[Jose Luis Ledesma]

Hello,

   We have puppet only in the lab's servers, we are planning to deploy to production in the near future. I found myself thinking about the same question some time ago.

   What we were thinking is, why to run puppet agent on every node? In fact puppetlabs says about setting the puppet agent in the crontab... What we have done in the laboratory is to disable puppet agent, and run it always from the puppetmaster/mco-client crontab  in the next way:

0 * * * * /usr/bin/mco puppet runonce --noop --splaylimit 900 

    I don't know if it's the best solution for productive-environment, opinions?

regards, 

[Cristian Falcas]

Hi,

At my work place we have puppet stopped completely. We only run it via
mco when we update the manifests. But this implies that you don't use
exported resources.

regards,
Cristian Falcas

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/cebb4ac0-7e75-416b-bbed-3cacdbcc9542%40googlegroups.com.
>
> For more options, visit https://groups.google.com/groups/opt_out.

[Jason Antman]

Christian - we don't use puppet just to "make changes", we use it as
(IMO) it was intended - to prevent divergence. We want it running every
30 minutes to make sure things stay how the manifests/modules say.

Jose - I think that may be the way I go, but I'm going to see if I can
get something (perhaps RIP's puppetcommander, or something like it) to
evenly balance the nodes.

Thanks,
Jason

On 12/31/2013 09:32 AM, Cristian Falcas wrote:
> Hi,
>
> At my work place we have puppet stopped completely. We only run it via
> mco when we update the manifests. But this implies that you don't use
> exported resources.
>
> regards,
> Cristian Falcas
>
> On Tue, Dec 31, 2013 at 3:32 PM, Jose Luis Ledesma
> <joseluis...@gmail.com> wrote:
>> Hello,
>>
>> We have puppet only in the lab's servers, we are planning to deploy to
>> production in the near future. I found myself thinking about the same
>> question some time ago.
>>
>> What we were thinking is, why to run puppet agent on every node? In fact
>> puppetlabs says about setting the puppet agent in the crontab... What we
>> have done in the laboratory is to disable puppet agent, and run it always
>> from the puppetmaster/mco-client crontab in the next way:
>>
>> 0 * * * * /usr/bin/mco puppet runonce --noop --splaylimit 900
>>
>>
>> I don't know if it's the best solution for productive-environment,
>> opinions?
>>
>> regards,
>>
>>

>> El martes, 31 de diciembre de 2013 13:50:11 UTC+1, Jason Antman escribi�:

[Jose Luis Ledesma]

What do you mean by balancing nodes?

[R.I.Pienaar]

----- Original Message -----
> From: "Jason Antman" <ja...@jasonantman.com>
> To: puppet...@googlegroups.com
> Sent: Tuesday, December 31, 2013 4:33:43 PM
> Subject: Re: [Puppet Users] MCollective/Puppet - one-time run with other options
>
> Christian - we don't use puppet just to "make changes", we use it as
> (IMO) it was intended - to prevent divergence. We want it running every
> 30 minutes to make sure things stay how the manifests/modules say.
>
> Jose - I think that may be the way I go, but I'm going to see if I can
> get something (perhaps RIP's puppetcommander, or something like it) to
> evenly balance the nodes.

puppetcommander isn't viable at this point as it doesn't support the new
puppet agent, however the code that implements 'mco puppet runall' could
easily be used in your own code if you wrap that into a little daemon or
cron script

[Jose Luis Ledesma]

The problem about using runall/concurrency is that you cannot know when it will finish, so setting it on the crontab could be a source of a problems if it has not yet finished when the next run is schedulee

[R.I.Pienaar]

----- Original Message -----
> From: "Jose Luis Ledesma" <joseluis...@gmail.com>
> To: puppet...@googlegroups.com
> Sent: Tuesday, December 31, 2013 4:51:04 PM
> Subject: Re: [Puppet Users] MCollective/Puppet - one-time run with other options
>

> The problem about using runall/concurrency is that you cannot know when it
> will finish, so setting it on the crontab could be a source of a problems if
> it has not yet finished when the next run is schedulee

You could just use a lock file and not run then. Or wait or whatever.

There are 3 variables:

- how many nodes are running
- how often they run
- how long to sleep between them

you cannot optimise all 3 these variables, the old puppetcommander would skip
nodes that it cannot run in the time so it would always finish in the allocated
time. This had the draw back that if you miss-specify your max run time or just
can't be bothered to monitor it over time as you added nodes you can end up with
nodes that just never run.

the new runall algorithm will run all nodes as quick as it can, but cannot guarantee
it will finish within your half an hour. This is the better algo since all your nodes
will run and your master will not be DOS'd. But the important thing is that all nodes
do get a run.

So now you have to just decide what is important for you. If you would like to know
when your concurrency and run times are such that it can't finish in 30 minutes given
your master resources just make your cronjob write a lock file and notify when the
next one finds a lock - you should then look why, maybe your manifests have gotten
so slow that it cant possibly finish anymore.

Or just run the thing in a loop forever let it make best efforts without killing your
master - in this case a simple daemon will work.

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/cff5bbd6-c856-44c7-836c-9e98355eb2c7%40googlegroups.com.

[Jason Antman]

Thanks for the lengthy explanation. I suppose that, for one thing, I
didn't understand the algorithm that was used in puppetcommander. The
one thing that will complicate this for me is my horribly poor ruby
skills, coupled with the fact that the MCo python bindings appear to be
abandoned (though I found what appears to be a not-yet-complete
replacement for them at https://github.com/rafaduran/python-mcollective).

Understood about the 3 variables, and I can more or less pick 2 of 3. At
the moment I'm running in daemon mode with splay = false, so I'm not
terribly worried about concurrency (as a matter of fact, my test
environment has 25 nodes and 21 of them appear to be running within the
same minute); my end goal is to have all my nodes running every 30
minutes and be as evenly dispersed as practical within that timeframe.

For the time being, as I have some more pressing issues, I'm going to
give up on the "evenly dispersed" part, and opt to use runall with a
concurrency of 10, and revisit this when my infrastructure grows to the
point that it becomes a problem (or when I finally pull the ticket to
fix this). I just did a "mco puppet runall 10" in my 25-node test
environment, and the mco command completed in 57 seconds. The puppet
master looked perfectly happy with that.

Thanks for all of the advice.

-Jason