Puppet and pushing changes

[Ohad Levy]

Hi All,

I've been trying to address the issue of when (once in a while) you need to push something to a client now, cant wait 30 minutes, or its a one time action.

The first path I went though was though was to consider something like puppetrun, and as we don't run puppetd as a daemon (running it from cron due to large memory consumption while idle), I went forward and implemented a basic daemon which uses puppet certificates (no need for ssh or anything else) infrastructure to kick in puppetd --onetime.

nevertheless, I realized that this would not solve all of my problems, many times, one needs to perform an action once, a good example could be package updates, or maybe you have an application "downtime" where you want to restart a service, or just reboot the server.
another good reason against it, is when you have a fairly large amount of nodes, calling all of your clients to recompile their configuration in a short time frame, is usually leading to a puppetmaster which is unable to perform, using tags for this special onetime puppetd run can help with file requests, but it is still something that the puppetmaster usually cant handle.

until now, I was using puppet for deployment, configuration, and making sure that everything stays that way, but I'm still using ssh for everything else, I wasn't too happy about it, so I had a look on func, which also seems very nice, has its own certificate (even due you could reuse the puppet certificates) and a small daemon that you can run commands / modules though, but this requires yet another daemon, another language (in this case its python  / shell and not puppet dsl) etc... which leads me to my question:

Why Puppet cant have a push directive as well? obviously we already have the infrastructure to connect from the server to the clients (e.g. puppetrun, or my xinetd version), we already have all of the puppet types, so it should be easy to execute on a client (possible today with ralsh or puppet directly) things like:
Service (manages restarts of services)
Packages (maybe something like ensure => latest for a one time package upgrades)
Exec for custom scripts
...
and maybe additional facts that could be executed only upon request

later on, if you have storeconfig ldap or just pure yaml node files, you can add grouping and other nice things....

Cheers,
Ohad

[Kyle Cordes]

Ohad Levy wrote:
> puppetrun, and as we don't run puppetd as a daemon (running it from cron
> due to large memory consumption while idle), I went forward and

Here is a semi-related thought.

Without regard for memory consumption, but rather due to the notion that
system-wide, manageable, canonical mechanisms are better than
app-specific mechanisms, I would prefer that Puppet did not have a
daemon mode, scheduler, sleep-N-minutes mechanism etc., at all; but
rather that the Puppet install process set it up in cron. Of course I
can set it up that way myself, but what I wish was that Puppet didn't
carry the burden of its own mechanism: no code for it, no command line
options, no init.d script, no wiki pages about it, no discussion of it, etc.

(I'm sure there is a good reason it made sense at the time, and
nonetheless I like Puppet a lot, I continue to be a fan of Puppet!)

--
Kyle Cordes
http://kylecordes.com

[Jason Rojas]

Ohad, I have been doing some thinking about this as well. I can
definitely see situations where I need a "cap shell" type of approach
to push a single change or something else, but usually those are for
administration needs.
Now feel free to correct me if I am wrong, but puppet's whole goal is
to alleviate you from having to perform those mundane tasks. Again as
I see it, if you cant wait a few minutes for changes to go out, then
well, its obviously an emergency and usually the best route is direct
access.

My next question is, is there a way you can enable listen on the
clients, define a schedule that keeps puppet from running (hack hack
hack) and then cron/schedue things on the puppetmaster side to do
puppetrun against "groups" of clients you have in a classification
tool or something.


Anyway, food for thought.

Also, whatever happened to that cobbler/puppet related tool you were
talking about a few months back?

-Jason

[Ohad Levy]

On Thu, Apr 2, 2009 at 10:35 AM, Jason Rojas <ja...@nothingbeatsaduck.com> wrote:

Ohad, I have been doing some thinking about this as well. I can
definitely see situations where I need a "cap shell" type of approach
to push a single change or something else, but usually those are for
administration needs.

sure, this are considered "emergency" or good response time to our customers.

Now feel free to correct me if I am wrong, but puppet's whole goal is
to alleviate you from having to perform those mundane tasks. Again as
I see it, if you cant wait a few minutes for changes to go out, then
well, its obviously an emergency and usually the best route is direct
access.

what happens if you have hundreds of clients that needs to  be fixed? we had an issue a few months ago where nscd got crazy due to some bad ldap / tls response, we needed to restarted nscd on all clients....

My next question is, is there a way you can enable listen on the
clients, define a schedule that keeps puppet from running (hack hack
hack) and then cron/schedue things on the puppetmaster side to do
puppetrun against "groups" of clients you have in a classification
tool or something.

I'm not really sure I understand your question, I can today use puppetrun (if running puppetd with listen option) or use my own version of mini puppet listener daemon to trigge a puppet run, but that will not solve my previous nscd example (it would make sure that nscd is running, which it is).
more than that, if you need to compile configurations for 250 clients in a couple of minutes, with the current puppetmaster implementation, thats almost impossible (e.g. you really need a lot of mongrels / cores).

Anyway, food for thought.

yep, thats the whole reason for this thread...

Also, whatever happened to that cobbler/puppet related tool you were
talking about a few months back?

Its a live n kicking, my company still <sigh> discuss which open source license is the most risk free for us...

[chakkerz]

Hello there

This might not be a solution to your problem (from a philosophical
stance anyway), but you could grant your user to run `puppetd -vt`
without password via sudoers on all hosts.
Then you could `ssh <host> puppetd -vt` from your workstation (or
write a wrapper that takes input from a list or whatever).

OR
if you're using keybased authentication allow your puppetmaster host
access to a set account from where using it's key it can only run
puppetd. Hosts.allow, firewalls etc may need to be configured to the
hosts (since so far each is pulling rather than pushing), and then you
could

for hosts in `puppetca --list --all | grep ^+ | cut -d ' ' -f 2`
do
ssh $hosts sudo puppetd -vt
done

but a push architecture is significantly more security vulnerable...
on the bright side though, if your central configuration host is
compromised, it being able to ssh to hosts is the least of your
worries (why attack individual hosts if you have the master key?)

cheers
chakkerz

[Ohad Levy]

On Thu, Apr 2, 2009 at 11:51 AM, chakkerz <chak...@gmail.com> wrote:

for hosts in `puppetca --list --all | grep ^+ | cut -d ' ' -f 2`
do
 ssh $hosts sudo puppetd -vt
done

 

sure, I know / do this, but I though that one of the goals of puppet is to avoid ssh and a for loop....
but seriously, what happens if ssh doesn't work? ( I mean, usually you need push when something is broken)

or you need to deploy something only on a subset of machines, restart a service, or whatever?

but a push architecture is significantly more security vulnerable...
on the bright side though, if your central configuration host is
compromised, it being able to ssh to hosts is the least of your
worries (why attack individual hosts if you have the master key?)

why care about the master key when you can simply change the puppet manifest ? ;)

Ohad

[Mike Renfro]

Ohad Levy wrote:

> sure, I know / do this, but I though that one of the goals of puppet
> is to avoid ssh and a for loop.... but seriously, what happens if ssh
> doesn't work? ( I mean, usually you need push when something is
> broken)
>
> or you need to deploy something only on a subset of machines, restart
> a service, or whatever?

dsh -- http://www.netfort.gr.jp/~dancer/software/dsh.html

Granted, it only works if ssh does, but it does let you shell to
pre-defined groups of systems, arbitrary lists of machines, and
optionally run them concurrently.

Some sharper person than me might work up a clean way of managing the
dsh configuration files with exported resources -- one file containing
all machines managed by puppet in /etc/dsh/machines.list and/or
/etc/dsh/group/all, another file per group of systems identified by
hostgroup (
http://reductivelabs.com/trac/puppet/wiki/Recipes/HostgroupFact ) into
/etc/dsh/group/${group}, etc.

--
Mike Renfro / R&D Engineer, Center for Manufacturing Research,
931 372-3601 / Tennessee Technological University -- ren...@tntech.edu

[Ohad Levy]

Mike,

dsh, clsh, clusterit, func, hope grown scripts  etc you name it, I think I know them all, the question remains, why shouldn't this kind of functionallity cant be achived with Puppet?
as far as I see it (and probably I dont see the whole picture) most of the internal infrastructure can support this...

am I the only one who wants to have such a feature? I really like to have one "langague/daemon/etc".

Cheers,
Ohad

[chakkerz]

I love this topic, first though:

> dsh, clsh, clusterit, func, hope grown scripts  etc you name it, I think I
> know them all, the question remains, why shouldn't this kind of
> functionallity cant be achived with Puppet?

i don't know them all but :) i'll have to check out clsh and
clusterit ... i think clsh is the one my technical lead (ab)used and
needed me to kill things in a hurry ... anyways though.

Also the mainfest was what i meant == the master key :)

I'll go a little deeper than before i think:
Push vs pull - push requires each host to be accessible to a central
service. The connection from each host will likely need to pass
through multiple barriers such as network firewalls, on host
firewalls, hosts allow (if tcpwrapped). There are also issues of
multiple security zones. One might argue that a centralized
configuration management host should be isolated and of the highest
security. Alas the powers that be have been known to keep this host(s)
on unfirewalled networks with unpatched on host firewalls.

Push is undesirable because the high cost to deploy, each firewall,
each host need to be configured to allow the connection. Which means
any migration (software or IP) may entail serious effort just for
access rights. Also push means that even though as Ohad pointed out
the manifest is the thing that will / can compromise each host, a pull
can be scheduled. Most people probably run hourly cycles. Maybe
offsetting production and development hosts (do you want a change to
appear quicker or slower on a prod host?) or having tiered solutions
with a puppet server for prod and another for dev. Regardless, pull
does give the illusion of control because each client needs to ask for
the change. So if you are diligent your entire range of hosts won't go
down the drain in one go. (we interpret cf in cfengine as cluster f**k
for a reason).

Push on the other hand implies each host gets the same change at the
same time. And even if you're tiered you can still do a lot of damage.

So i suggest this: given your need to be able affect each host
instantaneously,
1) why not have a second instance of puppet (i don't know if you can
run two on the same host though) for urgent things only, and query the
different instances at different intervals.

2) have a fact that based on time sets the urgency of your modules, so
your regular modules only trip every hour, and your urgent ones
straight away. So say have the fact check the time, and if it's 30
past the hour it does the regular changes, every minute does urgent
and every (seventh) day at midnight does the ultra rare?

regardless, tune your updates to happen every minute. The minutely
urgent modules should run quickly and all the other ones that evaluate
your fact as meaning they don't need to run should be fast, so your
urgent change should probably run alone, and after 10 minutes you just
change the urgency. I'd probably have a central file that sets the
urgency value for each module, so making the change, updating the
variable in the central file and a daemon restart later you have 1
minute to pray you didn't screw up what you're meaning to fix :).

Heck, you could even have a clever cronjob that runs on the
puppetmaster that checks the file for urgency variables, and 3 minutes
after having been touched it replaces it with the non-emergency one
and restarts puppetmaster if needed...

But implementing push ... you're playing with fire mate :)

Cheers
chakkerz

[Geoff Newell]

We use something similar. All our servers have the UCE agent installed. So we can initiate a puppetd run via UCE when required for

1. Reporting

2. Urgent updates

3. Standard updates. 

And UCE is ratified by our security bods. Sure UCE is clunky but you have to use the tools you've got. 

Geoff. 

[Marcus Vechiato]

Ohad,

Did you see func ? https://fedorahosted.org/func/

[]s

[Ohad Levy]

On Sat, Apr 4, 2009 at 1:29 PM, chakkerz <chak...@gmail.com> wrote:

I'll go a little deeper than before i think:
Push vs pull - push requires each host to be accessible to a central
service. The connection from each host will likely need to pass
through multiple barriers such as network firewalls, on host
firewalls, hosts allow (if tcpwrapped). There are also issues of
multiple security zones. One might argue that a centralized
configuration management host should be isolated and of the highest
security. Alas the powers that be have been known to keep this host(s)
on unfirewalled networks with unpatched on host firewalls.

True, but I would still argue that a fairly large amount of nodes does not have this limitation, or that SSL traffic is secure enough to open (which can become a standard in your FW rules).

Push is undesirable because the high cost to deploy, each firewall,
each host need to be configured to allow the connection. Which means
any migration (software or IP) may entail serious effort just for
access rights. Also push means that even though as Ohad pointed out
the manifest is the thing that will / can compromise each host, a pull
can be scheduled. Most people probably run hourly cycles. Maybe
offsetting production and development hosts (do you want a change to
appear quicker or slower on a prod host?) or having tiered solutions
with a puppet server for prod and another for dev. Regardless, pull
does give the illusion of control because each client needs to ask for
the change. So if you are diligent your entire range of hosts won't go
down the drain in one go. (we interpret cf in cfengine as cluster f**k
for a reason).

Are you telling you don't have other means of push already today (ssh??)
and at the end of the day, how many people don't run puppet twice an hour? (really I'm asking!)
there is a risk in having a push, but there is a risk in not having it as well.

Push on the other hand implies each host gets the same change at the
same time. And even if you're tiered you can still do a lot of damage.

So i suggest this: given your need to be able affect each host
instantaneously,
1) why not have a second instance of puppet (i don't know if you can
run two on the same host though) for urgent things only, and query the
different instances at different intervals.

interesting idea... I have to give it some more thought...

2) have a fact that based on time sets the urgency of your modules, so
your regular modules only trip every hour, and your urgent ones
straight away. So say have the fact check the time, and if it's 30
past the hour it does the regular changes, every minute does urgent
and every (seventh) day at midnight does the ultra rare?

I assume you are trying to address the server load problem, in this case I would guess (without checking too much) that the compile time would be similar to using tags.
 

regardless, tune your updates to happen every minute. The minutely
urgent modules should run quickly and all the other ones that evaluate
your fact as meaning they don't need to run should be fast, so your
urgent change should probably run alone, and after 10 minutes you just
change the urgency. I'd probably have a central file that sets the
urgency value for each module, so making the change, updating the
variable in the central file and a daemon restart later you have 1
minute to pray you didn't screw up what you're meaning to fix :).

Heck, you could even have a clever cronjob that runs on the
puppetmaster that checks the file for urgency variables, and 3 minutes
after having been touched it replaces it with the non-emergency one
and restarts puppetmaster if needed...

But implementing push ... you're playing with fire mate :)

sounds like an ssh loop is still easier ;) 

[Ohad Levy]

sure I did, my main point is that it could be integrated withing puppet without using additional tools...

[Larry Ludwig]

>
>
> Why Puppet cant have a push directive as well? obviously we already
> have the infrastructure to connect from the server to the clients
> (e.g. puppetrun, or my xinetd version), we already have all of the
> puppet types, so it should be easy to execute on a client (possible
> today with ralsh or puppet directly) things like:

What xinetd version?

-L

--
Larry Ludwig
Reductive Labs

[chakkerz]

Hello again Ohad

I've finally found some time to play with this.

I'll give you the back of the proof of concept version and i'll write
some doco for the site when i get a chance.

I've done two facts which are probably written in the worst way
possible, but i haven't done this before :)

[root@tangelo facter]# pwd
/etc/puppet/modules/custom/plugins/facter
[root@tangelo facter]# cat system_time_*
# system_time_hour.rb

Facter.add("system_time_hour") do
time = Time.now
hour = time.hour
setcode do
hour
end
end

# system_time_minute.rb

Facter.add("system_time_minute") do
time = Time.now
minute = time.min
setcode do
minute
end
end

Then my default node class works out what OS, and based on that calls
the class for that OS, either OS class inherits from shared-default
which contains the non OS specific files and that shared one contains
an inheritance (the include was not smiling on me that day) to some
default variables.

My default variables file defines:
class initialisevariables
{
##
## daily denotes an hour while hourly denotes a minute
##
$solaris_daily = "7"
$solaris_hourly = "30"

$linux_daily = "8"
$linux_hourly = "00"
}

which the OS specific class uses to set itself up (this is for Linux):
$change_daily = $linux_daily
$change_hourly = $linux_hourly

So so far so good :)

Now the only module i've tested this on is my resolv_conf one which I
added at the start:
class resolv
{
$resolv_conf_urgency = "hourly"
case $resolv_conf_urgency
{
"minutely" : { $process_now = "true" }
"hourly" : { if ($system_time_minute == $change_hourly)
{ $process_now = "true" }}
"daily" : { if ($system_time_hour == $change_daily )
{ $process_now = "true" }}
}

if (($skip_resolv != "true") and ($process_now == "true"))
{
....

I'm not fully happy with this as yet, but in principle it works.


As to your query regarding me not having a current way to push things
out immediately... no i don't. IF we need to do something like this we
modify our cfengine instance and then hook onto each box that needs
the change urgently and run it by hand. Puppet will hopefully change
things ... especially with this notion i'm putting in place :)

Cheers
chakkerz

[Ohad Levy]

Hi chakkerz,

First of all thanks for putting the effort in this.

secondly, I'm not sure, but what is the difference between what you just implemented with normal schedule meta parameter?
schedule allows you to decide how often a certain resource would be checked/applied, and if I understood your code below, this is what you have implemented.

again, if I didn't miss anything in your implementation,  you still have a minimum of time (e.g. by default up to 30 minutes) that it takes to rollout a change, and when we are talking about critical changes this is not an option.

I think I'll enhanced my mini puppet listener script and post it if anyone would be interested...

Cheers,
Ohad

[Ohad Levy]

On Tue, Apr 7, 2009 at 8:32 PM, Larry Ludwig <la...@reductivelabs.com> wrote:

What xinetd version?

I wrote a simple daemon which uses puppet certificates and namespaceauth to allow puppetrun on cron based hosts.

cheers,
Ohad

[Helmut Lichtenberg]

Ohad Levy schrieb am 08. Apr 2009 um 05:14:14 CEST:
> I think I'll enhanced my mini puppet listener script and post it if anyone
> would be interested...

That would be great.

Helmut

--
-------------------------------------------------------------------------
Helmut Lichtenberg <Helmut.Li...@fli.bund.de> Tel.: 05034/871-128
Institut für Nutztiergenetik (FLI) 31535 Neustadt Germany
-------------------------------------------------------------------------

[Peter Meier]

Hi

> I think I'll enhanced my mini puppet listener script and post it if anyone
> would be interested...

that would be great. Maybe it's even something for the contrib directory?

cheers pete

[Larry Ludwig]

Hi Ohad,

Can you post this somewhere?  This is something I was looking for and/or was going to develop.

-L

cheers,
Ohad

[James Turnbull]

If you log a ticket I am happy to add this into the contrib directory.

Regards

James Turnbull

[webx]

Maybe it's the purist in me, but I don't think emergency, one-off
changes are what puppet is/was designed for. It's supposed to be a
configuration management tool, and (using your example) firefighting
death spiraling daemons is not a puppet task. (in my opinion) There
are tools that are already designed for this, most of which have been
mentioned.

I'd add that pdsh (parallel distributed shell -- https://computing.llnl.gov/linux/pdsh.html)
is my tool of choice for massive site-wide fixes like you describe.
As an example, I can do the exact job you explained of restarting a
service across 400+ nodes in less than 20 seconds. It's a great tool,
written for exactly the situation you describe. There's something to
be said for having the right tool for the job, isn't there?

I understand the want for a one-solution-fits-all approach, which is
unfortunately typically fool's gold. There are existing tools to do
what you're looking for much faster (efficient) than puppet ever
could, given it's current design.

My two pennies.

--Bill

[Ohad Levy]

Hi Guys,

First working version can be found here:
http://github.com/ohadlevy/puppet/commit/f18af58687bbde62cfd756adaf14764c6f44e5de

I would assume many improvements could be made, patches are welcome.

I've given up on (x)inetd for now, as it seems that its impossible to implement it purely in ruby (openssl ruby implementation works only with socket, inetd doesn't provide a socket....)

puppetlisen should run on the client you want to trigger a run on, puppetrun is executed from an "admin" server - doesnt have to be your puppetmaster (just valid certificates).
permissions are the same like normal puppetrun (e.g. /etc/puppet/namespaceauth.conf).

if there is any good feedback, I would be happy to integerate it in the contrib/ext directory of puppet

cheers,
Ohad

[Luke Kanies]

On Apr 1, 2009, at 8:51 PM, Kyle Cordes wrote:

>
> Ohad Levy wrote:
>> puppetrun, and as we don't run puppetd as a daemon (running it from
>> cron
>> due to large memory consumption while idle), I went forward and
>
> Here is a semi-related thought.
>
> Without regard for memory consumption, but rather due to the notion
> that
> system-wide, manageable, canonical mechanisms are better than
> app-specific mechanisms, I would prefer that Puppet did not have a
> daemon mode, scheduler, sleep-N-minutes mechanism etc., at all; but
> rather that the Puppet install process set it up in cron. Of course I
> can set it up that way myself, but what I wish was that Puppet didn't
> carry the burden of its own mechanism: no code for it, no command line
> options, no init.d script, no wiki pages about it, no discussion of
> it, etc.

The code around being a daemon is a very, very small amount of code,
especially given that the server needs to be a daemon anyway.

And I have to admit that the reasons for puppetd being a daemon are
less valid than I'd hoped - in particular, the 'resource' API is
powerful and potentially really useful, with the right central tools.
I do hope we'll be making them more valid and more useful in the next
year or so, but I have lots of hopes and not enough time for them all.

In the meantime, it's easy enough to skip the daemon mode.

--
The hypothalamus is one of the most important parts of the brain,
involved in many kinds of motivation, among other functions. The
hypothalamus controls the "Four F's": 1. fighting; 2. fleeing;
3. feeding; and 4. mating.
-- Psychology professor in neuropsychology intro course
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com

[Kyle Cordes]

Luke Kanies wrote:
> In the meantime, it's easy enough to skip the daemon mode.

Maybe I'll do that next time I repackage a new version.

(Or maybe, and this is a dream rather than a request, the good folks
working on the official Debian packaging will decide that running it in
cron via a file in /etc/cron.whatever is more genuinely Debianish, and
they'll set it up that way.)

[Paul Lathrop]

I'd much prefer you cron folks keep configuring it manually :-P I
*like* Puppet as a daemon, leave the Debian package alone!

(only half kidding)

--Paul

[James Turnbull]

+1 to Paul's comments. I like having the daemon and if you really want
it the cron option is easy to configure. I'm intending to show both
in the 2nd edition of the Puppet book.

Regards

James Turnbull