Suggestion - puppet preload stage?

[Amos Shapira]

Hello,

I encounter issues regarding puppet "self update" that I'm sure are
not uncommon:
1. When puppet version updates it doesn't restart to run the rest of
the manifest with the new version.
2. When a new provider is installed (or extra configuration is done to
enable an existing provider), puppet still won't make it available.

example for (1): Our vagrant (http://vagrantup.com/) dev base boxes
still come with Puppet 2.6.3 while the manifest depends on Puppet 2.7
features. I can manually upgrade puppet manually (and that's what I do
on dev), but when the time to deploy to production comes it
practically means that I have to manually upgrade Puppet to 2.7 on all
hosts before I push the change.

example for (2): We use daemontools (http://cr.yp.to/daemontools.html)
to monitor some of our servers. Puppet comes with a "daemontools"
Service type provider but since the daemontools package isn't
installed yet the provider is made unavailable for the rest of the
run. A second puppet run, assuming it managed to install daemontools
in the first one before it failed on other things, succeeds in picking
up the daemontools as a provider.

another example for (2): I use opsview (https://github.com/dpeters/
puppet-opsview) to monitor hosts. When the type is loaded by puppet it
looks for a file called /etc/puppet/opsview.conf to find URL/username/
password of the opsview REST server. This file doesn't exist (or can
be out of date) in the first run causing the Type to fail.

My suggestion -

Define a new "preload" stage - resources which can affect puppet's own
execution (e.g. types, config files, puppet version updates) can be
placed in that stage and eventually a specific function call (or a
special built-in trigger inside the Puppet code?) will cause Puppet to
drop everything at the end of this stage and start from fresh.

The mechanism has to be careful to limit potential inifinite loops -
e.g. if nothing was changed during the stage then don't re-run, or if
the stage keeps changing things and puppet re-starts more than a set
limit then fail.

What do you think about this? Is there another solution/work-around/
standard-best-practice I'm not aware of?

Thanks.

[Gary Larizza]

Amos, have you seen this feature --> http://projects.puppetlabs.com/issues/6907

 

My suggestion -

Define a new "preload" stage - resources which can affect puppet's own
execution (e.g. types, config files, puppet version updates) can be
placed in that stage and eventually a specific function call (or a
special built-in trigger inside the Puppet code?) will cause Puppet to
drop everything at the end of this stage and start from fresh.

The mechanism has to be careful to limit potential inifinite loops -
e.g. if nothing was changed during the stage then don't re-run, or if
the stage keeps changing things and puppet re-starts more than a set
limit then fail.

What do you think about this? Is there another solution/work-around/
standard-best-practice I'm not aware of?

Thanks.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

--

Gary Larizza

Professional Services Engineer

Puppet Labs

[Amos Shapira]

Thanks for the pointer, Gary.

It seems to address the issue of missing providers, even better than my suggestion.

However, I'm not sure if/how it addresses other situations, e.g.:
1. Provider exists but has to be updated.
2. Puppet version update.
3. Configuration files - missing or have to be updated.
4. Other types of configurations which have to take place, e.g. add a network link (e.g. VPN, proxy), or make any other arbitrary change which can affect the rest of the puppet run (or future puppet runs).

What I'm thinking about is a generic way to say "OK, now I'm finished doing all sorts of tweaks to the system, please start puppet and take these tweaks into consideration".

Is that possible?

Cheers,

--Amos

To unsubscribe from this group, send email to puppet-users+unsubscribe@​googlegroups.com.

For more options, visit this group at http://groups.google.com/​group/puppet-users?hl=en.

[Luke Bigum]

Hi Amos,

One of the least hacky ways I've found around the chicken and egg scenario is to use Custom facts to determine what state Agent is in and then use or don't use your providers accordingly. For example I have a fact for determining if ruby-libvirt is installed and Puppet has set up the necessary networking so I know I can use the 'virt' custom provider to create VMs. Unfortunately it requires multiple passes, but removes the need for manual intervention and intermediate errors from the logs.

-Luke

To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/XUeixQ5zq7sJ.

To post to this group, send email to puppet...@googlegroups.com.

To unsubscribe from this group, send email to puppet-users...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

-- 
Luke Bigum

Information Systems
Ph: +44 (0) 20 3192 2520
luke....@lmax.com | http://www.lmax.com
LMAX, Yellow Building, 1A Nicholas Road, London W11 4AN

The information in this e-mail and any attachment is confidential and is intended only for the named recipient(s). The e-mail may not be disclosed or used by any person other than the addressee, nor may it be copied in any way. If you are not a named recipient please notify the sender immediately and delete any copies of this message. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Any view or opinions presented are solely those of the author and do not necessarily represent those of the company.

[Amos Shapira]

Thanks Luke (and sorry for late reply).

This sort of addresses another question I haven't got around to investigate - the question is whether Puppet has functions/facts to tell me things like whether a particular provider is available or not. I guess it can be somehow scripted.

But as you say - your hack won't avoid re-runs, it might allow cleaner messages, though.

Cheers,
--Amos

[Ken Barber]

Hi Amos - its been a long long long time mate :-).

> example for (1): Our vagrant (http://vagrantup.com/) dev base boxes
> still come with Puppet 2.6.3 while the manifest depends on Puppet 2.7
> features. I can manually upgrade puppet manually (and that's what I do
> on dev), but when the time to deploy to production comes it
> practically means that I have to manually upgrade Puppet to 2.7 on all
> hosts before I push the change.

You know - I've generally just used mcollective with the puppetral
plugin to do the Puppet agent RPM/deb upgrade in the case where the
client is the wrong version, and can now no longer interact with my
master:

http://projects.puppetlabs.com/projects/mcollective-plugins/wiki/AgentPackage
https://github.com/puppetlabs/mcollective-plugins/tree/master/agent/package

At the end of the day, it uses the Puppet resource abstraction layer
anyway behind the scenes, only difference is that it is making the
change directly without the Puppet master interaction. But I can see
why you would want Puppet to just deal with this on its own.

> 1. Provider exists but has to be updated.

I could be mistaken about what I'm about to say, as I haven't
experimented with this new behaviour all that much. I wouldn't take my
word for it - so experiment and see :-).

Surely the lazy evaluate solves this, I could be wrong - but if you
build up a relationship between the action that does the update - and
the provider that users that update then the update should occur
first. Then when the provider is evaluated lazily - all is well - it
has a new updated command/library/whatever?

So:

package { "daemontools":
ensure => "1.2.3",
}
Service {
require => Package["daemontools"]
}

Is a rough way of doing it, but the default _is_ of course generic
here, and run stages may be a better solution - as its easier to have
your 'require's changed/overridden later on. (because defaults don't
stack with overrides).

> 2. Puppet version update.

Updating Puppet can be tricky, from an agent perspective if you run it
as a daemon there was a problem when updating to the recent Facter
1.6.6 for example - as parts of Facter are loaded/evaluated
dynamically each (ie. the facts) - while others are loaded and cached
and never re-evaluated. This created a problem with I believe the ec2
fact from memory - which started to use a library in facter/util/ec2
(yes my fault). In that edge case you need to probably force a restart
of an agent after the package update really. Its crap I agree ... and
I'm all ears for a better solution/policy generally.

From the master perspective - you can run into similar scenarios as
well. Generally speaking, the 'puppet managing puppet' case is always
hard mate. I always spend the most amount of time helping clients
'manage' their puppetmaster with Puppet. And their are lots of
strategies here :-).

What are you hitting that is problematic at the moment? Or what in
particular do you see?

> 3. Configuration files - missing or have to be updated.

Again, build up the dependencies using relationships in Puppet - as
per 1. Should work in a similar manner (from the description of the
new feature that is).

> 4. Other types of configurations which have to take place, e.g. add a
> network link (e.g. VPN, proxy),

Same again.

> or make any other arbitrary change which can
> affect the rest of the puppet run (or future puppet runs).

You might be getting into the 'manage the puppet case', again -
anything specific you expect?

> What I'm thinking about is a generic way to say "OK, now I'm finished doing
> all sorts of tweaks to the system, please start puppet and take these tweaks
> into consideration".
>
> Is that possible?

To be absolutely honest with you - up until now, this has been done
with pre-puppet stuff. That is - done with kickstart, jumpstart or
whatever you use to build the system. Of course, this then becomes
non-idempotent which makes Puppet less useful overall.

Dude - try the lazy evaluate with the edge cases you've been seeing
and see how far you get - and lets discuss anything that still won't
work. We're really interested in solving this kind of chicken & egg
problems for good if we can.

ken.

[Brian Gallew]

I just did a 2.6.11 -> 2.7.12 migration with Puppet updating Puppet.  The only real hitch I ran into was my puppet masters would update themselves but somehow Apache/Passenger didn't get restarted, so I had to do that by hand.  Since I didn't start putting any 2.7 features into my manifests until *after* the 2.7 upgrade, the clients basically just upgraded themselves.  And while I do have puppet running as a daemon, it's with no-schedule, and the "real" run happens out of cron.  I think I'm actually going to kill off the daemon as it offers no benefit to me (and frequently wedged when I let it schedule updates).  Doing it in cron allows me to meet externally imposed scheduling restrictions (no changes while production is running).

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.

To unsubscribe from this group, send email to puppet-users...@googlegroups.com.

[Amos Shapira]

Thanks Ken,

I think I may have been confused with the pre-2.7.8 puppet still being installed before this upgrade was pushed out. Once the later version was installed the manifest seemed to work right.

For instance the first opsview run on a server will complain about the missing config file but still work after all since it re-reads the configuration file when it's actually used, whereas it used to just fail on 2.6.x. I still ended up using polysh to do the initial "yum upgrade puppet" on all servers but hopefully that won't have to be repeated too much.

For puppet upgrade itself - a possible, untested, work-around could be to Exec[] something which will restart the puppet agent if puppet was upgraded. Risky and ugly but might save someone from having to do this manually many servers.

For now I guess you can ignore my request.

Cheers,

--Amos