Managing upgrading of puppet recipes

[Pete Emerson]

In order to facilitate puppet manifest upgrades, I need to manage puppet recipes on a per-server basis.

My puppet classifier spits out a variable called $puppet_iteration.

I tried putting this variable in my site.pp:

import "/var/lib/puppet/files/static/applications/puppet/$puppet_iteration/system"

but it looks like the variable does not get interpreted in the import statement:

err: Could not retrieve catalog: Could not parse for environment mgmt: No file(s) found for import of '/var/lib/puppet/files/static/applications/puppet/$puppet_iteration/system' at /etc/puppet/manifests/site.pp:33

Can I get some suggestions for how to load specific manifests based on a path partially determined by my puppet classifier?

Thanks in advance,
Pete

[Ohad Levy]

why don't you use puppet environments?

Ohad

[Pete Emerson]

I take it you're referring to this:

http://reductivelabs.com/trac/puppet/wiki/UsingMultipleEnvironments

So on the puppetmaster server I'd need to generate a puppet.conf to create a [section] for each version I need to run. Since iterating over arrays doesn't appear to be allowed I guess I'd need to look into writing my own function, or simply externalize this to a script that I'd write.

On the client I'd manage the puppet.conf to set the environment parameter to the proper version (this I can easily do with a puppet recipe).

I'll take a look at this today and see how easy it is to write my own function to generate a file like this. I've never done this or written ruby. An adventure, for sure!

Any other suggestions on how to skin this cat?

Thanks, Ohad, for pointing me to this suggestion.

Pete

[James Turnbull]

Pete Emerson wrote:
> On the client I'd manage the puppet.conf to set the environment
> parameter to the proper version (this I can easily do with a puppet recipe).
>
> I'll take a look at this today and see how easy it is to write my own
> function to generate a file like this. I've never done this or written
> ruby. An adventure, for sure!

Feel free to post your function or any code queries to the puppet-dev
list. The people there may be able to assist or provide feedback and ideas.

Regards

James Turnbull

--
Author of:
* Pro Linux Systems Administration
(http://tinyurl.com/linuxadmin)
* Pulling Strings with Puppet
(http://tinyurl.com/pupbook)
* Pro Nagios 2.0
(http://tinyurl.com/pronagios)
* Hardening Linux
(http://tinyurl.com/hardeninglinux)

[Ohad Levy]

Maybe this can save you some time...

http://github.com/ohadlevy/puppet-multipuppetmaster/blob/582f98840e2c7bbea9d9e820a3060b09b0916306/templates/puppet.conf

Cheers,
Ohad

[Pete Emerson]

Brilliant! I think that will save me tremendously. I'll play with it and let you know. Thanks very much.

Pete

[Pete Emerson]

So far I've gone the route of having two static environments, "production" (which seems to be the default) and "development". I had my own $environment definition which was overriding puppet's definition of $environment that caused some problems for a little while, but I've got that all sorted out, and now by changing my database (which in turn changes the results in the puppet node classifier) I can easily flip back and forth between the production and development environments. Very slick!

If I wind up playing with your dynamic creation of puppet.conf, I'll let you know how it goes.

Thanks again for your pointer.

Pete

[Thomas Bellman]

Pete Emerson wrote:

> I take it you're referring to this:
>
> http://reductivelabs.com/trac/puppet/wiki/UsingMultipleEnvironments
>
> So on the puppetmaster server I'd need to generate a puppet.conf to
> create a [section] for each version I need to run. Since iterating over
> arrays doesn't appear to be allowed I guess I'd need to look into
> writing my own function, or simply externalize this to a script that I'd
> write.

Here's a definition I wrote to manage puppet.conf environments:

define puppet_environment()
{
$envdir = "$puppet_conf_prefix/$name"
augeas {
"puppet-environ--$name":
context => "/files/etc/puppet/puppet.conf/$name",
changes => [
"rm *",
"set manifestdir $envdir/manifests",
"set manifest $envdir/manifests/site.pp",
"set modulepath $envdir/modules",
],
require => Package["puppetmaster"],
notify => Service["puppetmaster"];
}
file {
$envdir:
ensure => directory;
}
}

Then use it like this:

$puppet_conf_prefix = "/etc/puppet/environments"
$puppet_environments = [ "production", "testing", "devel-1", "devel-2" ]

puppet_environment {
$puppet_environments: ;
}

The advantage of this over a template is that Puppet won't overwrite the
entire puppet.conf, just the environments in $puppet_environments.

I tend to use lots of "throw-away" environments for my development, that
may live for anywhere between an hour or several weeks, depending on how
long time the development of a certain feature takes. I use them somewhat
similar to how one uses topic branches in Git. When I develop a new
feature, I create an environment for it, do my editing there, and when
I feel ready to test it, I try to run it once (using puppetd --onetime)
on a node that normally runs the production environment. Often with
--noop as well until I think it actually does what I want. When I feel
more confident about my changes, I change that node to that environment
and run it that way for a few days, and when I think it's stable, I check
in my changes to the VCS, and check them out in the production environment,
switching back my test nodes to production as well. And then remove that
development environment.

Depending on how big or riskful a change I'm doing is, I may test my
changes on several different nodes, and/or I may shorten the time my
test node runs my development environment. (Most of the times they
are small enough that I can skip semi-permanent running entirely. I
also have a special development environment that I never erase, but
re-use again and again for very small changes, the kind that takes
just a few minutes; in fact, each sysadmin here has their own such
environment.)

I can have several of these temporary environments open in parallell,
if I'm developing and testing several different things.

With the above workflow, it would be quite disruptive to have Puppet
suddenly overwrite puppet.conf on the master with a version that doesn't
have my environment in it. And having to add my environment to my
Puppet manifests before being able to use it feels like too much
overhead. (In fact, I would like to not have to edit puppet.conf at
all to add or remove environments. Being able to configure a "wildcard"
environment in puppet.conf would be nice.)

/Thomas Bellman

[David Schmitt]

Pete Emerson wrote:
> So on the puppetmaster server I'd need to generate a puppet.conf to
> create a [section] for each version I need to run.

> **Since iterating over arrays doesn't appear to be allowed **
[emphasis mine]

> I guess I'd need to look into
> writing my own function, or simply externalize this to a script that I'd
> write.

This is soooo wrong. Sorry, but puppet allows you very easily to process
arrays with defines:

define puppet::config_section() {
include puppet::config_section_base
concatenated_file_part {
$name:
dir => "/var/lib/puppet/modules/puppet/config.d",
content => template("puppet/config_section.erb");
}
}

puppet::config_section{
[ "production", "development", "testing", "blah" ]:
}

Regards, DavidS

[Pete Emerson]

Here's the solution I wound up using. I created a puppet.conf.erb
template file and had puppet create the puppet.conf file dynamically.

The critical pieces (I only wanted directories starting with v, like
v01, v02, et cetera):

<%
dirs = Dir.entries("/etc/puppet/manifests").sort
-%>

[main]
...
[puppetmasterd]
...
[puppetd]
...

<% for dir in dirs -%>
<% if File.directory?("/etc/puppet/manifests/" + dir) and dir
!= '.' and dir != '..' and dir =~ /^v/ -%>

[<%= dir %>]
manifest = /etc/puppet/manifests/<%= dir %>/site.pp

<% end -%>
<% end -%>