Recipes release management : separate test and prod

[Jean-Baptiste Barth]

Hi there,

Is there a canonical way to manage my recipes' release process ? I mean test it before they're pushed in production.

For the moment we only have 1 puppetmaster here. I noticed that if I have a syntax error in one of my included .pp, then puppet runs fail on all nodes (even if they do not use this recipe), which is not acceptable. More generally, working directly on production recipes seems too riskee, since people who will manage those recipes are not devs but sysadmins, which means poor development skills (and no versionning for the moment).

I found the documentation about puppet "environments" ( http://docs.puppetlabs.com/guides/environment.html ) : does anybody use it here ? I also read some of you have 2 puppetmasters running on different ports, for test and prod : how do you do that (init scripts available somewhere? different conf/manifests directories? sync between test and prod?). Do you have a pool of client machines dedicated to your tests ? Do you handle this kind of problems directly with your version control system (post hooks?), or maybe at an other level in your external node provider ?

I was not able to find much information about that, but don't hesitate to redirect me to an article or a thread here if it has already been discussed.

Thank you,

Cheers,

--

Jean-Baptiste

[Daniel Pittman]

Jean-Baptiste Barth <jeanbapti...@gmail.com> writes:

> Is there a canonical way to manage my recipes' release process? I mean test

> it before they're pushed in production.

Not "canonical", as far as I know.

> For the moment we only have 1 puppetmaster here. I noticed that if I have a
> syntax error in one of my included .pp, then puppet runs fail on all nodes
> (even if they do not use this recipe), which is not acceptable.

We work on this by keeping the puppet configuration in Subversion. We then
have a Makefile at the top level which runs a puppet syntax check on any
modified file before a commit.

While the puppet check option is pretty terrible[1], it does catch the most
gratuitous syntax errors before they get into the repository.

> More generally, working directly on production recipes seems too riskee,
> since people who will manage those recipes are not devs but sysadmins, which
> means poor development skills (and no versionning for the moment).

You should explain to your current sysadmins that they are not good enough,
and that if they don't improve you will hire better sysadmins — seriously.

That said, wrapping up the process in something like a Makefile that will do
the Subversion commit, and using appropriate hooks so that the push to
production is automatic means you can work around almost anything:

All you have to do is document the point-and-drool version of the
instructions, then make sure they are tasked by their manager to follow those
instructions.[2]

Also useful is a culture of code review: when they write puppet code it should
be complete and thorough, not "experimenting on the production systems", so
they should be able to have a second member of staff review their change
*before* they are allowed to commit it.

> I found the documentation about puppet "environments"

> ( http://docs.puppetlabs.com/ guides/environment.html ) : does anybody use
> it here?

Yes. Other than the semi-documented limitations about custom types and
providers, we use them to good effect. They allowed me to set the standard
for my sysadmin staff to have them test everything before they release.

> I also read some of you have 2 puppetmasters running on different ports, for
> test and prod: how do you do that (init scripts available somewhere?
> different conf/manifests directories? sync between test and prod?).

I wouldn't bother with this, for all that it would get different types: it
seems over-kill for the return on the investment.

> Do you have a pool of client machines dedicated to your tests?

Sometimes. Other times we rely on testing the modified manifest in place.
The former, using a dedicated test machine, is vastly preferable.

Mostly, though, I think your problem will be best solved by getting the other
sysadmins doing the right thing — which might mean getting management involved
to make sure it actually happes.

Regards,
Daniel

Footnotes:
[1] It doesn't do *ANY* validation of types, so you can have something
obviously stupid like 'file { "foo": requiref => blah }' and it passes.
It would be really nice if it checked even just that the keys were
vaguely valid.

[2] Of course, this will take a few rounds of them finding the flaws in the
instructions, and you improving the document, but eventually...

--
✣ Daniel Pittman ✉ dan...@rimspace.net ☎ +61 401 155 707
♽ made with 100 percent post-consumer electrons

[Dan Carley]

On 5 July 2010 11:01, Jean-Baptiste Barth <jeanbapti...@gmail.com> wrote:

For the moment we only have 1 puppetmaster here. I noticed that if I have a syntax error in one of my included .pp, then puppet runs fail on all nodes (even if they do not use this recipe), which is not acceptable. More generally, working directly on production recipes seems too riskee, since people who will manage those recipes are not devs but sysadmins, which means poor development skills (and no versionning for the moment).

I found the documentation about puppet "environments" ( http://docs.puppetlabs.com/guides/environment.html ) : does anybody use it here ? I also read some of you have 2 puppetmasters running on different ports, for test and prod : how do you do that (init scripts available somewhere? different conf/manifests directories? sync between test and prod?). Do you have a pool of client machines dedicated to your tests ? Do you handle this kind of problems directly with your version control system (post hooks?), or maybe at an other level in your external node provider ?

Mostly echoing Daniel Pittman's comments..

 - We use environments a lot. The default production environment is for, well, production. All other environments are for development use. Each has their own SVN branch and Puppet `manifest`/`modulepath`/extlookup directives.

 - In addition to custom types, providers and facts, flapping back and forth between environments. It's also not possible to have separate fileservers for additional paths between environments. But it's probably not something that will affect you at this stage.

 - We have virtualised dev/staging machines which reflect the roles of our production machines, only on a slightly smaller scale. Any changes get pushed here and monitored first. We also have some shell scripts which are a shorthand for `--test --noop` to quickly review what is going to happen for a given machine. When we're happy with the results then the changes get merged to production.

 - Syntax errors are caught by a pre-commit hook. It won't guard against stupid yet syntactically correct code. But it does vastly reduce the time spent dealing with simple humanoid typos.

 - Be sure to version control your manifests from day one. Without doing so, you'll have no way of knowing who changed what and when. Nor the ability to roll back to a point in history. It will also provide you with an easy mechanism to peer review any code that you're not confident with, as Daniel notes.

[Brian Gallew]

First of all, if your sysadmins aren't advocates of version control (even more so than your developers), fire them now, Now, NOW.  Or at least hire a competent senior sysadmin to train them properly.

Back to the topic at hand.  We have only two environment (production and development).  Since the puppet.conf file is itself managed by puppet, adding "environment = <%= environment %>" to the [puppetd] stanza means that switching a system between production and development is as simple as "puppetd --test --environment development".  That change is "sticky", in that it won't revert without explicitly doing "puppetd --test --environment production".

We keep our manifests in Bazaar.  When we've got a successfully tested/reviewed set of changes in development, we do a "bzr commit" on the development side, and then a "bzr merge bzr://bzr/puppet/puppet-dev;bzr commit" on the production side.  The only real down side is that moving code to production means that either *everything* in development is now ready for production, or that you can isolate the interesting changes for the commit/merge.  Since there's really only one of us working on manifests at the moment, it's not a big deal.

For future needs, we've got a 2nd puppetmaster set up identically that we can use if we're afraid the changes might crash the production puppetmaster. 

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[Jeff McCune]

On Mon, Jul 5, 2010 at 3:01 AM, Jean-Baptiste Barth
<jeanbapti...@gmail.com> wrote:
> Hi there,
> Is there a canonical way to manage my recipes' release process ? I mean test
> it before they're pushed in production.

I recommend adding a "tests" directory in your modules which include
the classes in the manifests directory.

This allows you to quickly "try out" discrete pieces of your
configuration using puppet stand alone.

If you'd like to test complete configurations, I recommend setting up
a "testing" environment for this purpose. I stage changes into the
testing environment and exercise them before promotion to production
using puppetd --environment=testing

Of course, version control is really nice here. Specifically, Git's
ability to switch branches quickly in the same directory works nicely
with environment configurations.

--
Jeff McCune
http://www.puppetlabs.com/

[christopher floess]

> We work on this by keeping the puppet configuration in Subversion. We then
> have a Makefile at the top level which runs a puppet syntax check on any
> modified file before a commit.
>

Uh, would you mind divulging to someone with a little less experience,
how this syntax check works? I can read make, if you just want to post
the contents of it.

Thanks in advance,
Chris

[Daniel Pittman]

christopher floess <skep...@gmail.com> writes:

Requires GNU Make, and possibly bash as the default shell:

check := pp erb rb
check_pp = puppet --confdir=/tmp --vardir=/tmp --parseonly --ignoreimport \
--color=false --modulepath=./modules $< && echo "$< syntax OK"
check_erb = echo -n "$<: " && erb -x -T '-' $< | ruby -c
check_rb = echo -n "$<: " && ruby -c $<

define check_conflicts
if /bin/egrep -q -x -e '^(<<<<<<<|=======|>>>>>>>)' '$<'; then echo "ERROR: conflict marker found in $@"; /bin/false; else :; fi
endef

all: pull test
push: all
@echo "Committing your changes to the upstream server..."
@svn commit

to_test := $(foreach ext,$(check),$(shell find * -name "*.$(ext)" -printf '.check/%p\n' | sort))

test: $(to_test)
.check/%: ext=$(subst .,,$(suffix $(<F)))
.check/%: %
@mkdir -p $(dir $@)
@$(check_conflicts)
@$(or $(check_$(ext)),$(error syntax rule check_$(ext) is not defined))
@touch $@

> Thanks in advance,

Don't thank me until you see the code. ;)

Daniel

Also, this is edited to excise support for git, and remove extraneous bits and
some other extension-based syntax checks for non-puppet bits we use.

[Jean-Baptiste BARTH]

Thanks for all your advices, will be very very useful.

Just one thing about versionning : I personnally put into git
everything I can. The sysadmins at my new job don't have this habit,
this is just a fact.
So it's obvious I'll spend some afternoons to teach them git, svn or
any vcs, but sometimes we have to cope with the reality ;)

Thanks again,

Jean-Baptiste

[Nan Liu]

puppet --parseonly can perform syntax check on the file. A detailed SVN precommit hook example is available at: http://projects.puppetlabs.com/projects/puppet/wiki/Puppet%20Version%20Control

Thanks,

Nan

[James Turnbull]

Dan Carley wrote:
>
> - In addition to custom types, providers and facts, flapping back and
> forth between environments. It's also not possible to have separate
> fileservers for additional paths between environments. But it's probably
> not something that will affect you at this stage.
>

Dan & Daniel (and others... :))

Just so people know this is fixed in 2.6 and when we're finished
hammering the alpha into a beta - we'd love you guys to test this.

Regards

James Turnbull

--
Puppet Labs - http://www.puppetlabs.com
C: 503-734-8571

[Dan Carley]

On 6 July 2010 21:21, James Turnbull <ja...@puppetlabs.com> wrote:

Just so people know this is fixed in 2.6 and when we're finished

hammering the alpha into a beta - we'd love you guys to test this.

Sure, I'll try and put some time aside for testing.

It sounds like 2.6 should resolve a few of the bugbears I've accumulated over time.