Better solution for puppet SVN releases

[Andreas Paul]

Hi,

we are currently managing our puppet modules with one SVN workspace for each admin.
The post commit hook script updates /etc/puppet/ directory and triggers the puppet kick of the correct server.

The problem we have with this solution is that sometimes there are many small checkins to one change, because the admin forgot to change small details in the config file, e.g. forgot to change the access logfile name of the vHost, forgot a redirect, misspelling in the comments etc.

What we end up with are many micro checkins, which can be used to tell every small mistake the admin has done.

What we want is a solution which lets the admin test his changes on one server without checking these changes into the "main" SVN repository.
So that the SVN repository only contains the final releases of the changes.


I have to say that we also manage the dev and QA servers with this puppetmaster. Would dividing of these different stages into puppet environments help us?
What I really want to know is, do you understand my problem and if you had the same problem, how did you solve it? SVN branches? Multiple puppetmasters, one to test with a dedicated test SVN repository?

Thanks!

[Alexandre Fouché]

I had a similar problem, i use Git, and i though i could use branches and Git tricks, but it does not mixes well with Puppet environments. What i do below permits to use many puppet code branches, while having my puppetmaster serve all different branches at the same time using environments

In my case, my puppetmaster does not checkout/pull from a code repo, but could, it does not matter. So in my case, i do a rsync --delete  of my current workingdir from my workstation to the puppetmaster /etc/puppet/environments/{manifests,modules}. My puppetmaster has environments configured, so it can have test environments for the work in progress, and the production environment. The environments permits to have many copies of my puppet manifests and modules for the staging or testing puppet clients, while still being able to serve a Production environment to the prod puppet clients

On my workstation, i commit only when i am sure and everything is tested, and this completely independant from what is on the puppetmaster. As an improvement and for better traceability, i might setup my Production puppet environment as a code repo checkout of my master branch or some tag. I would not do it for the test environments, because like you, i do not want to have to commit many small commits to be able to checkout/pull on the puppetmaster

My puppet environments are rsynced like this. I have a script (which i will probably put on github once i have time, along with all my recipes) which creates a environments/directory and rsyncs directly into it, and change the /etc/puppet.conf to set the environment (from an puppet.conf.erb)

    # ls -ld /etc/puppet/environments/*
    lrwxrwxrwx. 1 root puppet   11 Jul 25 14:47 /etc/puppet/environments/production -> /etc/puppet
    drwxr-xr-x. 4 root root   4096 Aug  7 07:45 /etc/puppet/environments/alex
    drwxr-xr-x. 4 root root   4096 Sep 26 11:58 /etc/puppet/environments/alextest
    drwxr-xr-x. 4 root root   4096 Aug 19 07:51 /etc/puppet/environments/staging
   
    # find -type d /etc/puppet/
    /etc/puppet/
    /etc/puppet/modules
    /etc/puppet/modules/java
    /etc/puppet/modules/java/manifests
    /etc/puppet/modules/bluepill
    /etc/puppet/modules/bluepill/files
    /etc/puppet/modules/bluepill/manifests
    /etc/puppet/modules/mongodb
    (...)
    /etc/puppet/manifests
    /etc/puppet/manifests/nodes
    /etc/puppet/manifests/classes
    (...)
    /etc/puppet/environments
    /etc/puppet/environments/staging
    /etc/puppet/environments/staging/modules
    /etc/puppet/environments/staging/modules/java
    /etc/puppet/environments/staging/modules/java/manifests
    /etc/puppet/environments/staging/modules/bluepill
    /etc/puppet/environments/staging/modules/bluepill/files
    /etc/puppet/environments/staging/modules/bluepill/manifests
    /etc/puppet/environments/staging/modules/mongodb
    (...)
    /etc/puppet/environments/staging/manifests
    /etc/puppet/environments/staging/manifests/nodes
    /etc/puppet/environments/staging/manifests/classes
    (...)
    /etc/puppet/environments/alextest
    /etc/puppet/environments/alextest/modules
    /etc/puppet/environments/alextest/modules/java
    /etc/puppet/environments/alextest/modules/java/manifests
    /etc/puppet/environments/alextest/modules/bluepill
    /etc/puppet/environments/alextest/modules/bluepill/files
    /etc/puppet/environments/alextest/modules/bluepill/manifests
    (...)
    /etc/puppet/environments/alextest/manifests
    /etc/puppet/environments/alextest/manifests/nodes
    /etc/puppet/environments/alextest/manifests/classes
    (...)
    /etc/puppet/environments/alex
    /etc/puppet/environments/alex/modules
    /etc/puppet/environments/alex/modules/mongodb
    /etc/puppet/environments/alex/modules/mongodb/manifests
    (...)

2011/10/18 Andreas Paul <a.p...@enbw.com>

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/FLNYwiOI1fwJ.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[Luke Bigum]

We use Git here and we tag branches to certain environments. We have a
testing branch which everyone commits in to and what most of our
testing machines are using. We then have other environments like
Staging and Production that have specific tag checked out that (will
eventually) align with our Developers' Continuous Integration and
Agile release cycle. Our testing environment has lots of micro check-
ins, commits for every little syntax error I make. To test a specific
machine though, I often temporarily change the environment back to
Testing to make sure nothing is broken, then switch it back to it's
real environment. This is because we don't have a test machine for
every single system (staff workstations, etc).

The idea we're aiming towards is that all Puppet changes up to the end
of an iteration will be included in the next release. A tag is "cut"
and checked out into the Staging environment. All those changes get
applied and our own Developer written code is tested alongside any
Puppet changes. If there are any problems we can update the tag, same
as if there was a bug in our code's release candidate. Then when our
code goes to production, we check out the production Puppet
environment with the same tag (plus any updates).

[Christopher Wood]

I use two separate puppetmasters for lab and production. (The lab side has more of the "dratted commas" style of checkin.)

Testing changes to one particular type of server is where virtualization really shines. You can spin up a scratch virtual machine with the same manifests as your target host, snapshot the VM, and then apply your manifest changes. If the changes break you can revert the VM and fix your manifests. Once things are to your satisfaction you can commit to production.

For VMWare I like snapshotting at the grub screen on each lab host on first boot (before the first puppet run), given that reverting to the last snapshot is higher priority than powering on. That way I have a server where I can conveniently start "clean" on every boot as long as I remember to scrub certs from the puppetmaster.

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To view this discussion on the web visit

> [1]https://groups.google.com/d/msg/puppet-users/-/FLNYwiOI1fwJ.

> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>

> References
>
> Visible links
> 1. https://groups.google.com/d/msg/puppet-users/-/FLNYwiOI1fwJ

[Paul Morgan]

On Tue, Oct 18, 2011 at 01:52:18AM -0700, Andreas Paul wrote:

> we are currently managing our puppet modules with one SVN workspace for each
> admin.
> The post commit hook script updates /etc/puppet/ directory and triggers the
> puppet kick of the correct server.

we use git branches. the master branch is _always_ production-ready.
for every issue, we:

* create a topic branch
* hack away
* push the topic branch to origin/$branch_name
* a post-push service hook creates, updates, or deletes
/opt/puppet/$branch_name

On the puppetmaster, puppet.conf has:

[main]
manifestdir = /opt/puppet/$environment
manifest = $manifestdir/site.pp
modulepath = $manifestdir/modules:/opt/puppet/3pp

We then test the topic branch via the puppet agent
by running 'puppet agent --environment $branch_name ...'

> The problem we have with this solution is that sometimes there are many
> small checkins to one change, because the admin forgot to change small
> details in the config file, e.g. forgot to change the access logfile name of
> the vHost, forgot a redirect, misspelling in the comments etc.
>
> What we end up with are many micro checkins, which can be used to tell every
> small mistake the admin has done.

Our topic branches have lots of these so-called micro-commits,
but that's ok. Once $branch_name is clean, we:

* squash the micro-commits

git rebase -i <hash>

# prepare a linear commit history
git fetch origin
git rebase origin/master
git checkout master
git merge $branch_name

# delete local and remote branch
git branch -d $branch_name
git push origin :$branch_name

* ship

git push origin

> What we want is a solution which lets the admin test his changes on one
> server without checking these changes into the "main" SVN repository.
> So that the SVN repository only contains the final releases of the changes.

The key is to use rebase to squash the micro-commits in the branch.
I'm not sure if svn has that capability yet, but a quick google
search showed at least one script that sort of simulated a rebase
with svn. Or perhaps you could try git-svn.

> I have to say that we also manage the dev and QA servers with this
> puppetmaster. Would dividing of these different stages into puppet
> environments help us?

We manage differences via parameterized modules.
For example:

class foo {
require foo::params
package {'fubar': ensure => $foo::params::fubar_version}
}

class foo::params {
$fubar_version = $fqdn ? {
# exceptions
something.qa.example.com => '2.1.0-1.f15',

# patterns
/^.*\.prod\.example\.com$/ => '1.2.3-4.el5',
/^.*\.qa\.example\.com$/ => '1.2.4-1.el5',
/^.*\.dev\.example\.com$/ => latest,
default => '1.2.3-4.el5',
}
}

The above is not perfect, but it allows us to keep our
node manifests under version control.

> What I really want to know is, do you understand my problem and if you had
> the same problem, how did you solve it? SVN branches? Multiple
> puppetmasters, one to test with a dedicated test SVN repository?

We thought about multiple puppetmasters, but it's more important
to us to maintain a clean version control history in one place.

hth,
-paul