Change Management Practices.
Teyo Tyree
Russell Miller
In the course of training and consulting with Puppet, the question of change management best practices has come up over and over again. On the edges, we have small teams that can get away with simply version controlling their code using an SCM as an incremental backups while rolling out change in a fairly adhoc fashion and larger teams that need branches, QA, and DEV environments, and perhaps even separate repositories for each module. There is also the issues of roll back and testing. We are curious how the community approaches these problems in hopes of developing some best practices. So what do you guys/gals do?
The policy that I set up at my previous workplace with some measure of success involved a "dev" and "release" branch. You could make as many changes as you wanted to the dev release, but when it came time to promote it, you would have to send a ticket, have someone else review/test the release, then promote it to release using a custom tool that created a pointer (using svn externals) setting it as the current release tag.
This worked pretty well, but it had some deficiencies too. For example, if no one else was available to review your release, it slowed things down, and sometimes tickets got lost. And sometimes bad releases DID get out.
--Russell
Marc Fournier
Here we don't (yet) have different code bases for production and
development, but are considering it. Instead, we each have a clone of
the manifests in our home-dirs and test new stuff by running:
puppetd -t --environment <username>
on relevant dev machines, then push/pull the changes into the central
repository on the puppetmaster once everything seems ok.
As we have different puppetmaster servers (more or less one for each
customer), we try to share the most we can by putting almost everything
in modules, stored in seperate repositories on github. Then using
git-submodule (currently testing git-subtree [1] as a replacement) to
glue them all together in one big repository on each puppetmaster. This
forces us to write cross-platform manifests, in a "one application =
one module" fashion.
Marc
Digant C Kasundra
We'll be developing extensive practices in this area when we move to 0.25 and start using the environment support (that's why we wanted it, actually: our change management practices are difficult without them).
Generally speaking we plan to have all servers pointing to a "stable" environment where only bugfixes or emergency changes are introduced. We will develop in a testing/unstable branch and when we finish our testing, we will tag that as stable and move servers over to using this environment as the departments approve of the change.
--
Digant C Kasundra <dig...@stanford.edu>
Technical Lead, ITS Unix Systems and Applications, Stanford University
Nigel Kersten
We store all our configurations in VCS.
All edits are done to a "common" branch.
Teams responsible for specific platforms (say "mac") perform a VCS
integrate from the common branch to a mac_unstable branch.
The team members tend to live on their own unstable branch, and
regularly bug fix in common and repeat the integrates to
"mac_unstable".
Once that has settled, a VCS integrate is performed from
"mac_unstable" to "mac_testing".
A larger group of pilot users not in the immediate team group live on
this branch/environment.
Once that has settled, a VCS integrate is performed from "mac_testing"
to "mac_stable".
This allows all the teams to follow their own release cycles, yet all
still share a common code base.
We have an extensive series of presubmit hooks that do things like run
puppet --parseonly --detailed-exitcodes on each .pp file, that
validate that file source urls like puppet:///module/file actually
point to a real file, style guide enforcement like trailing commas,
whitespace, tabs, etc etc.
We also have test servers where we can do things like redirect all
environments to refer to the "common" development environment for ease
of testing fixes without going through the whole release process.
We sit down every so often and look back at the developed code base to
see where we can combine solutions from different groups and share as
much configuration as possible.
It's actually a bit more complicated than this because our
environments are more fine-grained than just "mac_unstable" so we can
target different OS versions differently and different hardware types
(workstation, laptop) differently, but the same principles all hold.
Oh, and we also have matching apt repositories for the
unstable/testing/stable environment suffixes, and perform simultaneous
releases/promotions in apt and puppet.
> --
> Teyo Tyree :: www.reductivelabs.com
>
> >
>
--
Nigel Kersten
nig...@google.com
System Administrator
Google, Inc.
Ohad Levy
warning: this is going to be long and complicated, i think this is the part which was most challenging in setting up our puppet infrastructure.
some background and assumptions:
- we have many sites / locations / data centers.
- we have many different host types (e.g. server profiles)
- we have site administrators (responseble only for their site)
- we have global administrators, each one is responsible for a different topic globally.
- we usually have a very short maintenance window on our servers (and its never on the same time across the different sites).
- each host type might have a different maintenance window.
we were looking for a way, of how we can have the same setup in all of our locations, but still managed by puppet so we came up with the following requirements:
- every change should be done by puppet (unless really really critical and bla bla - never happened).
- "global administrators" (or I don't know how else to call them, in contrast to site administrators) are the only one to control their respective parts in the manifest.
- site administrators should have a way to rollout changes immediately and override global settings if something breaks (like they used to before puppet).
- everything must be visible to every administrator
- every change must be tracked etc.
- must provide a release cycle.
- must be easy to audit
configuration database / repository
- Consists of all Global (aka standard) configuration.
- Consists of all Sites customizations (or global configuration overrides).
- Access Control, Change Log
- Using Version Control
-Ticketing System, Milestone management (e.g. downtimes) etc
puppet modules ( we defined 3 types of modules )
- Host Type - primary purpose of a host
--- Each host is associated with one host type
--- Always inherits the Host base class for common policy
- Service modules - used (shared) by other modules, e.g. ssh, syslog, apache, x11 etc.
---Avoids repeating the same code again and again
---Breaks down the scope of each module, allowing different module owners per topic
---Changes requires more attention as many host types might use the same service module
---Should be plug-and-play
- Sites Modules - sites customizations
--- The only place a site can edit his settings (Sites can not change global modules)
--- Each site can only change his own site module
--- Site modules provide a mechanism to override the global modules in a visible and auditable way
--- As Site modules are not tagged (versioned) any change in the site specific settings, will be rolled out immediately.
on top of all that we tag the modules, and associate them with a version:
Versioning – The ability to refer (tag) to a certain state/PiT
Host Type and Service modules are associated with a version, e.g. ssh v1.0
Site Modules are not tagged in order to reduce complexity
Why ?
Allows each module owner to “release” testing and stable versions of his module
Sites subscribes to modules in a certain version, allowing each site to select which and when changes will be applied
Environments – The ability to select a group of modules in a certain version
Each site is associated with at least two environments(usually more) – Production and Testing:
Testing – an environment which refer to versioned modules under a test or productive tag.
Production – A selection of modules in a certain version that has already been approved for Production usage.
Development environment – The state where the modules are in the latest check-in version, but are not tagged (e.g. not a testing candidate yet).
Development --> Testing --> Production (Per Host Type)
Each module owner should review the local sites adaptation, to find out the reasons for change
- Improve communications
- Improve host-type / service module quality
- Auditing ensures that our infrastructure remains harmonized
on top of that, we also have global environments, this is needed for services which are identical across the sites but and not under the site control, for example puppetmasters, ldap servers etc.
hope it helps :)
Ohad
Andreas Zuber
We developed some strategies to make puppet module development easy to test
and we have a way to rollout our configuration to different environments by
simply taging the versions we think they are stable enough. Basically we use
the power of the DVCS Git to get that done.
Development or changes of puppet manifests:
If we build a new Server we can choose between two different post install
scrips. The standard script simply installs puppet and register the node on
the puppetmaster for one of the three environments we have
(testing->integration->production) The other one installs a puppetmaster and
an empty git repo with a post-receive hook. This way it can be used as a
development-server.
And this is how it works:
- take your copy of your puppet-modules repo and commit your changes (don't
push them to the real puppetmaster)
- add the new development-server as a remote target to git.
- push the branch to the development-server
This is where the post-recieve hook kicks in. it copies the HEAD to the
modules directory of the development-server (he has his own puppetmaster) and
runs "puppetd --test --debug". That's it! Just make sure the
development-server includes all classes you want to test.
You can also do cosmetics on your commits while you fix some syntax errors or
small bugs, because you never pushed your changes to a public repo.
Rollout:
If the new version is ready and we believe everything is right, we push our
changes to the puppetmaster (if there are changes and you have to merge, go
back to your development-server and check if everythis is still ok).
On the puppetmaster there is another post-receive hook that copies the HEAD to
the testing environment. If everything went fine, we simply tag the version
with a integration-YYYYMMDDxx tag and push that tag. the post-receive hook
then copies that to the integration environment... and so on..
Rollback:
This comes for free, simply tag an older version with a newer tag (and restore
the damn file that ensure => absent killed from the backup :-) ).
We don't have that much servers in puppet at the moment ~80 and only 2 people
working on the manifests so i don't know if this scales if numbers grow.
Maybe someone here with experience on larger setups has some suggestions on
this? would be very welcome.
I hope someone understands my poor english.
Andy
Dan Bode
Our build processes uses 3 environments, noop setting, SVN, inventory fakedb (csv file), and custom reports
ENVs
- dev - separate server, you can do whatever you want here, puppet can rebuild this server in case you go to far :)
- test environment - low impact machines - a place to apply changes to a large group of machines without getting fired if something goes wrong (although developers might yell at you)
- prod - high impact machines - only when you are absolutely sure ....
3 separately visioned directories
- conf - manifests + site configuration file
- dev - services and modules
- global - everything not supported by environments (tools=[external node classifier, custom reports, build process code], plugins)
INVENTORY - defines node attributes for external_node_classifier (and provisioning process)
environment column - the external_node_classifier will send a node to this new environment if he is not currently there.
Build Process:
1. identify changes
2. move test agents to dev puppet server (via inventory fake db)
DEV ENV:
3. Iterative development
4. test
5. create a release
TEST ENV:
6. checkout latest test release (automatically forces everything into SAFE MODE, noop=true)
7. activate environment (turn safe mode = false)
8. create production release
PROD ENV:
6. checkout latest prod release (automatically forces everything into SAFE MODE, noop=true)
7. activate environment (turn safe mode = false)
Reporting:
create new reports dir ENV/RELEASE/SAFE_MODE/
for each report type (ENV/RELEASE/SAFE_MODE/) - one report per host
if the report already exists, then the puppetd run does not coincide with a new release and emails should be sent if anything changes.
Limitations:
- puppet servers manage part of the process by changing the puppet.conf file
- environments do not support plugins, external_node_classifiers, etc..
- noop can only be set on the client side(not per environment)
MIsc:
there are some other minor points not mentioned.
- svn commit hooks do things like verify code and notify admins when releases are created
- conf and dev repos are separate so that users can make conf changes on the test environment (skipping the dev steps in the process) and so that conf changes can be triggered by scripts (EX: rootpwchange script), and maybe eventually a web interface.
- when relevent, puppet server checks for the hosts in its environment (inventory file) and broadcasts a puppetrun call.
- other...
Phase 2:
1. Tickets are created in some kind of a ticketing system
2. Code releases reference tickets
3. system associates all infrastructure changes(from reports) - with the corresponding build IDs. Probably by autogenerating tiki pages in the build management system (trac or whatever)
For any change in the infrastructure, you can see the ticket, the source changes and the resulting changes in the infrastructure. Then the infrastrucutre change process becomes a well understood and auditable process.
enjoy!
Dan
Matt
The main puppet tree lives in svn and consists of the majority of
service modules, we're only deploying to CentOS boxes so this makes it
relatively easy. We have:
* two environments - development, production
* three puppetmasters - dev, testing, production
* Jira system to log changes/requests/additions to application and
service modules.
Various application manifests/modules are svn:externals to the related
project, that way ops team can own service/generic modules while
application teams can develop app specific modules.
The puppet trunk can be checked out, modules changed or different
versioned externals pulled in, build a distribution, then release to
the dev environment and test.
Once a puppet release to production is required, a release is built
from the puppet trunk and deployed to the testing master, then finally
to the production master. Rollback is in the form of deploying the
previous distribution.
Issues
-- The development master can be a bottleneck.
-- Concurrent work streams on application / service modules can cause
a release to be bad.
This is our first stab at it, and i'd like to investigate how people
are using environments and tags to always have a releasable puppet
trunk.
2009/7/17 Teyo Tyree <te...@reductivelabs.com>: