Multi environment and multi system

[Joao Morais]

How to manage a Puppet Server with multi-environment and multi-system?

Background:

I am using a `puppet-environments` repository with some branches that
represents my environments. r10k provides the sinchronization between
repository and Puppet Server. Environments (branches) are something like
devel, staging, production.

Every new implementation starts on devel and they are tested on
development environments. If everything is ok, these changes are merged
to staging, after that to production. One base configuration, one
system. So far so good.

Problems start when I try to create a workflow with more than one system
that share the same base configuration.

For instance: environments dev and prod; systems A and B.

A) I cannot merge base configurations, eg fix vulnerabilities, from dev
to prod because this would also promote configurations of systems A and
B. Sometimes base configurations and system configurations does not
evolve at the same time. Currently I workaround this creating a branch
`root` where all the base configurations are commited; dev and prod
receive merge from root. Btw this sounds to me as a poor solution, I
cannot think about a fast-forward merge anymore.

B) It it just impossible to use automatic merges to promote system
configurations because A and B are not related. So although I am using a
super-killer repository, I need to copy files from one branch to another
by hand.

I am miserably failing to improve this scenario without creating a
hard-to-maintain-workaround. Your ideas and thoughts will be very much
appreciated.

[Maxim Nikolaev]

You can use hiera and set there all environment and system variables. So you'll have same manifest for all environments , but devided code variables.

[Joao Morais]

On 03/10/15 08:22, Maxim Nikolaev wrote:
> You can use hiera and set there all environment and system variables. So you'll have same manifest for all environments , but devided code variables.

Hi. Most of my configurations already come from Hiera. This makes things
easier. However the problem persists: I cannot promote configurations
(manifests and hieradata) of system A using merge because both unrelated
systems A and B share the same branch in the repository.

[Maxim Nikolaev]

Can you split repositories for manifests and config?

[Richard Gray]

On 03/10/15 13:08, Joao Morais wrote:
> How to manage a Puppet Server with multi-environment and multi-system?

This is something I have also been grappling with lately. We too, have a
number of environments (development, staging, production) that a host
might belong to, and a deployment model that propagates changes through
the environments from dev, to staging, to production. When we initially
deployed this infrastructure, we created Puppet dynamic environments
corresponding to those physical environments (i.e. Puppetfile and Hiera
repos, each with dev, staging, and production branches on each). In
hindsight, I think this was a mistake to have this direct relationship
between Puppet environments and physical host environments, for the
reasons you suggest. Changes accumulate quickly in the dev environment,
and it becomes challenging to promote certain changes to staging or
production. You either merge the entire branch, and have to coordinate
releases carefully with everyone working on dev, or you cherry pick
particular commits from Dev, and then have to somehow manage keeping the
environments in sync.

My plan now, is to change this to a model where nearly every host (dev,
staging, and prod) is in a single 'production' Puppet dynamic
environment. Rather than having puppet environments that correspond to
physical environments, the single production Puppet environment
represents the stable tested version of our Puppet code and Hiera
config. That is, the way we deploy a dev or staging box, should be
exactly the same as the way we deploy a production box. Where we are
developing or testing a puppet module, we create a new feature
branch/environment, and move a host to that environment for testing.
Then once the change is tested, the feature branch is merged into
production and applied to all hosts, and the testing host is moved back
into the production environment. The idea here is that environments
other than 'production' are generally for short-term use.

Because all hosts are on the same Puppet environment most of the time,
we capture the differences between them in Hiera. Each of our hosts has
a site code, which captures the host's geographic location and physical
environment in a short alphanumeric id. This id is exposed as a fact,
and incorporated into our Hiera hierarchy so we can target configuration
to all hosts in a particular site. As with modules, config changes can
be tested in a temporary feature branch before merging to production,
and we can also use the Hiera hierarchy to scope changes to individual
hosts, whole sites, or to all environments.

Gary Larizza describes a similar model, more cogently than I have, in a
blog post here:
http://garylarizza.com/blog/2014/03/26/random-r10k-workflow-ideas/

I'd be interested to hear from anyone who is already using such a model,
and particularly if you've run into any problems with it.

--
Richard Gray


_____________________________________________________________________________

This email has been filtered by SMX. For more info visit http://smxemail.com
_____________________________________________________________________________

[Joao Morais]

On 04/10/15 18:23, Richard Gray wrote:
>
> On 03/10/15 13:08, Joao Morais wrote:
>> How to manage a Puppet Server with multi-environment and multi-system?
>

> Rather than having puppet environments that correspond to
> physical environments, the single production Puppet environment
> represents the stable tested version of our Puppet code and Hiera
> config.

Hi. Your model sounds interesting and I think that we would use
something like that in our company. Our main problem is that we want to
manage a really big bunch of unrelated systems. An example: something
will change in the system-A configuration, so this change is (or these
changes are) versioned in branch devel. After tests the change(s) should
be merged to prod, but only changes made to system-A. Since devel is
shared between other systems I cannot do it in an automated or a non
error prone way.

We are planning to split every system configuration into different repos
in order to have a more cohesive repository, but we don't know yet if
this is the right move and how to do it in an easy to manage way.

> Gary Larizza describes a similar model, more cogently than I have, in a
> blog post here:
> http://garylarizza.com/blog/2014/03/26/random-r10k-workflow-ideas/

Yeah, our r10k model started with these and other Garry's readings.

[Joao Morais]

On 04/10/15 13:36, Maxim Nikolaev wrote:
> Can you split repositories for manifests and config?

Do you mean manifests and hieradata in the same repository, but split
into one repository per system? We can do it and this is currently the
move we are trying to achieve. Problem is that we don't know yet if this
is the right move and how to implement it in a nice way. We want to
share the same Puppet Server to all of the managed systems.

[Garrett Honeycutt]

Hi Joao,

What do you mean by base configuration and system configuration? Also,
are you using separate repo's for each module?

Best regards,
-g

--
Garrett Honeycutt
@learnpuppet
Puppet Training with LearnPuppet.com
Mobile: +1.206.414.8658

[Joao Morais]

On 05/10/15 21:22, Garrett Honeycutt wrote:
> On 10/2/15 8:08 PM, Joao Morais wrote:
>>
>> How to manage a Puppet Server with multi-environment and multi-system?
>

> Hi Joao,
>
> What do you mean by base configuration and system configuration?

Hi Garrett. Base configuration is common manifests and hieradata shared
by all installations of all systems, like hardening and Apache httpd.
There are some corporative rules achieved by this base configuration.

> Also, are you using separate repo's for each module?

Yes. The `environments` repo has manifests, hieradata and a `Puppetfile`
file used by r10k to manage modules.

[Joao Morais]

On 06/10/15 07:25, Joao Morais wrote:
> On 05/10/15 21:22, Garrett Honeycutt wrote:
>> On 10/2/15 8:08 PM, Joao Morais wrote:
>>>
>>> How to manage a Puppet Server with multi-environment and multi-system?
>>
>> Hi Joao,
>>
>> What do you mean by base configuration and system configuration?
>
> Hi Garrett. Base configuration is common manifests and hieradata shared
> by all installations of all systems, like hardening and Apache httpd.
> There are some corporative rules achieved by this base configuration.

... and about system configuration: out Puppet Server manages several
VMs hosting software for several different and unrelated clients. A
system configuration is something like JBoss properties, datasources,
logging and other system related configurations which we need to promote
from one environment to another.