Hiera/Puppet: How to handle different versions of applications in multiple environments?

[Devminded]

Hello everybody.

I'm a developer new to puppet and are working with a complex system made up of several subsystems. We have regulatory requirements which forces us to have several production environments (at-least one per jurisdiction). This is causing us some pains.

The main problem is that the different jurisdictions "requires" different versions (and sometimes content) of the deployed applications. Due to a drawn out certification process some environments lags behind in versions by up to six month (that's how long it can take to get changes certified). Given that different environments require different versions of both puppet modules and application versions how do we handle this while still guaranteeing system integrity?

I'm in the process of trying out some other things, like the roles and profiles pattern. Looking at the 'version' properties in my example below; is it a good idea to keep it hardcoded and release a new version of the puppet module for each version of the software (order_service in this case) and assign those modules per environment or should I make the application versions a hiera variable as well? And if I do decide to keep versions in hiera how do I ensure that no-one (in ops) deploys a version that has not been integration-tested with the rest of the system?

Keeping versions in hiera would make everything much more dynamic but also so much more difficult to ensure consistency, especially in a Continuous Delivery workflow...

Any takers? What obvious thing did I miss?

*Example*

class role::app_server {

    include profile::base

    include profile::linux

    include profile::jboss

    include profile::order_service::webapp

}

class profile::order_service::webapp {

    class { 'order_webapp'

        version             => '3.1.2',

        db_address          => hiera('db_address'),

        etc...

    }

}

class profile::jboss {

    class { 'jbosseap':

        version             => 6.3,

        port                => hiera('port'),

        broadcast_address   => hiera('broadcast_address'),

        user_roles          => hiera('user_roles'),

        user_groups         => hiera('user_groups')

    }

    class { 'java'

        version             => '>=1.7.0'

    }

}

[Pete Brown]

Hi,

I think the best way to achieve this is to use class variables for all
the versions of packages you want to manage and use hiera as the
backend.
You can actually use the $::environment fact in your hiera.yaml file
when defining the datadir.

I tend to put my hiera tree in a separate repository but you could
just as easily put it in your environment repository,

I would also recommend using r10k to manage your environment and hiera
checkouts.

Does that all make sense?

Pete.

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/06c4d567-6785-45ec-ae4a-8a41c236d2fe%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Devminded]

Hi Pete.

I ended up doing just that, class variables for versions of applications backed by Hiera. I am trying to avoid putting %{environment} in Hiera to keep different jurisdictions from interfering with each other. I just use directory environments for both Hiera data and puppet modules.

Now I did end up with the issue I was afraid of: A messy mix of infrastructure data owned by Ops (IP-addresses, nodes, clustering, firewall, etc) with all the application data that is owned by delivery (what things are tested and integrated together).

I'm thinking that my hierarchy is wrong and will try to separate the application versions from infrastructure data and retrieve them from separate repos. I'm fairly new to Hiera so I don't know if that will even work... will have a look into r10k as well.

Something like this:

---

:hierarchy:

  - "%{::role}_infrastructure"

  - "%{::role}_versions"

  - "..."

  - "common"

Wish I learned all this "Ops" stuff years ago instead of fumbling in the dark...

[Jake Lundberg]

When it comes to split responsibilities, such as that between the Ops teams and Dev teams, we usually do a couple things:

1.  Use hiera to store key/value pairs that can be visible worldwide (like application versions).  This allows developers to be able to test their own stuff, and when ready, submit pull requests into the production environment branches

2.  Use node scoped variables for Ops-only information

3.  Make Ops the keeper of facts and secrets in the node definitions

4.  Structure our hierarchy to include applications, datacenters and/or business units (ours is actually more complex, but this is simplified)

5.  Parameterize all classes we wish to use hiera for.  

5.1 In some cases, explicitly call hiera based on availability of node scoped variables (this is more for pre-hiera backwards compatibility)

6.  Use folder based environments (though it seems Puppet suggests moving away from these, they give us more flexibility).   We use a script we found on the github called branch2env.py (slightly modified to fit our environment better).   I haven't used r10k, but it sounds like it does something similar.

The actual node definition might look something like:

node vars-dc1-finance {

  # This is the "folder" for puppet/hiera code to look at

  $environment = 'production'

  # These are "facts" that aid in hiera lookups

  $datacenter = 'dc1'

  $biz = 'finance'

  # This is a node scoped secret

  $order_service::webapp::password = 'foofoo'

}

node finance-webapp.dc1.com inherits vars-dc1-finance {

  include base

  include order_service::webapp

}

A basic hierarchy might be:

- nodes/%{::fqdn}

- dc/%{datacenter}

- biz/%{biz}

- common

Some example hiera yamls:

dc/dc1.yaml:

order_service::webapp::endpoint1: 'https://endpoint1.dc1.com'

base::yum_repo_server: 'repo.dc1.com'

biz/finance.yaml:

java::version: '1.7.0_60'

order_service::webapp::version: '1.2.3'

biz/hr.yaml:

java::version: '1.6.0_30'
order_service::webapp::version: '1.0.0'

common.yaml:

#This pulls from a node scoped variable

order_service::webapp::password: %{order_service::webapp::password}

# These are defaults in the event something higher in the hierarchy does define them.  

java::version: '1.6.0_18'

order_service::webapp::version: '1.0.0'

Now for some basic class definitions:

class order_service::webapp ($version, $password, $endpoint1) {

  include java

  package { "order_service-${version}" :

    ensure => installed

  }

  file { "/etc/creds/order_service" :

    content => "${password}"

  }

  # This template uses the $endpoint1 variable via <%= @endpoint1 %>

  file { "/etc/overrides/order_service":

    content => template("order_service/webapp/overrides.xml.erb")

  }

}

class java ($version) {

  package { "jdk-${version}" :

    ensure => installed

  }

}

Our hiera and modules are environment aware.   And environment is stored in puppet.conf via an erb.

hiera.yaml:

:datadir: "/etc/puppet/hiera/%{environment}"

puppet.conf:

 environment = production (This is added via an erb template that reads the node scoped $environment variable)

 modulepath = /etc/puppet/modules/$environment

Our nodes are included in site.pp from a predefined location.   This allows us to do a couple things:  

1. Keep tighter control of production assets 

2. Allow developers to add in node definitions in cases this is necessary (mainly due to old policies).

e.g. 

import /etc/puppet/nodes/prod/* (points to a for-Ops-only repo)

import /etc/puppet/nodes/dev/*  (points to a developer accessed repo)

We went through a few iterations of getting this right and figuring out how to reduce duplication of data as much as possible.   Just try to remember a few things:

1.  Try to keep your node definitions as simple as possible.  When possible only add facts not derived from facter to node definitions.

2.  Keep configuration data out of your classes.  Even defaults.  Use common.yaml or similar to add in defaults. 

3.  Unless absolutely necessary DO NOT use resource style declarations (e.g. class { "java": version => '1.7.0_60'}).  Hiera handles parameter lookups quite well and you won't get duplicate declaration issues.

HTH,

Jake