How to do release managment integration with puppet?

[Schofield]

I am at the beginning of merging the traditional OS/Middleware update process with application development release process.  The goal is to be able to test a complete versioned OS/Middleware/Application stack as it moves through dev/test/qa/prod environments.  The key here is that entire stack is labeled under a single version. In other terms, the node and everything on it is aligned under a single version. Right now the OS/Middleware is managed by puppet.  The application release process is not.  I'm hoping some folks would like to comment on the following questions I am pondering.

• How to best version puppet classes and control the classification of nodes based on the version of the node?
• Are there ENC's that integrate with SVN/GIT that make this easy?
• Should application deployment be brought under the control of puppet?
• If so how do you easily do this without burdening applications developers with having to learn something about or have access to puppet?
• Is there a puppet API that an existing application release processes can leverage to trigger OS/Middleware updates when an application version is updated?

Looking forward to your comments.

[Craig White]

----
I think the general consensus is that puppet is less than optimal for most software deployments and there are better tools for the job (for example, we use capistrano here).

Puppet is more of a configuration management tool and I suspect that even if you could shoehorn your software deployment needs into something that puppet could manage, it would probably be slow and confusing.

As for ENC's and 'developer' access, I use foreman (http://theforeman.org) to provide 'other' user access and I can contain their rights/access within foreman and yet still, allow them to say install NGINX (with all of it's pre-requisite packages) from a web based GUI.

Again, seriously look at capistrano for handling software deployments and I'm sure that there's others that can do the job too.

Craig

[Gareth Rushgrove]

On 30 November 2012 23:02, Craig White <craig...@ttiltd.com> wrote:
>
> On Nov 30, 2012, at 10:37 AM, Schofield wrote:
>
>> I am at the beginning of merging the traditional OS/Middleware update process with application development release process. The goal is to be able to test a complete versioned OS/Middleware/Application stack as it moves through dev/test/qa/prod environments. The key here is that entire stack is labeled under a single version. In other terms, the node and everything on it is aligned under a single version. Right now the OS/Middleware is managed by puppet. The application release process is not. I'm hoping some folks would like to comment on the following questions I am pondering.
>> • How to best version puppet classes and control the classification of nodes based on the version of the node?
>> • Are there ENC's that integrate with SVN/GIT that make this easy?
>> • Should application deployment be brought under the control of puppet?
>> • If so how do you easily do this without burdening applications developers with having to learn something about or have access to puppet?
>> • Is there a puppet API that an existing application release processes can leverage to trigger OS/Middleware updates when an application version is updated?
>> Looking forward to your comments.
> ----
> I think the general consensus is that puppet is less than optimal for most software deployments and there are better tools for the job (for example, we use capistrano here).
>

I'm not sure 'general consensus' is really true, I think it depends,
in particular on size of stack. Capistrano is incredibly easy to get
started with but has negatives too (like being push based, often
installing/compiling dependencies per machine, not being able to
specify broader version dependencies like on the database server)

> Puppet is more of a configuration management tool and I suspect that even if you could shoehorn your software deployment needs into something that puppet could manage, it would probably be slow and confusing.
>

Here's a question. Would you define a puppet module for Graphite?
Graphite turns out to be a Django base web application. Would you use
Capistrano to deploy Graphite? If your own applications were Django
based would you use Cap? Not quite as simple I feel.

Replying the the original points separately too.

G

> As for ENC's and 'developer' access, I use foreman (http://theforeman.org) to provide 'other' user access and I can contain their rights/access within foreman and yet still, allow them to say install NGINX (with all of it's pre-requisite packages) from a web based GUI.
>
> Again, seriously look at capistrano for handling software deployments and I'm sure that there's others that can do the job too.

>
> Craig
>

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>



--
Gareth Rushgrove
@garethr

devopsweekly.com
morethanseven.net
garethrushgrove.com

[Gareth Rushgrove]

On 30 November 2012 17:37, Schofield <dbsch...@gmail.com> wrote:
> I am at the beginning of merging the traditional OS/Middleware update
> process with application development release process. The goal is to be
> able to test a complete versioned OS/Middleware/Application stack as it
> moves through dev/test/qa/prod environments. The key here is that entire
> stack is labeled under a single version. In other terms, the node and
> everything on it is aligned under a single version. Right now the
> OS/Middleware is managed by puppet. The application release process is not.
> I'm hoping some folks would like to comment on the following questions I am
> pondering.
>
> How to best version puppet classes and control the classification of nodes
> based on the version of the node?

Although I've not done it myself I've seen people use Puppet
environments for versioning. You can also use environment variables on
the nodes to help with classification.

> Are there ENC's that integrate with SVN/GIT that make this easy?
> Should application deployment be brought under the control of puppet?
> If so how do you easily do this without burdening applications developers
> with having to learn something about or have access to puppet?

The vcsrepo module (https://forge.puppetlabs.com/puppetlabs/vcsrepo)
supports defining git/bzr/svn checkouts, but another approach would be
to package your applications up as system packages (debs for
debian/ubuntu, rpms for redhat/centos). This has a number of nice
advantages, including making managing the application version just a
matter of defining a package resource in puppet.

> Is there a puppet API that an existing application release processes can
> leverage to trigger OS/Middleware updates when an application version is
> updated?

So, assuming you do package your applications up as system packages as
described then triggering updates to supporting components could be
done in the package dependencies, or it could be done with Puppet
requires, assuming the dependencies are on the same node. Cross node
dependencies are harder however. Some ideas for that thorny problem:

https://github.com/jamtur01/puppet-noah/
http://www.slideshare.net/lusis/beyond-the-node-arkestration-with-noah

G

>
> Looking forward to your comments.
>

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/vAGrAj_5qpUJ.

[Nick Cammorato]

Right now we're deploying via cap and something custom and part of the deployment is a config yaml file which puppet parses via facter in order to determine how to do the OS/middleware config.  It works fairly well, but it feels kludgy to do it this way.

I've been wanting to cook something up like what it sounds like you want for a while, since I hate push-based models, and I hate using ssh as a transport.  This means there is almost nothing that makes me happy around right now.

The basic idea is to have a web application that handles deployment, maintains a version to revision database(one of our old systems uses SVN and does this via tags, which gets, well, bad after a few years), plus metadata surrounding the release, like configuration information.  Deployment is done by issuing an mcollective command to do a pull/clone/checkout/update/whatever from a VCS onto the target server.  Config information is transferred into puppet via the master over REST from the deployment server using the ruby DSL, with everything stored in one big hash.  You would require that module and then use those variables in your own modules, with a sanity check to verify the code actually updated.

[j4m3s]

I run a small startup offering a SAAS application (cloud based, no local servers at all).  We have always tried to "do things right" so that we can scale - and my view is (happy to be challenged) that our application stack should absolutely be managed just as any other part of the configuration.  We package our software components using "system packages" (deb files on ubuntu in our case).  There was a time when we created the debs manually, but our software build process does them automatically now using git-buildpackage.  the Continuous Integration server gets the latest changes from source control, tests them and creates the debs.  deb packages handle versions, so if we want we can specify that one part of the app depends on a particular version of another part.  As far as possible though we try to release the application stack as a whole so we avoid untested combinations.  (this is a key recommendation of "the" Continuous Delivery book by the thoughtworks guys).

The version numbers of the packages in the different environments is managed by puppet, using hiera.  The contents of the config files are also managed by puppet (again using hiera).  As far as I can tell this is the perfect scenario - I'm really interested to know what problem Noah is referring to/ trying to solve (we haven't found it yet).  If we need more capacity, we build another prod app server or db server etc.  The app layers support dynamic load balancing - e.g. app servers automatically register themselves with the web server pool when they start-up.  The web-servers automatically register with the load balancer etc. 

Being open with you, we could probably manage without all of this (for now) - we haven't achieved the scale to need most of it yet.  But having it like this means we spend much much less time worrying about environments and software releases and more time getting the software right for the users.  I'd love to hear what problems we're heading for that this setup can't cope with, so we can change course if necessary.

Regards, James.

[Jakov Sosic]

Excellent post James!

That's the way to do it.



--
Jakov Sosic
www.srce.unizg.hr

[Schofield]

Thanks for the great response James.  I think you have done exactly what I am looking to do.  I want to ask more about controlling the versions of the application packages in hiera.  I have multiple different dev groups who each work on different release schedules.  I don't want to have operations/puppet folks involved with every release as application versions move through dev/test/prod.  So how would one go about letting dev teams control which versions of applications are mapped to which environments?

[Brendan O'Bra]

This is a great thread - it's really nice to hear others are pondering similar challenges ;)  

Count me in as "Puppet does this well, I'm unclear what the issues are"

I'm also in the process of pondering this very process.  Here's what I've come up with (which seems very similar to James'):  

We are a Redhat/Centos shop, and our release artifacts are packaged and delivered as RPMs. The RPMs are built as part of our Continous Integration activities, and deployed as versioned artifacts to a Nexus repository.  Puppet is then used to retrieve the versioned artifacts to create a Yum repository.   Controlling which version of which artifact is accomplished by pointing the puppet agent nodes at the Yum repo that contains the version of the artifacts that are desired to be installed in a particular environment.  We also release as a "platform" BTW.

I'd also add that using standard packaging (RPMs) forces us to grapple with dependencies and coupling (in a good way) between components, and allows a very clean use of Puppet (via the Yum Provider).

My .02 ;)

On Mon, Dec 3, 2012 at 5:57 AM, Schofield <dbsch...@gmail.com> wrote:

Thanks for the great response James.  I think you have done exactly what I am looking to do.  I want to ask more about controlling the versions of the application packages in hiera.  I have multiple different dev groups who each work on different release schedules.  I don't want to have operations/puppet folks involved with every release as application versions move through dev/test/prod.  So how would one go about letting dev teams control which versions of applications are mapped to which environments?

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/nlqpx5hjPX4J.

To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

--
GVoice: 707.410.0371 

LinkedIn: http://www.linkedin.com/in/brendanobra

[Schofield]

What I am understanding is that people are having success building/testing/deploying the OS/Middleware/Application stack by wrapping applications in a puppet resource.  Thus bringing all configuration management under puppet.  This is good as it is the direction I wanted to go.  I would like to shift the direction of the discussion a little bit to address how the application versions are updated in puppet.  James mentioned updating hiera and Brendan mentioned pointing to a different yum repo.  I am sure there are multiple other ways to solve this same problem as well.  But what these solutions have in common is that an individual at some point in time has to make some change to the puppet system.  Is there any tooling or API that can be leveraged to allow these changes to be made and control who makes them?   This would be for integrating the update to puppet in to our enterprise release management processes. 

[Brendan O'Bra]

" But what these solutions have in common is that an individual at some point in time has to make some change to the puppet system."

Please elaborate - Do you mean that someone has to "tell" Puppet about what versions to use?  My ultimate solution will be emitting a manifest containg the names and versions that go into a "release" as part of our CI/release activities that is used to "tell" Puppet what a "platform release" consisiste

On Mon, Dec 3, 2012 at 8:25 AM, Schofield <dbsch...@gmail.com> wrote:

What I am understanding is that people are having success building/testing/deploying the OS/Middleware/Application stack by wrapping applications in a puppet resource.  Thus bringing all configuration management under puppet.  This is good as it is the direction I wanted to go.  I would like to shift the direction of the discussion a little bit to address how the application versions are updated in puppet.  James mentioned updating hiera and Brendan mentioned pointing to a different yum repo.  I am sure there are multiple other ways to solve this same problem as well.  But what these solutions have in common is that an individual at some point in time has to make some change to the puppet system.  Is there any tooling or API that can be leveraged to allow these changes to be made and control who makes them?   This would be for integrating the update to puppet in to our enterprise release management processes. 

--

You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/EsJpWatvCU0J.

To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[Schofield]

On Monday, December 3, 2012 11:34:53 AM UTC-5, Brendan O'Bra wrote:

" But what these solutions have in common is that an individual at some point in time has to make some change to the puppet system."

Please elaborate -

Do you mean that someone has to "tell" Puppet about what versions to use?  My ultimate solution will be emitting a manifest containg the names and versions that go into a "release" as part of our CI/release activities that is used to "tell" Puppet what a "platform release" consisiste

Exactly.  In your case that 'someone' is the person doing the CI/release activities and the update to puppet is done by generating the manifest as part of the workflow.  

Evaluating this problem for the first time, there seems to be lots of potential solutions for generating the manifest with updated application versions or new applications altogether. Best practices for process release integration may not be fully established yet and everyone is doing their own thing based on what works for them. 

[James Fellows]

Though we have lots of automated testing we still rely on the human touch for checking UI look and feel, and doing exploratory testing (in the pre production environment).   Once it has passed this step the human can approve it for production deployment by updating the version number in hiera. I guess we could have a button on a web screen or something to make it easier to promote it, but editing hiera directly is easy enough.  If you have a central change management system, it could update hiera - but you'll still want it approved for promotion by a human won't you, to take into account live service stability/ load, etc?

James

[Ramin K]

We have a set of machines that we spin up per event. Use a custom AMI
and let Puppet add the final 1-2% of config to catch anything that's
changed recently. Puppet runs only once and the machines spin down after
4-10 hours. The custom app code doesn't change often which seemed like a
decent candidate for letting the Dev team push versions to Puppet.
Here's how it works

Hiera can't search two different dir trees for the same backend so I
set up a second backend, json. Mostly do this so I have final say via
yaml on versions/config. Probably overkill.

hiera.yaml
---
:hierarchy:
- %{clientcert}
- %{environment}/%{role}/%{calling_module}
- %{environment}/%{role}
- %{calling_module}
- %{role}
- %{environment}
- common

:backends:
- yaml
- json

:yaml:
:datadir: '/puppet/etc/environments/%{environment}/hieradata'

:json:
:datadir: '/puppet/data/%{environment}/hieradata'

/puppet/data/production/hieradata/mycode.json
{"mycode_ensure":"latest","mycode_version":"somebranch-12-11-27"}

When Dev does a release they update a set of core machines via
Capistrano and then push the new version into Puppet do the ephemeral
machines will check out the latest version when they spin up.

I've been meaning to extend the idea as well as move to more of a pull
method for software release, but it hasn't been pressing.

Ramin

[Schofield]

On Monday, December 3, 2012 2:32:15 PM UTC-5, j4m3s wrote:

Though we have lots of automated testing we still rely on the human touch for checking UI look and feel, and doing exploratory testing (in the pre production environment).   Once it has passed this step the human can approve it for production deployment by updating the version number in hiera. I guess we could have a button on a web screen or something to make it easier to promote it, but editing hiera directly is easy enough.  If you have a central change management system, it could update hiera - but you'll still want it approved for promotion by a human won't you, to take into account live service stability/ load, etc?

Do you use hiera to define not only the versions but also which applications should be on which environments?  How do you introduced application version 1.0 to dev for the first time?

[James Fellows]

Sorry, I have only really given you part of the story haven't I.   We have a puppet class hierarchy that we use to specify the various components that need to go on a particular type of host - e.g app server or web server or db server. A dev machine includes several roles (machines are identified by a -dev suffix on the hostname but maybe we could do it based on the environment variable?).  

We use the puppet environment variable to say which environment the machine is in.   That in turn decides the config variables and package versions.

Dev machines do require additional manual intervention.  The deb packages get installed as normal (version "latest" or whatever) but the app component deb installers install themselves plus a symlink pointing to the deb install location ( e.g. /usr/lib/myapp/current -> /usr/lib/myapp/deb).  Because the deb installers won't overwrite files that have been modified externally, the developers can point the symlink (/usr/lib/myapp/cuurent) for the relevant component to point to their local checkout/ compile directory and they can be working on the code while the rest of their installation keeps up to speed with other package versions and configs.  They need to copy the puppet-created config file for the particular app  into their working directory and symlink from its normal /etc/myapp location) but then they can modify it etc as part of the dev process.

I know I'm making a mess of explaining this, sorry - feel free to point out bits that either aren't clear or where we could improve our process!  

----- Reply message -----
From: "Schofield" <dbsch...@gmail.com>
To: <puppet...@googlegroups.com>
Subject: [Puppet Users] Re: How to do release managment integration with puppet?

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/5n3FDDQjhlYJ.