Versioning multiple components

[jdtangney]

We are extending our Continuous Delivery process and I am looking for advice about managing and installing/rolling back across multiple hosts.

Each release consists of several components, each of which is installed by a Debian package (.deb file.) Ours is a distributed system, so different packages are installed on different hosts. But they all need to stay in sync, meaning we need to move a release through the pipeline as a whole. In other words, all hosts need to be brought up to the same release level by installing a group of packages. When things go wrong, all those packages need to be rolled back across all hosts in order to revert to the previous release.

Do you have any thoughts about how we should approach this? Are there tools available that can help or should we roll our own? I have heard that tools like Artifactory and Nexus Artifact Manager but I know nothing about them. 

Apologies for the vague and open-ended question.

--johnt

[Diptanu Choudhury]

A couple of questions, do you have a Deployment Pipeline already? If not then you will have to think about constructing one so that the artifacts can be deployed automatically to different environments. If you do, then first of all you will have to look at your current deployment pipeline, does it always aid in moving forward by checkout out code from the source control and progressing the built artifacts all the way through to your different environments? Or does your deployment pipeline allow you to deploy any arbitrary version of your application?

Rollbacks are trickier because they involve restoring the state of the environment to the exact same state where a previous version of the application was. I think it would be better if you provide some more information about your current state of CD.

We have been doing all this quite successfully for a long time now.

--
Thanks,
Diptanu Choudhury

Senior Consultant, ThoughtWorks
Mobile - +447808024919
Web - www.linkedin.com/in/diptanu

Twitter - @diptanu

[Yaakov Nemoy]

When we have a very complicated deployment like this, we simplify our
deployment pipeline by building up a parallel network with all the
correct versions installed. The actual deployment is pointing the load
balancers to the new parallel network. Rollbacks are simple. When the
deployment is done, we just throw away all the old hosts from the old
environment.

[Ryan Miller]

Do you need to interact with loadbalancers and database schemas for rollout (you only mention .debs).  If  you're really just dealing with installing/rollingback set package versions across multiple server roles and environments (with dependencies) then either chef or puppet will do this very well, with many other potential benefits besides.  Puppet has explicit support for environments, roles, dependencies, packages, and global versions.  If you need to do stateful things with loadbalancers and databases, then you might need to get into capistrano, controltier, or mcollective, potentially using them to trigger puppet runs as well as interact with other components.

Ryan

[Nathaniel Eliot]

I've used similar techniques to Yaakov, with good results. The
combination of IaaS and server build automation (Chef/Puppet/etc) is
really powerful; recapitulation and data migration are often *far*
less difficult and worrisome than doing an in-instance
upgrade/rollback cycle. Doing recaps semi-regularly also reduces the
problem of cruft left over from previous solutions, making it easier
to reason logically about the current state of systems.

At Infochimps we're still feeling the pain of poor packaging, and are
currently leaning toward Artifactory to help address that. Keeping
versions together, however, is an essentially solved problem; we
version our cookbooks, manage versions with environments, and use
those cookbooks to manage versioning for everything else as necessary.

--
Nathaniel Eliot
T9 Productions

[Juan Morete]

Here we are using .deb as artifacts and chef for keeping them in sync and also to manage configurations... we do version our cookbooks and keep version numbers on attributes also using environments to set the cookbook version to be used... this is not yet taking care of loadbalancers and schema changes as we are still implementing it... from nothing to .deb + chef there is a huge difference specially if you have to rebuild servers from scratch or simply add extra instances to add more capacity... really powerful.

Juan Morete

[Yaakov Nemoy]

Something we do to handle the data migrations is ask the devs to patch
the old version to use the same data migration. Then we pre apply the
data migration before the rest of the deployment. This way, rolling
back does not mean rolling back schema changes too.