Package based web application deployment workflow

[gareth rushgrove]

I'm playing with a package based deployment workflow for simple web
applications. It's a proof of concept at the moment so that means a
one file sinatra application and it's dependencies.

What I'm looking for is something that uses packages for everything (I
like packages), allows me to define everything including my
application in chef/puppet if I want AND provides an interface that's
as good as cap or similar. Basically I just want to bridge the divide:
http://blog.lusis.org/blog/2011/08/22/the-configuration-management-divide/

So far I'm:

1. Checking code in
2. Jenkins takes the code and builds a debian package with fpm,
including all the dependencies with bundler (a bit like a java war
file I guess). I'm changing the name of the package each version,
because I want multiple versions available at any one time (think
capistrano releases folder with a symlink). So I have webapp-v1,
webapp-v2, etc.
3. I'm also building a meta package, called webapp, which gets it's
version number incremented as you'd expect and depends on the relevant
named version package above. So package webapp version 1 depends on
webapp-v1, next successful build would create webapp version 2 which
depends on webapp-v2.
4. Jenkins then adds the packages to a local repository (using freight)

The ruby, debian, freight, jenkins, sinatra, bundler stuff is all just
detail. Replace with your technology/package type of choice.

The packages install the application as follows:

webapp-v1 -> /srv/webapp/v1/{application files}
webapp-v2 -> /srv/webapp/v2/{application files}
webapp-v3 -> /srv/webapp/v3/{application files}

I'm pretty happy with that, even if the version number in the package
name feels odd.

My question is about what should own the version of the application
which is currently live, which comes down to what should own a current
symlink pointing to one of the version folders above. e.g.

/srv/webapp/v1
/srv/webapp/v2
/srv/webapp/current <- symlink to /srv/webapp/v2

As far as I see it, it could be owned by:

1. The meta package (ie. webapp) via a post install script
2. The versioned package (ie. webapp-v2) via a post install script
3. Nothing to do with packages, managed only via configuration management

I'm leaning towards the meta package owning the symlink as it allows
for a few useful features:

* rollback to a previous version by installing a previous copy of the
meta package
* rollout the new version of the code (in webapp-vx) ahead of making
it live (install webapp)
* both of the above can be easily controlled via puppet/chef
* I can set my config management system to use the latest version of
webapp and deploys become just a matter of triggering a run across the
relevant nodes (say with mcollective/cap)
* Or I can control the explicit version in my recipes/manifests by
specifying a version of webapp to be installed
* You could also easily drive this via a local client (ala capistrano)
if you preferred. remotely execing aptitude search | grep webapp
pretty much gives you available versions and triggering deploys is
just a package install.

What I'd appreciate is basically a sanity check. In particular what am
I obviously doing wrong?

Cheers

Gareth

--
Gareth Rushgrove
Web Geek

morethanseven.net
garethrushgrove.com

[Noah Campbell]

Version numbers in the package name seems strange, but makes sense. Think of python24, python25, python31, etc but just at a higher frequency.

Anything in post install can be complicated given that you're overwriting something on the filesystem. Rollback becomes more complicated because before writing your symlink you'll need to preserve it…what happens if something goes wrong with that process. Using something like knife, mcollective, pssh, dssh, rundeck or something is preferable. Consider the changing of the symlink an operational or orchestration step which is orthogonal to the package installation. In your current setup, you may have only a handful of machines, but if that number grows and they're sitting behind a load balancer, you don't want the changing of the symlink to be happening at random times due to package installation latency or package installation failure.

-Noah

Noah Campbell
415-513-3545
noahca...@gmail.com

[Jim Browne]

On Mon, Aug 29, 2011 at 3:12 PM, gareth rushgrove <gareth.r...@gmail.com> wrote:

1. The meta package (ie. webapp) via a post install script
2. The versioned package (ie. webapp-v2) via a post install script
3. Nothing to do with packages, managed only via configuration management 

I'm leaning towards the meta package owning the symlink as it allows
for a few useful features:

Your leaning is correct IMO and based on my experience.  At CNET everything was an RPM, all apps were in versioned RPMs, and there was a (name escapes me) RPM that handled the sym-links.  CNET had a system called the Software Factory (SWF2) that took this even further and allowed one to install multiple versions of an app on multiple different ports and have them all running simultaneously (e.g. Apache13 on 80, Apache20 on 81, Apache22 on 82, etc.) and deployed all based on package name.

It required a lot of work to bring standard apps (e.g. Apache, Resin, Tomcat, etc.) into that scheme, but the flexibility you are sensing is there and does work well at large scale with a boatload of apps running concurrently.

You may want to go even further and add a layer of indirection between your webapps and the infrastructure it relies upon (e.g. being able to switch your app from launching Tomcat 5 vs. Tomcat 6 based on a simple package deployment).

[Richard Crowley]

On Mon, Aug 29, 2011 at 3:12 PM, gareth rushgrove
<gareth.r...@gmail.com> wrote:

Is the motivation for "incrementing" the package name the desire for
atomic deploys? If so, and especially if you're doing Ruby stuff
(since Ruby apps tend to aggressively `require` everything), I
wouldn't worry about that optimization. I've never met a truly atomic
deploy. By the time you get to thinking about assets, multi-request
scenarios, and multiple servers, it all falls apart.

By not worrying about that, you're free to just keep many versions of
the same package in the repository (something Freight can actually
handle, unlike reprepro) and rollbacks can be done via apt-get or your
favorite CM tool.

>
> My question is about what should own the version of the application
> which is currently live, which comes down to what should own a current
> symlink pointing to one of the version folders above. e.g.
>
> /srv/webapp/v1
> /srv/webapp/v2
> /srv/webapp/current <- symlink to /srv/webapp/v2
>
> As far as I see it, it could be owned by:
>
> 1. The meta package (ie. webapp) via a post install script
> 2. The versioned package (ie. webapp-v2) via a post install script
> 3. Nothing to do with packages, managed only via configuration management

My comments above notwithstanding, I try to avoid postinstall scripts
as they leave files on your system unaccounted for from a `dpkg-query
-S` (or equivalent) point of view. So I suppose regardless of whether
it's a symbolic link or the version number itself, I vote for the CM
tool owning this.

A fourth option, is to include the symbolic link directly in the
package, which still allows package upgrades/downgrades to deploy a
different version of the app and doesn't leave mystery on your
filesystem.

>
> I'm leaning towards the meta package owning the symlink as it allows
> for a few useful features:
>
> * rollback to a previous version by installing a previous copy of the
> meta package
> * rollout the new version of the code (in webapp-vx) ahead of making
> it live (install webapp)
> * both of the above can be easily controlled via puppet/chef
> * I can set my config management system to use the latest version of
> webapp and deploys become just a matter of triggering a run across the
> relevant nodes (say with mcollective/cap)
> * Or I can control the explicit version in my recipes/manifests by
> specifying a version of webapp to be installed
> * You could also easily drive this via a local client (ala capistrano)
> if you preferred. remotely execing aptitude search | grep webapp
> pretty much gives you available versions and triggering deploys is
> just a package install.

The only one of these that requires the indirection afforded by the
version-named packages is rolling out a new version of the code ahead
of making it live. Perhaps the whole scheme then hinges on how often
you expect to do that?

>
> What I'd appreciate is basically a sanity check. In particular what am
> I obviously doing wrong?

Sane, if a bit indirect.

Richard

[Brian Pitts]

On 08/29/2011 06:12 PM, gareth rushgrove wrote:
> The packages install the application as follows:
>
> webapp-v1 -> /srv/webapp/v1/{application files}
> webapp-v2 -> /srv/webapp/v2/{application files}
> webapp-v3 -> /srv/webapp/v3/{application files}
>
> I'm pretty happy with that, even if the version number in the package
> name feels odd.
>
> My question is about what should own the version of the application
> which is currently live, which comes down to what should own a current
> symlink pointing to one of the version folders above. e.g.
>
> /srv/webapp/v1
> /srv/webapp/v2
> /srv/webapp/current<- symlink to /srv/webapp/v2
>
> As far as I see it, it could be owned by:
>
> 1. The meta package (ie. webapp) via a post install script
> 2. The versioned package (ie. webapp-v2) via a post install script
> 3. Nothing to do with packages, managed only via configuration management

I like the approach you're describing.

For 1 or 2, why create the symlink via a post install script? You could
include the symlink like any other file in the package. Ubuntu's libc6
package does this, for example.

$ dpkg -L libc6 | grep /usr/lib64
/usr/lib64
$ readlink -f /usr/lib64
/usr/lib

Its best to avoid scripts if you can; handling all the situations in
which they might be run can get complicated.

http://www.debian.org/doc/debian-policy/ch-maintainerscripts.html

--
All the best,
Brian Pitts

[Martin Barry]

$quoted_author = "gareth rushgrove" ;

>
> The packages install the application as follows:
>
> webapp-v1 -> /srv/webapp/v1/{application files}
> webapp-v2 -> /srv/webapp/v2/{application files}
> webapp-v3 -> /srv/webapp/v3/{application files}
>
> I'm pretty happy with that, even if the version number in the package
> name feels odd.
>
> My question is about what should own the version of the application
> which is currently live, which comes down to what should own a current
> symlink pointing to one of the version folders above. e.g.
>
> /srv/webapp/v1
> /srv/webapp/v2
> /srv/webapp/current <- symlink to /srv/webapp/v2

I've seen something similar where puppet was doing an "ensure => latest" on
the meta package which meant that new versions were installed as required.
In fact the meta package depended on all valid releases so obsolete releases
could be removed with an "autoremove".

The symlink was managed by puppet so that changing a variable would adjust
which release a particular server was running. This could be rolled forward
and back without needing to mess further with the packages.

cheers
Marty

[Spike Morelli]

On 29 Aug 2011, at 23:12, gareth rushgrove wrote:

What I'm looking for is something that uses packages for everything (I
like packages), allows me to define everything including my
application in chef/puppet if I want AND provides an interface that's
as good as cap or similar. Basically I just want to bridge the divide:
http://blog.lusis.org/blog/2011/08/22/the-configuration-management-divide/

In the comments I think some folks made a point that what is needed to bridge the divide was something more push based rather than pull based, or in other words 'I want my code out there right now'. I'm not saying that is the right way to go, just making a point that even if the packaging is automatic and transparent to the devs it might not help bridging that divide.

3. I'm also building a meta package, called webapp, which gets it's
version number incremented as you'd expect and depends on the relevant
named version package above. So package webapp version 1 depends on
webapp-v1, next successful build would create webapp version 2 which
depends on webapp-v2.

we used to do something similar and eventually ended up creating a 'ticket' service that would give you a unique incremental number to be used for the versioning. It might be too complex for your setup, but it has some nice properties depending on what you need. The other thing we did at some point was to use the short hash as part of the name as doing a dpkg -l tells you right away what repository/changeset is installed.

But as other people said you could also just use the same name and different versions which are supported by freight.

1. The meta package (ie. webapp) via a post install script
2. The versioned package (ie. webapp-v2) via a post install script
3. Nothing to do with packages, managed only via configuration management

I'm leaning towards the meta package owning the symlink as it allows
for a few useful features:

that's what we used to do, and as others have suggested packaging the symlink is probably a better option. In many cases we ended up still needing post-install scripts and it worked, but then we weren't using anything like puppet/chef.

That said, in theory which app is running has nothing to do with deploying the app, so option 3. is what I would go for right now. As an addendum to that, would you be open to question the existence of the symlink in the first place? My rationale is the following: if you use CM why not just altering the front-end's config file to point to whatever is the app you currently want live? That will also allow you to have multiple apps running at the same time. This seems to also be more correct as if we agree which app is running has nothing to do with the app, the app dropping a symlink would conflict with that.

cheers,

Spike

--

http://www.spikelab.org/

http://twitter.com/spikelab

http://uk.linkedin.com/in/spikemorelli

[Nikhil M.]

@garreth

That is great idea and we had spiked something something similar with RPMs.

After a green CI build, the CI would upload the RPM to a repository.

Simultaneous deployments to a large number of nodes is very fast since it is under

Chef's (or Puppet's) purview to grab it from the fast-upstream RPM repo and provide it.

Another thing one can think of doing if the team is ruby savvy is to package the ruby-app

as a .gem and then deploy it using Chef or Puppet via an internal Gem Server.

Versions can easily be a part of the metadata inside of this rubygem.

Thanks,

Nikhil M.

--
Cordially,
Nikhil Mungel.

[gareth rushgrove]

Cheers for all the replies folks.

Still torn between just leaving the link in config management or
putting it into the package (cheers everyone who pointed out I didn't
need an evil post install script). But that appears to be because both
have and will work. The autoremove idea is also neat.

On 30 August 2011 09:32, Spike Morelli <f...@spikelab.org> wrote:
> On 29 Aug 2011, at 23:12, gareth rushgrove wrote:
>
> In the comments I think some folks made a point that what is needed to
> bridge the divide was something more push based rather than pull based, or
> in other words 'I want my code out there right now'. I'm not saying that is
> the right way to go, just making a point that even if the packaging is
> automatic and transparent to the devs it might not help bridging that
> divide.
>

My rationale is that push works until it doesn't. But the jump from
using cap off the shelf to rolling something bespoke is largish, and
importantly the interface sucks. What I want to get to is a nice
command line interface, say (not thought this through at all yet, just
for reference)

command package - create package of project, either local or on CI
command upload to web2 - if you're not using a package repo, allow for
local file upload
command list on db - list releases currently available to the db nodes
(ie. packages in repo or in local repo mode files on disk)
command deploy v3 to web - deploy version 3 of the package to the web nodes

The information on nodes could come from config files or a chef/puppet
interface.

The command to deploy could be a matter of an ssh loop, or pushed over
to mcollective/rundeck.

The point being I end up with a uniform interface for package based
application deploys going from a simple push model (create package
locally, upload, install) to a more complex setup (local package repo,
fancy deploy runs defined in mcollective/rundeck, node data in
puppet/chef). This wouldn't stop you just using the package and
defining stuff in your CM system. How the command line tool might (if
at all) be useful in that model I haven't yet considered.

> we used to do something similar and eventually ended up creating a 'ticket'
> service that would give you a unique incremental number to be used for the
> versioning. It might be too complex for your setup, but it has some nice
> properties depending on what you need. The other thing we did at some point
> was to use the short hash as part of the name as doing a dpkg -l tells you
> right away what repository/changeset is installed.

I'm using Jenkins for this at the moment, each successful build gets a
unique incrementing identifier.

G

--

[Spike Morelli]

On 30 Aug 2011, at 22:38, gareth rushgrove wrote:

Still torn between just leaving the link in config management or
putting it into the package

can we try to list the differences? I believe they are both valid approaches and it exclusively depends on what tradeoffs you pick. For example, do we agree on the statement "application deployment is not related to service's configuration"? and If we do, is using CM (or a configuration package) the logic consequence? etc.

My rationale is that push works until it doesn't.

I'd love to hear when in your experience push stops working, admittedly I'm very much used to pull and only had a few experience with push centric shops.

command package - create package of project, either local or on CI
command upload to web2 - if you're not using a package repo, allow for
local file upload
command list on db - list releases currently available to the db nodes
(ie. packages in repo or in local repo mode files on disk)
command deploy v3 to web - deploy version 3 of the package to the web nodes

sounds great, we did something similar for our image based systems so that developers had a simple cli to control provisioning and deployments. This is very much aligned with what is happening with PaaS cli so seems most definitely the right way to go.

The point being I end up with a uniform interface for package based
application deploys going from a simple push model (create package
locally, upload, install) to a more complex setup (local package repo,
fancy deploy runs defined in mcollective/rundeck, node data in
puppet/chef). This wouldn't stop you just using the package and
defining stuff in your CM system. How the command line tool might (if
at all) be useful in that model I haven't yet considered.

we didn't use the client at all to drive the ops part of the work, however we did integrate our provisioning system with that levering the same libs the client was built on to do things like inventory lookups.

[Martin Barry]

$quoted_author = "Spike Morelli" ;

>
> On 30 Aug 2011, at 22:38, gareth rushgrove wrote:
> >
> > My rationale is that push works until it doesn't.
>
> I'd love to hear when in your experience push stops working, admittedly
> I'm very much used to pull and only had a few experience with push centric
> shops.

The main issues are with "push only" scenarios where you now need to keep
state of which servers were down when you pushed (or servers need to trigger
a push when they come back up), your new server build process needs to
trigger a push etc.etc.

For those who like the timeliness of "push" what seems to work more reliably
is still putting it in configuration management and then mimic a "push" with
a "triggered pull".

cheers
Marty