Using Puppet for application deployment
Pete Emerson
server instances.
I do this by having a custom puppet node classifier that talks to a
database that defines what version of an application is supposed to be
on a particular node:
parameters:
application: "webapp",
webapp_version: "0.5",
webapp_config: "123",
webapp_symlink: "0.4"
My puppet recipe then makes sure that webapp version 0.5 is installed
(via yum and rpms), makes sure the right versioned configuration files
are in place, and makes sure that the "current" symlink points at 0.4
so that we can roll to a different version by flipping a symlink. The
puppet recipe has a lot of "requires" to make sure that the upgrade of
the application is graceful and does things in the right order.
My upgrade then goes like this, all controlled via a custom web interface:
1) Insert a jobgroup for the upgrade and a job for each server
instance into a db
2) A job processor then takes the first group of machines to upgrade,
changes their webapp version to the new version, and runs puppet on
them
3) Lather, rinse, repeat step 2 until complete (roll the application
out to prevent downtime), although if there is a puppet failure, all
pending jobs get cancelled.
Given that there are so many ways to skin the same cat, I'm wondering
how others are doing their application deployments using Puppet. Or,
if you are using Puppet but do your app deployments via some other
mechanism, how do you do it, and why aren't you using Puppet to do it?
Pete
Joe McDonagh
--
Joe McDonagh
AIM: YoosingYoonickz
IRC: joe-mac on freenode
L'ennui est contre-r�volutionnaire
Julian Simpson
I use capistrano for application deployment.
Given that there are so many ways to skin the same cat, I'm wondering
how others are doing their application deployments using Puppet. Or,
if you are using Puppet but do your app deployments via some other
mechanism, how do you do it, and why aren't you using Puppet to do it?
Pete
Michael DeHaan
Can you share some examples of where your dividing line is and some of
the tasks that are hard to handle in Puppet today?
Ideally I'd like to figure out how to incorporate more of that into
puppet-core. Bruce has been working on a version control type, that
could be used to check sources
out of version control and eliminate some packaging needs for hosted
apps, though I'm curious as to what other niches might not be covered.
--Michael
Julian Simpson
Can you share some examples of where your dividing line is and some of
the tasks that are hard to handle in Puppet today?
Ideally I'd like to figure out how to incorporate more of that into
puppet-core. Bruce has been working on a version control type, that
could be used to check sources
out of version control and eliminate some packaging needs for hosted
apps, though I'm curious as to what other niches might not be covered.
--Michael
--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
--
Julian Simpson
Software Build and Deployment
http://www.build-doctor.com
http://twitter.com/builddoctor
Michael DeHaan
> I've considered this for about ten seconds, but how about supporting some
> deployment tools in Puppet? So you could write a recipe that deploys
> middleware, application and other infrastructure for example.
Personally, I'd rather figure out what doesn't fit into the model well
than force a lot of external dependencies on other tools someone would
need to grok. The idea that Puppet automation and 'deployment' are
different shouldn't
be the case, IMHO. That all being said, there's nothing stopping
the creation of a Puppet type to run a deployment tool, I think that
if you did that, you're doing the analog of an exec task, and that's
not so interesting.
I do agree though -- Puppet needs to have examples of how to construct
manifests for middleware, applications, etc... though I don't think
that's too hard to do today provided things lurk on a single server,
and addressing the multi-server case is
something we'll eventually want to conquer.
I think there's a huge advantage of having all of your deployment + OS
+ config modelled in the same system.
(Then again, I also believe in creating RPMs where possible so you can
have package dependencies, but that's another story ... I know that
doesn't fit into everyone's workflow so we have to make it easier to
deploy things both ways. I think the VC
type will go a pretty good way towards that.)
--Michael
Joe McDonagh
Configure a system with apache2, a basic security configuration, some
enabled modules, a bunch of packages admins expect to have available
(screen, irb, etc).
Example of what capistrano can do best:
A. Deploy your puppet manifests to your puppet server.
B. Deploy your web content out of svn onto your web server.
They're two distinct areas of 'deployment'. Puppet is more like, systems
deployment, cap is app or content deployment in my eyes.
Joe McDonagh
awesome at ad-hoc command execution across multiple hosts. (Like your
func). Except I think func may scale to a lot more hosts since the
ssh-agent isn't so good at a lot of parallel connections.
Michael DeHaan
> Example of what capistrano can do best:
>
> A. Deploy your puppet manifests to your puppet server.
> B. Deploy your web content out of svn onto your web server.
>
> They're two distinct areas of 'deployment'. Puppet is more like, systems
> deployment, cap is app or content deployment in my eyes.
Yes, I want to change that observation.
For instance, there are pretty good examples of deploying rails apps
with Puppet, but it's not well shared. We similarly need to create
examples for other types of applications.
The Source Control Management type that Bruce is working on (should be
available soon?) covers your case B and should be tremendously helpful
there.
I believe the two should not necessarily be distinct and it's very
good to have them in the same model so you can have dependencies
between them.
This all goes down the line of doing minimal kickstart + Puppet.
I don't want Puppet to be like kickstart and reach a arbitrary
deployment line and stop -- it is too interesting for that.
--Michael
Julian Simpson
> They're two distinct areas of 'deployment'. Puppet is more like, systemsYes, I want to change that observation.
> deployment, cap is app or content deployment in my eyes.
Jesús Couto
Not that I understand it much :-P. But in general, I really didnt see why application deployment would be different for system deployment... except in operational terms. Things like "ok, new version of jar files on a load balanced enviroment, have to do one leg first, then the other, to ensure no outage...", that are very much procedural while Puppet manifest are more useful on a description of required software level. Cause you could do some weird thing like using a different class or parameter or something to update one leg first and then converging to the correct version, but at the default 30 minutes interval it would take you 1 hour to deploy, for example.
------------------------------
Jesús Couto F.
Pete Emerson
For our setup the version of software is exposed in the
puppet_node_classifier which connects to a database. Additionally, the
version of the configuration files and which puppet environment to use
are available as well. To do an upgrade, I change the version in the
db for a particular host to whatever I want it to be and then force
puppet to run. No waiting for 30 minutes, I can't not have control
over when the upgrade is going to happen. Rollback is an upgrade to
the previous version, and when rolling out to all of our production
boxes, our software modifies the database in slices and then verifies
that the upgrade was successful before proceeding to the next slice,
and if puppet fails to upgrade, all future boxes waiting to be
upgraded get canceled. I think it's a pretty slick system.
Pete
On Mar 12, 2:46 am, Jesús Couto <jesus.co...@gmail.com> wrote:
> I was asking about this months ago and was pointed tohttp://dev2ops.org/storage/downloads/FullyAutomatedProvisioning_White...
Michael DeHaan
> useful on a description of required software level.
Sort of.
The long story is that we don't have a really native feeling way to
model multinode deployments and workflow now, but we can think of
modeling it based on a set of checkpoint conditions.
Meanwhile you can see a bit of a preview here:
http://github.com/reductivelabs/puppet-external-resource
external_resource { "some remote condition":
frequency => 1,
timeout => 30,
check => "/some/external/script/that/will/return/true"
}
The trick is of course that the remote check software does not exist
as part of anything in Puppet core, yet, so that would be up to you.
However blocking until a database is present on a remote server is
actually possible now. (Provided you write the script to check for
it).
I would agree that the concept of taking a node out of a load balanced
configuration and modelling that configuration is more difficult to do
intelligently, as the system needs to understand a fair amount of
context
about your HA environment.
It's not so much of an issue of declarative vs procedural, but more so
a question of can the system model your configuration. Right now,
those systems today that are 'procedural' don't model at all, they are
more of raw code.
--Michael
Jesús Couto
> that are very much procedural while Puppet manifest are moreSort of.
> useful on a description of required software level.
The long story is that we don't have a really native feeling way to
model multinode deployments and workflow now, but we can think of
modeling it based on a set of checkpoint conditions.
--
------------------------------
Jesús Couto F.
David Schmitt
If you want to prototype something like this, you can use a define
outside of a node in the site.pp and use checks against $fqdn to
distribute resources among hosts.
Maybe even use the external_resource type that's currently floating
around to sequence the deployment.
Which leads me to another idea: inter-node dependencies:
| node a { mysql_db { "foo": ... } }
|
| node b { app { "x": after => A>>Mysql_db["foo"]; } }
(please ignore the crude syntax)
Best Regards, David
--
dasz.at OG Tel: +43 (0)664 2602670 Web: http://dasz.at
Klosterneuburg UID: ATU64260999
FB-Nr.: FN 309285 g FB-Gericht: LG Korneuburg
Joe McDonagh
>
>
>
> Which leads me to another idea: inter-node dependencies:
>
> | node a { mysql_db { "foo": ... } }
> |
> | node b { app { "x": after => A>>Mysql_db["foo"]; } }
>
>
>
David Schmitt
> David Schmitt wrote:
>>
>>
>>
>> Which leads me to another idea: inter-node dependencies:
>>
>> | node a { mysql_db { "foo": ... } }
>> |
>> | node b { app { "x": after => A>>Mysql_db["foo"]; } }
>>
>>
>>
> David, are you suggesting this, or are you saying that this works?
>
I'm suggesting something like this. Internally it'd probably work like
external_resource (see -dev list for details) but would query the
storeconfig DB instead of calling a script on the client.
Joe McDonagh
it. I am with you on the inter-node dependencies.
Michael DeHaan
> On 3/16/2010 6:13 PM, Joe McDonagh wrote:
>>
>> David Schmitt wrote:
>>>
>>>
>>>
>>> Which leads me to another idea: inter-node dependencies:
>>>
>>> | node a { mysql_db { "foo": ... } }
>>> |
>>> | node b { app { "x": after => A>>Mysql_db["foo"]; } }
>>>
>>>
>>>
>> David, are you suggesting this, or are you saying that this works?
>>
>
> I'm suggesting something like this. Internally it'd probably work like
> external_resource (see -dev list for details) but would query the
> storeconfig DB instead of calling a script on the client.
>
>
Yeah I had the same idea.
I think Luke was suggesting a higher level of modelling, but that can
work now, provided there's a way to easily create the content for the
remote resource check.
The Func-like approach would be for each node to surface what it's
"gates" were, and then set up authorization to allow other nodes to
ask them if a particular task was performed.
I'm not /quite/ sure that is sufficient though, or otherwise the gates
would have to be pretty smart -- for controlling an app upgrade you
would need more of a check than just seeing if the DB was present or a
version number was in a table.
--Michael
David Schmitt
> I don't immediately see it in my dev list, but I'll take your word for
> it. I am with you on the inter-node dependencies.
Should have better checked. It was here:
http://groups.google.com/group/puppet-users/msg/1d54d9db85dae153
Luke Kanies
I would essentially agree with that, with one caveat - that's
something Puppet's not very good at today, but I'm willing to make
some bets (not promises, quite yet) about it being good at it within a
year or so.
--
Fallacies do not cease to be fallacies because they become fashions.
--G. K. Chesterton
---------------------------------------------------------------------
Luke Kanies -|- http://reductivelabs.com -|- +1(615)594-8199
Luke Kanies
I essentially agree on both sides: Puppet isn't yet that great at
solving this problem, but using two tools sucks.
We're basically still in the formative stage of what we're going to do
here, but we've got some pretty good ideas. Mostly, it builds around
a much more powerful data model in the catalog, combined with a
centralized system that tracks inter-node dependencies.
Keeping in mind that this is all still a bit vague:
We'll be extending the data model to support service state
dependencies and simple state machines, along with (probably) some
kind of fill-in-the-value (which Markus insists on calling 'futures')
catalog. This, I think, gets us most of the way to what we need on a
single machine:
If you have two services, where one service specifies that it being in
a 'running' state requires that the other service must also be in a
'running' state, when you got to restart the required service then the
requiring service can automatically be brought down first.
If you have a single composite service (e.g., a "web app" composed of
a db and apache), then you could have a specification that defines how
it should behave depending on whether it should be running or not, and
then external data that defines which value you want in a given case.
Once we start tracking resource collection as a dependency (because
one host's configuration is now dependent on another's), plus a
centralized database containing all compiled catalogs, we've basically
got builtin node-order dependencies, all emergent rather than
declared. You could get dependencies through declaration, I think,
but it doesn't scale as well.
The 'external_resource' module I sent in last week[1] is a good
example of what the first bit will look like, but it has a lot of
"here be dragons" associated with it. To make it supportable in the
long term, you need to add builtin interfaces for checking service
state (probably with central correlation rather than direct
connectivity) plus the inter-node dependency system needs to recognize
this as a dependency.
Obviously, this is 80% theory, 15% plan, and 5% bs, but we do have
ideas for how it'll work, and we're making baby steps toward getting
some of it actually done.
In the meantime, I'd appreciate any results from people testing this
external resource stuff.
1 - http://github.com/reductivelabs/puppet-external-resource
--
The Internet, of course, is more than just a place to find pictures of
people having sex with dogs. -- Time Magazine, 3 July 1995
Luke Kanies
> [crossposting to puppet-dev, please trim follow-ups appropriately]
>
The external resource stuff does just about exactly this, except that
it's not a special syntax. It's not as manageable as I'd like --
there's no way, for instance, to really understand the dependencies
between hosts without reading all of the code -- but it's a good start
and should help us set the model.
--
Seize opportunity by the beard, for it is bald behind.
-- Bulgarian Proverb
David Schmitt
> If you have two services, where one service specifies that it being in a
> 'running' state requires that the other service must also be in a
> 'running' state, when you got to restart the required service then the
> requiring service can automatically be brought down first.
+1
> Once we start tracking resource collection as a dependency (because one
> host's configuration is now dependent on another's), plus a centralized
> database containing all compiled catalogs, we've basically got builtin
> node-order dependencies, all emergent rather than declared.
+1
> Obviously, this is 80% theory, 15% plan, and 5% bs, but we do have ideas
> for how it'll work, and we're making baby steps toward getting some of
> it actually done.
+1
Marcin Owsiany
> On 3/17/2010 8:41 PM, Luke Kanies wrote:
>> If you have two services, where one service specifies that it being in a
>> 'running' state requires that the other service must also be in a
>> 'running' state, when you got to restart the required service then the
>> requiring service can automatically be brought down first.
>
> +1
There might be exceptions, though. Consider an application that requires
a database to be up for startup, but once it's running, is able to
survive a database restart, degrading gracefully (serving from cache or
whatever). You would not want to degrade ungracefully by shutting it
down with the DB.
--
Marcin Owsiany <mar...@owsiany.pl> http://marcin.owsiany.pl/
GnuPG: 1024D/60F41216 FE67 DA2D 0ACA FC5E 3F75 D6F6 3A0D 8AA0 60F4 1216
"Every program in development at MIT expands until it can read mail."
-- Unknown
David Schmitt
> On Thu, Mar 18, 2010 at 10:58:52AM +0100, David Schmitt wrote:
>> On 3/17/2010 8:41 PM, Luke Kanies wrote:
>>> If you have two services, where one service specifies that it being in a
>>> 'running' state requires that the other service must also be in a
>>> 'running' state, when you got to restart the required service then the
>>> requiring service can automatically be brought down first.
>>
>> +1
>
> There might be exceptions, though. Consider an application that requires
> a database to be up for startup, but once it's running, is able to
> survive a database restart, degrading gracefully (serving from cache or
> whatever). You would not want to degrade ungracefully by shutting it
> down with the DB.
Indeed. But this is a question of modelling, not of the required
mechnics. Such an application would need to have running, degraded and
stopped states defined. This could even be used to start a lighttpd
serving a static "we're offline" site instead of a normally running
application container.