I want to use god to monitor apache and restart it automatically if it
is down. However, if god sees apache down, it will immediately try to
restart it which could lead to a race condition if puppet was
restarting httpd due to a config change.
Is this a problem? Could there be a situation in which the puppet and
god restarts conflict?
God's config file is just ruby, and has the concept of behaviours - so
will restart only under certain conditions. If there were a
guaranteed way to know that puppet was restarting httpd, I could tell
god not to get involved. How could I establish if this was the case?
> I want to use god to monitor apache and restart it automatically if it > is down. However, if god sees apache down, it will immediately try to > restart it which could lead to a race condition if puppet was > restarting httpd due to a config change.
> Is this a problem? Could there be a situation in which the puppet and > god restarts conflict?
> God's config file is just ruby, and has the concept of behaviours - so > will restart only under certain conditions. If there were a > guaranteed way to know that puppet was restarting httpd, I could tell > god not to get involved. How could I establish if this was the case?
There's no ability to do so within Puppet at this point. I expect apache2ctl or whatever is smart enough not to try to restart twice at the same time, and actually, the normal apache restart process never kills the process AFAIK, but for less sophisticated apps, there's at least a possibility of a race condition.
I just spent a week with a client who really focused on lifecycle issues. They're using ControlTier for their application lifecycle, but it got me thinking about whether Puppet should adopt some lifecycle abilities, and this could maybe fit in there. E.g., you could create a lock file during restarting, and then remove it when done.
-- When a man sits with a pretty girl for an hour, it seems like a minute. But let him sit on a hot stove for a minute, and it's longer than any hour. That's relativity. --Albert Einstein --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
> On Behalf Of Luke Kanies > I just spent a week with a client who really focused on lifecycle > issues. They're using ControlTier for their application lifecycle, > but it got me thinking about whether Puppet should adopt some > lifecycle abilities, and this could maybe fit in there. E.g., you > could create a lock file during restarting, and then remove it when > done.
I'm gonna highjack this thread to put in my two cents.
Lifecycle abilities ++
There is definitely some low-hanging fruit here IMO, but also a holy grail.
In the simple case, we already do some process orchestration via notify => [Exec["prepare"], File["fileserve-sync"], Exec["complete"]]
For the last day I've been wondering what there is in the sysadmin world to do this:
I gather you can do some of this with Capistrano but I'm starting to prefer the declarative language of puppet.
Had an interesting exchange on with #puppet weekenders Volcane & holoway. What's missing in the sysadmin world is "cross-system orchestration" -- or is it choreography? My first instinct is that this would have been well-addressed in the HA community. It's not. I'm definitely looking at ControlTier! I liked holoway's idea of some form of message queue. What about workflow?
To begin with, I want to automate the deployment and test of my continuous build. Puppet gets me 90% of the way there out of the box! Ultimately I need to automate migrations too.
Being a puppet newb and having already pointed out how we need more walkthroughs & blogs, I'm working on a blog describing how to use puppet with Hudson continuous build:
It's _almost_working_ - just need orchestration. This example uses Solaris ZFS + zones but it's probably just as easy to use XEN + Linux LVM.
Here's a summary: * At midnight the build server will kick off a build. When finished, it writes the file /var/lib/build/latest-1_1 with the current build * The puppet client on the global zone is watching and waiting for that file to change. When it does, it jumps into action. * First, it blows away the install from the previous day and restarts the zone with a clean Solaris installation. * Next, it copies the build over using the puppet fileserver. Either the global zone or the test server can do this. * Next, the test server runs the installer shell script. * When the install is done, it makes another snapshot before any automated tests begin
This communication is confidential and intended solely for the addressee(s). Any unauthorized review, use, disclosure or distribution is prohibited. If you believe this message has been sent to you in error, please notify the sender by replying to this transmission and delete the message without disclosing it. Thank you.
E-mail including attachments is susceptible to data corruption, interception, unauthorized amendment, tampering and viruses, and we only send and receive e-mails on the basis that we are not liable for any such corruption, interception, amendment, tampering or viruses or any consequences thereof.
This email, its content and any attachments is PRIVATE AND CONFIDENTIAL to TANDBERG Television, Part of the Ericsson Group.
On Sun, Aug 17, 2008 at 3:25 AM, Thomas, Dave <dtho...@tandbergtv.com> wrote: > There is definitely some low-hanging fruit here IMO, but also a holy > grail.
> In the simple case, we already do some process orchestration via > notify => [Exec["prepare"], File["fileserve-sync"], Exec["complete"]]
> For the last day I've been wondering what there is in the sysadmin world > to do this:
> I gather you can do some of this with Capistrano but I'm starting to > prefer the declarative language of puppet.
You've basically got, to my knowledge:
Capistrano Control Tier Vertebra (coming soon)
The Control Tier guys are awesome - very friendly, they have a cool product that's been battle scarred in real production environments. It's been open source for a long time, but hasn't gained much traction, mainly due to the fact that the authors were our Making Money(tm) on large Control Tier deployments.
I use the peanut butter and jelly analogy a lot when talking about tools like this and puppet. I like peanut butter on toast, and I like jelly on toast, but when you put them together, it's magically delicious.
Adam
-- HJK Solutions - We Launch Startups - http://www.hjksolutions.com Adam Jacob, Senior Partner T: (206) 508-4759 E: a...@hjksolutions.com
>> From: puppet-users@googlegroups.com > [mailto:puppet-users@googlegroups.com] >> On Behalf Of Luke Kanies >> I just spent a week with a client who really focused on lifecycle >> issues. They're using ControlTier for their application lifecycle, >> but it got me thinking about whether Puppet should adopt some >> lifecycle abilities, and this could maybe fit in there. E.g., you >> could create a lock file during restarting, and then remove it when >> done.
> I'm gonna highjack this thread to put in my two cents.
> Lifecycle abilities ++
> There is definitely some low-hanging fruit here IMO, but also a holy > grail.
> In the simple case, we already do some process orchestration via > notify => [Exec["prepare"], File["fileserve-sync"], Exec["complete"]]
This simple kind of lifecycle could be easily handled with metaparameters -- just add before/after hooks. I'd prefer to be a touch smarter about it, by doing something like defining possible states a resource could be in, and then do the work based on states -- e.g., service X must be in state 'stopped' before you upgrade package Y.
The metaparameters would be a simple start, and might make the most sense out of the gate, though.
> I gather you can do some of this with Capistrano but I'm starting to > prefer the declarative language of puppet.
> Had an interesting exchange on with #puppet weekenders Volcane & > holoway. What's missing in the sysadmin world is "cross-system > orchestration" -- or is it choreography? My first instinct is that > this > would have been well-addressed in the HA community. It's not. I'm > definitely looking at ControlTier! I liked holoway's idea of some > form > of message queue. What about workflow?
Multi-host workflow is definitely a whole different ball-game. To use a declarative system like Puppet, you'd clearly need cross-host messaging, so hostA could ask hostB to move serviceB to stateX, do its work, and then move the state back.
Your first question becomes whether you want this work managed centrally, like ControlTier (and nearly everyone else) does, or if you'd like to make it possible for hosts to mediate the work themselves.
I'd love to add some of this stuff to Puppet, but I think it's a good ways from there, and nearly all of that work will be another application sitting on top of Puppet. If we can do a good job of modeling more of Puppet as states, transitions, and events, then it should be pretty damn easy to sit all kinds of crazy workflow apps on top of it.
We're not all that far off for that, but it's still a bit.
> To begin with, I want to automate the deployment and test of my > continuous build. Puppet gets me 90% of the way there out of the box! > Ultimately I need to automate migrations too.
> Being a puppet newb and having already pointed out how we need more > walkthroughs & blogs, I'm working on a blog describing how to use > puppet > with Hudson continuous build:
> It's _almost_working_ - just need orchestration. This example uses > Solaris ZFS + zones but it's probably just as easy to use XEN + Linux > LVM.
> Here's a summary: > * At midnight the build server will kick off a build. When > finished, > it writes the file /var/lib/build/latest-1_1 with the current build > * The puppet client on the global zone is watching and waiting for > that file to change. When it does, it jumps into action. > * First, it blows away the install from the previous day and > restarts the zone with a clean Solaris installation. > * Next, it copies the build over using the puppet fileserver. > Either > the global zone or the test server can do this. > * Next, the test server runs the installer shell script. > * When the install is done, it makes another snapshot before any > automated tests begin
Yeah, this kind of work shouldn't be all that hard to add to Puppet. You could certainly fake it by just adding hooks to do scripts before or after events.
-- Seize opportunity by the beard, for it is bald behind. -- Bulgarian Proverb --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
We recommend Puppet almost everywhere (except for our windows users)
we go. ControlTier and Puppet are quite complimentary and both
management styles are required in a real life web operation. Different
tools for different jobs.
If you are familiar with the cross system requirements outlined in
this thread (we call it "command dispatching") or the basics of how a
tool like capistrano works, then these basic tutorials should make a
lot of sense:
http://open.controltier.org/Tutorials/index.html
We're going to document our experiences using ControlTier and Puppet
soon. I'll post them here when we do.
-Damon
controltier.com
dev2ops.com
On Aug 18, 9:38 am, Luke Kanies <l...@madstop.com> wrote:
> On Aug 17, 2008, at 5:25 AM, Thomas, Dave wrote:
> >> From: puppet-users@googlegroups.com
> > [mailto:puppet-users@googlegroups.com]
> >> On Behalf Of Luke Kanies
> >> I just spent a week with a client who really focused on lifecycle
> >> issues. They're using ControlTier for their application lifecycle,
> >> but it got me thinking about whether Puppet should adopt some
> >> lifecycle abilities, and this could maybe fit in there. E.g., you
> >> could create a lock file during restarting, and then remove it when
> >> done.
> > I'm gonna highjack this thread to put in my two cents.
> > Lifecycle abilities ++
> > There is definitely some low-hanging fruit here IMO, but also a holy
> > grail.
> > In the simple case, we already do some process orchestration via
> > notify => [Exec["prepare"], File["fileserve-sync"], Exec["complete"]]
> This simple kind of lifecycle could be easily handled with
> metaparameters -- just add before/after hooks. I'd prefer to be a
> touch smarter about it, by doing something like defining possible
> states a resource could be in, and then do the work based on states --
> e.g., service X must be in state 'stopped' before you upgrade package Y.
> The metaparameters would be a simple start, and might make the most
> sense out of the gate, though.
> > For the last day I've been wondering what there is in the sysadmin
> > world
> > to do this:
> > I gather you can do some of this with Capistrano but I'm starting to
> > prefer the declarative language of puppet.
> > Had an interesting exchange on with #puppet weekenders Volcane &
> > holoway. What's missing in the sysadmin world is "cross-system
> > orchestration" -- or is it choreography? My first instinct is that
> > this
> > would have been well-addressed in the HA community. It's not. I'm
> > definitely looking at ControlTier! I liked holoway's idea of some
> > form
> > of message queue. What about workflow?
> Multi-host workflow is definitely a whole different ball-game. To use
> a declarative system like Puppet, you'd clearly need cross-host
> messaging, so hostA could ask hostB to move serviceB to stateX, do its
> work, and then move the state back.
> Your first question becomes whether you want this work managed
> centrally, like ControlTier (and nearly everyone else) does, or if
> you'd like to make it possible for hosts to mediate the work themselves.
> I'd love to add some of this stuff to Puppet, but I think it's a good
> ways from there, and nearly all of that work will be another
> application sitting on top of Puppet. If we can do a good job of
> modeling more of Puppet as states, transitions, and events, then it
> should be pretty damn easy to sit all kinds of crazy workflow apps on
> top of it.
> We're not all that far off for that, but it's still a bit.
> > To begin with, I want to automate the deployment and test of my
> > continuous build. Puppet gets me 90% of the way there out of the box!
> > Ultimately I need to automate migrations too.
> > Being a puppet newb and having already pointed out how we need more
> > walkthroughs & blogs, I'm working on a blog describing how to use
> > puppet
> > with Hudson continuous build:
> > It's _almost_working_ - just need orchestration. This example uses
> > Solaris ZFS + zones but it's probably just as easy to use XEN + Linux
> > LVM.
> > Here's a summary:
> > * At midnight the build server will kick off a build. When
> > finished,
> > it writes the file /var/lib/build/latest-1_1 with the current build
> > * The puppet client on the global zone is watching and waiting for
> > that file to change. When it does, it jumps into action.
> > * First, it blows away the install from the previous day and
> > restarts the zone with a clean Solaris installation.
> > * Next, it copies the build over using the puppet fileserver.
> > Either
> > the global zone or the test server can do this.
> > * Next, the test server runs the installer shell script.
> > * When the install is done, it makes another snapshot before any
> > automated tests begin
> Yeah, this kind of work shouldn't be all that hard to add to Puppet.
> You could certainly fake it by just adding hooks to do scripts before
> or after events.
> --
> Seize opportunity by the beard, for it is bald behind.
> -- Bulgarian Proverb
> ---------------------------------------------------------------------
> Luke Kanies |http://reductivelabs.com|http://madstop.com
[mailto:puppet-users@googlegroups.com] On Behalf Of Damon Edwards
> We're going to document our experiences using ControlTier and Puppet > soon. I'll post them here when we do.
Likewise. When I've learned enough ControlTier get my continuous build to work, I'll post back up here.
Dave **********************************************************************
This communication is confidential and intended solely for the addressee(s). Any unauthorized review, use, disclosure or distribution is prohibited. If you believe this message has been sent to you in error, please notify the sender by replying to this transmission and delete the message without disclosing it. Thank you.
E-mail including attachments is susceptible to data corruption, interception, unauthorized amendment, tampering and viruses, and we only send and receive e-mails on the basis that we are not liable for any such corruption, interception, amendment, tampering or viruses or any consequences thereof.
This email, its content and any attachments is PRIVATE AND CONFIDENTIAL to TANDBERG Television, Part of the Ericsson Group.
|
