Puppet and God
Stephen
I want to use god to monitor apache and restart it automatically if it
is down. However, if god sees apache down, it will immediately try to
restart it which could lead to a race condition if puppet was
restarting httpd due to a config change.
Is this a problem? Could there be a situation in which the puppet and
god restarts conflict?
God's config file is just ruby, and has the concept of behaviours - so
will restart only under certain conditions. If there were a
guaranteed way to know that puppet was restarting httpd, I could tell
god not to get involved. How could I establish if this was the case?
S.
Luke Kanies
There's no ability to do so within Puppet at this point. I expect
apache2ctl or whatever is smart enough not to try to restart twice at
the same time, and actually, the normal apache restart process never
kills the process AFAIK, but for less sophisticated apps, there's at
least a possibility of a race condition.
I just spent a week with a client who really focused on lifecycle
issues. They're using ControlTier for their application lifecycle,
but it got me thinking about whether Puppet should adopt some
lifecycle abilities, and this could maybe fit in there. E.g., you
could create a lock file during restarting, and then remove it when
done.
--
When a man sits with a pretty girl for an hour, it seems like a
minute. But let him sit on a hot stove for a minute, and it's longer
than any hour. That's relativity. --Albert Einstein
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
Thomas, Dave
[mailto:puppet...@googlegroups.com]
> On Behalf Of Luke Kanies
> I just spent a week with a client who really focused on lifecycle
> issues. They're using ControlTier for their application lifecycle,
> but it got me thinking about whether Puppet should adopt some
> lifecycle abilities, and this could maybe fit in there. E.g., you
> could create a lock file during restarting, and then remove it when
> done.
I'm gonna highjack this thread to put in my two cents.
Lifecycle abilities ++
There is definitely some low-hanging fruit here IMO, but also a holy
grail.
In the simple case, we already do some process orchestration via
notify => [Exec["prepare"], File["fileserve-sync"], Exec["complete"]]
For the last day I've been wondering what there is in the sysadmin world
to do this:
WebTier => stop jboss
DbTier => stop oracle
DbTier => start oracle
WebTier => start jboss
I gather you can do some of this with Capistrano but I'm starting to
prefer the declarative language of puppet.
Had an interesting exchange on with #puppet weekenders Volcane &
holoway. What's missing in the sysadmin world is "cross-system
orchestration" -- or is it choreography? My first instinct is that this
would have been well-addressed in the HA community. It's not. I'm
definitely looking at ControlTier! I liked holoway's idea of some form
of message queue. What about workflow?
To begin with, I want to automate the deployment and test of my
continuous build. Puppet gets me 90% of the way there out of the box!
Ultimately I need to automate migrations too.
Being a puppet newb and having already pointed out how we need more
walkthroughs & blogs, I'm working on a blog describing how to use puppet
with Hudson continuous build:
http://www.peoplemerge.com/drupal/hudson_and_puppet
It's _almost_working_ - just need orchestration. This example uses
Solaris ZFS + zones but it's probably just as easy to use XEN + Linux
LVM.
Here's a summary:
* At midnight the build server will kick off a build. When finished,
it writes the file /var/lib/build/latest-1_1 with the current build
* The puppet client on the global zone is watching and waiting for
that file to change. When it does, it jumps into action.
* First, it blows away the install from the previous day and
restarts the zone with a clean Solaris installation.
* Next, it copies the build over using the puppet fileserver. Either
the global zone or the test server can do this.
* Next, the test server runs the installer shell script.
* When the install is done, it makes another snapshot before any
automated tests begin
**********************************************************************
This communication is confidential and intended solely for the
addressee(s). Any unauthorized review, use, disclosure or distribution
is prohibited. If you believe this message has been sent to you in
error, please notify the sender by replying to this transmission and
delete the message without disclosing it. Thank you.
E-mail including attachments is susceptible to data corruption,
interception, unauthorized amendment, tampering and viruses, and we
only send and receive e-mails on the basis that we are not liable for
any such corruption, interception, amendment, tampering or viruses or
any consequences thereof.
This email, its content and any attachments is PRIVATE AND
CONFIDENTIAL to TANDBERG Television, Part of the Ericsson Group.
www.tandbergtv.com
**********************************************************************
Adam Jacob
> There is definitely some low-hanging fruit here IMO, but also a holy
> grail.
>
> In the simple case, we already do some process orchestration via
> notify => [Exec["prepare"], File["fileserve-sync"], Exec["complete"]]
>
> For the last day I've been wondering what there is in the sysadmin world
> to do this:
>
> WebTier => stop jboss
> DbTier => stop oracle
> DbTier => start oracle
> WebTier => start jboss
>
> I gather you can do some of this with Capistrano but I'm starting to
> prefer the declarative language of puppet.
You've basically got, to my knowledge:
Capistrano
Control Tier
Vertebra (coming soon)
The Control Tier guys are awesome - very friendly, they have a cool
product that's been battle scarred in real production environments.
It's been open source for a long time, but hasn't gained much
traction, mainly due to the fact that the authors were our Making
Money(tm) on large Control Tier deployments.
I use the peanut butter and jelly analogy a lot when talking about
tools like this and puppet. I like peanut butter on toast, and I like
jelly on toast, but when you put them together, it's magically
delicious.
Adam
--
HJK Solutions - We Launch Startups - http://www.hjksolutions.com
Adam Jacob, Senior Partner
T: (206) 508-4759 E: ad...@hjksolutions.com
Lee Thompson
On Aug 17, 1:39 pm, "Adam Jacob" <a...@hjksolutions.com> wrote:
> The Control Tier guys are awesome - very friendly, they have a cool
> product that's been battle scarred in real production environments.
>
>
> --
Worth a few minutes to go through the ControlTier CTL slideware for an
intro...
http://ctl.controltier.org/presentations/index.html
This one for instance
http://ctl.controltier.org/presentations/svlug-intro.pdf
Lee
Luke Kanies
>
>> From: puppet...@googlegroups.com
> [mailto:puppet...@googlegroups.com]
>> On Behalf Of Luke Kanies
>> I just spent a week with a client who really focused on lifecycle
>> issues. They're using ControlTier for their application lifecycle,
>> but it got me thinking about whether Puppet should adopt some
>> lifecycle abilities, and this could maybe fit in there. E.g., you
>> could create a lock file during restarting, and then remove it when
>> done.
>
> I'm gonna highjack this thread to put in my two cents.
>
> Lifecycle abilities ++
>
> There is definitely some low-hanging fruit here IMO, but also a holy
> grail.
>
> In the simple case, we already do some process orchestration via
> notify => [Exec["prepare"], File["fileserve-sync"], Exec["complete"]]
This simple kind of lifecycle could be easily handled with
metaparameters -- just add before/after hooks. I'd prefer to be a
touch smarter about it, by doing something like defining possible
states a resource could be in, and then do the work based on states --
e.g., service X must be in state 'stopped' before you upgrade package Y.
The metaparameters would be a simple start, and might make the most
sense out of the gate, though.
>
> For the last day I've been wondering what there is in the sysadmin
> world
> to do this:
>
> WebTier => stop jboss
> DbTier => stop oracle
> DbTier => start oracle
> WebTier => start jboss
>
> I gather you can do some of this with Capistrano but I'm starting to
> prefer the declarative language of puppet.
>
> Had an interesting exchange on with #puppet weekenders Volcane &
> holoway. What's missing in the sysadmin world is "cross-system
> orchestration" -- or is it choreography? My first instinct is that
> this
> would have been well-addressed in the HA community. It's not. I'm
> definitely looking at ControlTier! I liked holoway's idea of some
> form
> of message queue. What about workflow?
Multi-host workflow is definitely a whole different ball-game. To use
a declarative system like Puppet, you'd clearly need cross-host
messaging, so hostA could ask hostB to move serviceB to stateX, do its
work, and then move the state back.
Your first question becomes whether you want this work managed
centrally, like ControlTier (and nearly everyone else) does, or if
you'd like to make it possible for hosts to mediate the work themselves.
I'd love to add some of this stuff to Puppet, but I think it's a good
ways from there, and nearly all of that work will be another
application sitting on top of Puppet. If we can do a good job of
modeling more of Puppet as states, transitions, and events, then it
should be pretty damn easy to sit all kinds of crazy workflow apps on
top of it.
We're not all that far off for that, but it's still a bit.
Yeah, this kind of work shouldn't be all that hard to add to Puppet.
You could certainly fake it by just adding hooks to do scripts before
or after events.
--
Seize opportunity by the beard, for it is bald behind.
-- Bulgarian Proverb
Damon Edwards
we go. ControlTier and Puppet are quite complimentary and both
management styles are required in a real life web operation. Different
tools for different jobs.
If you are familiar with the cross system requirements outlined in
this thread (we call it "command dispatching") or the basics of how a
tool like capistrano works, then these basic tutorials should make a
lot of sense:
http://open.controltier.org/Tutorials/index.html
We're going to document our experiences using ControlTier and Puppet
soon. I'll post them here when we do.
-Damon
controltier.com
dev2ops.com
Thomas, Dave
> -----Original Message-----
> From: puppet...@googlegroups.com
[mailto:puppet...@googlegroups.com] On Behalf Of Damon Edwards
> We're going to document our experiences using ControlTier and Puppet
> soon. I'll post them here when we do.
Likewise. When I've learned enough ControlTier get my continuous build
to work, I'll post back up here.
Dave