tools for managing and composing shell scripts?

391 views
Skip to first unread message

Miles Fidelman

unread,
Mar 18, 2012, 10:03:52 AM3/18/12
to dev...@googlegroups.com
Over the years, I've built up a lot of shell scripts for provisioning
and configuration - which makes converting to something like chef or
puppet a somewhat daunting task (or even something like FAI).

I'm also a big fan of composing tool chains from piece parts - scripts
linked by pipes, mashups of functions accessed via RESTful APIs, etc.

Rundeck looks pretty interesting - given that it can manage shell
scripts and their execution via ssh. I'm sort of wondering what other
kinds of tools people are using for adding a job and configuration
control overlay to shell scripts. And what about orchestration tools
that that take a "mashup maker" approach? Pointers, thoughts, ....

Thanks,

Miles Fidelman

--
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra


Bryan Berry

unread,
Mar 18, 2012, 11:01:34 AM3/18/12
to dev...@googlegroups.com
miles,

we've all been there and we have all tried to shoehorn our shell scripts into a modular system. Spare yourself the pain, it's a losing game. You have to bite the bullet and learn a configuration management system, be it cfengine, puppet, chef, or foobarbaz

It may seem daunting but let me tell you it is absolutely worth the effort

I have actually recorded two podcasts on related topics and it may be worth your time to listen:
http://www.foodfightshow.org/2012/03/episode-5-getting-started-with-chef.html   about getting started w/ chef but most of the concepts also apply to puppet

Zack Williams

unread,
Mar 18, 2012, 11:03:34 AM3/18/12
to dev...@googlegroups.com
On Sun, Mar 18, 2012 at 7:03 AM, Miles Fidelman
<mfid...@meetinghouse.net> wrote:
> Over the years, I've built up a lot of shell scripts for provisioning and
> configuration - which makes converting to something like chef or puppet a
> somewhat daunting task (or even something like FAI).
>
> I'm also a big fan of composing tool chains from piece parts - scripts
> linked by pipes, mashups of functions accessed via RESTful APIs, etc.
>
> Rundeck looks pretty interesting - given that it can manage shell scripts
> and their execution via ssh.  I'm sort of wondering what other kinds of
> tools people are using for adding a job and configuration control overlay to
> shell scripts.  And what about orchestration tools that that take a "mashup
> maker" approach?  Pointers, thoughts, ....

Are you looking for a script organization tool, or a one-to-many
execution tool?

I'm a fan of using Rakefiles to organize a bunch of one- or few- liner
scripts into a namespaced whole. I generally wrap these with small
boilerplate shell script that makes them filesystem location
independent and allows me to pass variables in as required. I keep
these in git, for replication/backup.

You also get the full power of ruby so it's a good gateway drug to
other systems...

- Zack

gareth rushgrove

unread,
Mar 18, 2012, 12:10:26 PM3/18/12
to dev...@googlegroups.com
> other systems…
>

I'd echo Bryan with regards the worth of looking at the (sometimes
large and potentially intimidating) configuration management tools.

On the other hand I do think there is a place for more adhoc script
execution things, especially in smaller but still dynamic environments
and mainly for asking questions of the systems. I've used Fabric
(http://fabfile.org/) successfully here, partly because it provides
the framework for managing hosts and roles and doing parallel
execution, and the arguments can just be shell scripts. Once it's
setup anyone who's happy with bash can easily jump in, and it's a good
way of collecting useful one-liners. Past a certain number of hosts
something like mcollective or salt are probably more useful, they use
(different) message brokers to avoid problems with lots of ssh
sessions.

G

> - Zack

--
Gareth Rushgrove
Web Geek

morethanseven.net
garethrushgrove.com

Bryan Berry

unread,
Mar 18, 2012, 12:22:31 PM3/18/12
to dev...@googlegroups.com
+1 for fabric, its an awesome adhoc scripting tool

that said, I rarely reach for it as knife scripts are even easier to write w/ the knife exec sub-command. 

Does puppet has something like knife exec?

My $0.02, it is worth diving into puppet/chef/foobarbaz and later filling in the gaps w/ fabric/capistrano/mcollective. However, fabric and capistrano have much lower learning curves. I just found that I had written fabfiles that would have been easier to implement (and idempotent) w/ chef recipes.

Miles Fidelman

unread,
Mar 18, 2012, 1:52:33 PM3/18/12
to dev...@googlegroups.com
Bryan Berry wrote:
> miles,
>
> we've all been there and we have all tried to shoehorn our shell
> scripts into a modular system. Spare yourself the pain, it's a losing
> game. You have to bite the bullet and learn a configuration management
> system, be it cfengine, puppet, chef, or foobarbaz
>
> It may seem daunting but let me tell you it is absolutely worth the effort
>

Well you've obviously drunk the koolaid :-)

But that's not the problem I'm trying to address. As far as I can tell,
pretty much any configuration management system is only one part of a
toolchain - and right now, I don't see any complete toolchains, many of
the tools seem don't seem to compose very well (e.g., point me at
something that ties together provisioning with IP address management
with DNS record management), and a lot of pieces seem to be distribution
specific (e.g., FIA for debian, kickstarter for redhat-like distros).

Re. configuration tools specifically: I don't find chef, or puppet, or
bcg2, or cfengine recipes/rules all that clearer or more powerful than a
shell script, or perhaps a tcl script with expect statements - while I
find the restrictions associated with picking one of these something I'm
not ready to do.

Hence my interest in something like rundeck, that lets you manage and
compose scripts, AND configurations stored in other systems.

Miles Fidelman

unread,
Mar 18, 2012, 2:02:23 PM3/18/12
to dev...@googlegroups.com
Zack,

Rake looks pretty interesting. Not sure I want to bit off ruby (doing
most of our development in Erlang and JavaScript at the moment), but it
does seem to do some useful things.

Any thoughts on Rake vs. tcl/tk?

Thanks!

Miles

ranjib dey

unread,
Mar 18, 2012, 4:14:11 PM3/18/12
to dev...@googlegroups.com
tcl is a full blown scripting language (with tk being the ui part), you can compare it against ruby. While rake is a ruby library that gives nice wrapper around shell , introduces a notion of task and let you establish dependencies among tasks. Its like a build tool. consider it analogous of make or ant

AJ Christensen

unread,
Mar 18, 2012, 4:17:05 PM3/18/12
to dev...@googlegroups.com

Is this a joke thread? Still not quite gotten the hang of American sarcasm.

--AJ

Miles Fidelman

unread,
Mar 18, 2012, 4:25:13 PM3/18/12
to dev...@googlegroups.com
Zack Williams wrote:
> On Sun, Mar 18, 2012 at 7:03 AM, Miles Fidelman
> <mfid...@meetinghouse.net> wrote:
>> Over the years, I've built up a lot of shell scripts for provisioning and
>> configuration - which makes converting to something like chef or puppet a
>> somewhat daunting task (or even something like FAI).
>>
>> I'm also a big fan of composing tool chains from piece parts - scripts
>> linked by pipes, mashups of functions accessed via RESTful APIs, etc.
>>
>> Rundeck looks pretty interesting - given that it can manage shell scripts
>> and their execution via ssh. I'm sort of wondering what other kinds of
>> tools people are using for adding a job and configuration control overlay to
>> shell scripts. And what about orchestration tools that that take a "mashup
>> maker" approach? Pointers, thoughts, ....
> Are you looking for a script organization tool, or a one-to-many
> execution tool?
(realized I just jumped past this, right to your suggestion that I look
at Rakefiles)

Both, but more focused on organization. An ideal tool would:
- provide for editing/managing/versioning scripts (script = anything
that can be invoked at the command line)
- a library of control functions for use within scripts
- invoking scripts, combinations of scripts, pipelines of scripts (in
the Unix sense of pipes) - locally, remotely, across multiple machines
- remote script execution via ssh, rather than some kind of agent
- providing a simple database for keeping track of variables used by
scripts (e.g., IP addresses, DNS records for use by a provisioning
script) - that can be accessed from scripts
- accessing the above via cli, RESTful API, GUI
- cross-platform
- (nice-to-have) minimal environmental requirements (i.e., a step above
the unix shell, say the gnu buildtool suite rather than the JVM + a mass
of libraries, or the ruby ecosystem -- a self-configuring basic
environment would be nice, like perl+cpan)

I'm sure we've each cobbled something together along these lines - be it
through manual processes, standard installs, or something fancier. I'm
kind of wondering what different people have done, what pieces fit
together better than others, what's available that's more integrated but
that doesn't create a monolithic environment.

Rundeck seems to do most (all?) of this, running on Java. It looks
nice, but it's also really new.

tcl/tk provides a lot of this - but without the organizing functions
(tcl/tk + git, or maybe a tcl/tk IDE perhaps?)

I'm basically wondering what else might be out there - perhaps less
visible - newer than tcl/tk, more mature than Rundeck perhaps.
Essentially doing some research before provisioning several new machines
and a bunch of VMs (easy enough to do from manual checklists, but I'm
setting the stage for some future scaling).

Miles

AJ Christensen

unread,
Mar 18, 2012, 4:29:39 PM3/18/12
to dev...@googlegroups.com

The clojure 'pallet' tool (cm) has some of these principles.

I've seen sloppy admins compose and reuse Bash snippets with Chef all the time. Easy trap for beginners.

--AJ

Miles Fidelman

unread,
Mar 18, 2012, 4:44:57 PM3/18/12
to dev...@googlegroups.com
AJ Christensen wrote:
>
> Is this a joke thread? Still not quite gotten the hang of American
> sarcasm.
>

Why would you think that?

and later, AJ Christensen wrote:

> The clojure 'pallet' tool (cm) has some of these principles.
>

seems interesting, if a little java and cloud centric

> I've seen sloppy admins compose and reuse Bash snippets with Chef all
> the time. Easy trap for beginners.
>

Please elaborate --- are you saying that composing bash snippets is a
trap, or using Chef to do it is a trap? (If the former, who are you
calling a beginner? If the latter, I agree - not what Chef is built
for, and not the tool I'd use for managing shell scripts - hence my query.)

Scott Likens

unread,
Mar 18, 2012, 7:15:05 PM3/18/12
to dev...@googlegroups.com
On Sunday, March 18, 2012 7:03:52 AM UTC-7, Miles Fidelman wrote:
Over the years, I've built up a lot of shell scripts for provisioning
and configuration - which makes converting to something like chef or
puppet a somewhat daunting task (or even something like FAI).

I'm also a big fan of composing tool chains from piece parts - scripts
linked by pipes, mashups of functions accessed via RESTful APIs, etc.

Rundeck looks pretty interesting - given that it can manage shell
scripts and their execution via ssh.  I'm sort of wondering what other
kinds of tools people are using for adding a job and configuration
control overlay to shell scripts.  And what about orchestration tools
that that take a "mashup maker" approach?  Pointers, thoughts, ....

Thanks,


If your looking for a middle-ground that you can force as you will  you might find SM appealing https://bdsm.beginrescueend.com/

Miles Fidelman

unread,
Mar 18, 2012, 7:27:15 PM3/18/12
to dev...@googlegroups.com
Scott Likens wrote:
>
>
> If your looking for a middle-ground that you can force as you will
> you might find SM appealing https://bdsm.beginrescueend.com/

Two people have pointed at this, now, and the description sure sounds
interesting. Anybody know how active the project is - there doesn't
seem to be a mailing list or FAQ, and it seems to have one primary
developer.

Scott Likens

unread,
Mar 18, 2012, 8:22:25 PM3/18/12
to dev...@googlegroups.com
Miles,

It has 2 or 3 active developers to the best of my knowledge.  As far as the mailing list... See This response

I really can't speak for how active it is or anything less as I don't talk to Wayne.

Paul Graydon

unread,
Mar 18, 2012, 8:25:59 PM3/18/12
to dev...@googlegroups.com
On 3/18/2012 4:03 AM, Miles Fidelman wrote:
Over the years, I've built up a lot of shell scripts for provisioning and configuration - which makes converting to something like chef or puppet a somewhat daunting task (or even something like FAI).

Excuse me, I'm going to go for the utterly blunt response:

So what..?  I know as a sysadmin we try our best to be "appropriately lazy", but that means evaluating the benefits too compared to the work put in.  The positives of a config management system outweigh the negatives by a big enough margin to make any desire not to implement it border on pure laziness, not appropriate laziness.  Just being daunted by it is not a good enough excuse.

Yes, it's a pain in the ass trying to move from what is for you a smooth working set of scripts, but you gain a system that works easily, and that's going to be both understandable and enhanceable by anyone else that comes along after you (or should you be 'run over by a bus'.)*

No one is requiring you to convert the entire thing on one go.  I've a long way to go with the infrastructure I've inherited (an absolute mess of scripts and inconsistencies), but I've already got implemented a recipes in Chef that handle all the baseline common stuff, e.g. applications that should be on all (ntp etc), their configuration and so on, and am slowly adding more complex aspects of it.  Yes it's a pain in the arse, but it's important.  Just like all those niggling security requirements are a pain in the arse but you wouldn't be seen dead not implementing them.

Paul

* Every time I've gone into a sysadmin job, or picked up consulting work, I've found I've inherited a bunch of unusual bash and perl scripts, undocumented, inconsistent and bizarre in methodology (ever notice how no two sysadmins write perl / bash scripts alike?).  It means it takes a stupid amount of time to piece things together and understand quite how they interact.  Every time something breaks or goes wrong I'm left playing guessing games and tracking down whether it's a fringe case from some obscure aspect of a script or quite what (side note, it's amazing just how many problems I fix by enabling 'use strict; use warnings' and clearing up errors reported).

Miles Fidelman

unread,
Mar 18, 2012, 9:01:37 PM3/18/12
to dev...@googlegroups.com
Scott Likens wrote:
> Miles,
>
> It has 2 or 3 active developers to the best of my knowledge. As far
> as the mailing list... See This response
> <https://bdsm.beginrescueend.com/development/list>
>
>

You mean this one?

"None yet, stay tuned I might setup a librelist. "


That's what makes me wonder.

Miles

Chad Woolley

unread,
Mar 18, 2012, 9:10:22 PM3/18/12
to dev...@googlegroups.com
On Sun, Mar 18, 2012 at 6:01 PM, Miles Fidelman
<mfid...@meetinghouse.net> wrote:
> Scott Likens wrote:
>>
>> Miles,
>>
>> It has 2 or 3 active developers to the best of my knowledge.  As far as
>> the mailing list... See This response
>> <https://bdsm.beginrescueend.com/development/list>
>
> You mean this one?
>
> "None yet, stay tuned I might setup a librelist. "
>
>
> That's what makes me wonder.
>
> Miles

It's active on GitHub, and you can submit issues and pull requests
there (they accepted one 12 days ago):

https://bdsm.beginrescueend.com/development/github/

I do know Wayne, and he's usually very responsive to input, but I
think he prefers IRC, which is probably why he hasn't set up a mailing
list for this yet (they suck up valuable coding time, right?). Dunno
how far-baked BDSM is or if you should bet the farm on it, but Wayne
wrote RVM, which is a widely used project in the Ruby community
(written in shell script), and has actively maintained that for years
now.

If the lack of a mailing list really bothers you, open a github issue
asking for one ;)

-- Chad

Miles Fidelman

unread,
Mar 18, 2012, 9:11:19 PM3/18/12
to dev...@googlegroups.com
Paul Graydon wrote:
> On 3/18/2012 4:03 AM, Miles Fidelman wrote:
>> Over the years, I've built up a lot of shell scripts for provisioning
>> and configuration - which makes converting to something like chef or
>> puppet a somewhat daunting task (or even something like FAI).
>
> Excuse me, I'm going to go for the utterly blunt response:
>
> So what..? I know as a sysadmin we try our best to be "appropriately
> lazy", but that means evaluating the benefits too compared to the work
> put in. The positives of a config management system outweigh the
> negatives by a big enough margin to make any desire not to implement
> it border on pure laziness, not appropriate laziness. Just being
> daunted by it is not a good enough excuse.
>
> Yes, it's a pain in the ass trying to move from what is for you a
> smooth working set of scripts, but you gain a system that works
> easily, /and /that's going to be both understandable and enhanceable
> by anyone else that comes along after you (or should you be 'run over
> by a bus'.)*

Pardon me for being blunt as well, then. I don't see a particularly
high return on investment for investing time in converting to a tool (be
it puppet, chef, cfengine, or what have you) that addresses only a small
piece of an overall operations toolchain. I'd rather invest time and
effort in getting my existing toolchain more automated, documented, and
under configuration control.

>
> * Every time I've gone into a sysadmin job, or picked up consulting
> work, I've found I've inherited a bunch of unusual bash and perl
> scripts, undocumented, inconsistent and bizarre in methodology (ever
> notice how no two sysadmins write perl / bash scripts alike?). It
> means it takes a stupid amount of time to piece things together and
> understand quite how they interact. Every time something breaks or
> goes wrong I'm left playing guessing games and tracking down whether
> it's a fringe case from some obscure aspect of a script or quite what
> (side note, it's amazing just how many problems I fix by enabling 'use
> strict; use warnings' and clearing up errors reported).

You ever notice that no two systems are alike?

Miles Fidelman

unread,
Mar 18, 2012, 9:30:45 PM3/18/12
to dev...@googlegroups.com
I just sent Wayne an email. Main thing about presence/absence of a list
is as a way to build a user/developer community around a project.

Sean OMeara

unread,
Mar 18, 2012, 9:39:18 PM3/18/12
to dev...@googlegroups.com, dev...@googlegroups.com
Congratulations! You've broken the troll barrier!

Try actually building your next project in a proper CM tool and come back for your prize!

-s

John Vincent

unread,
Mar 18, 2012, 9:58:56 PM3/18/12
to dev...@googlegroups.com
On Sun, Mar 18, 2012 at 9:11 PM, Miles Fidelman
<mfid...@meetinghouse.net> wrote:
> Pardon me for being blunt as well, then.  I don't see a particularly high
> return on investment for investing time in converting to a tool (be it
> puppet, chef, cfengine, or what have you) that addresses only a small piece
> of an overall operations toolchain.  I'd rather invest time and effort in
> getting my existing toolchain more automated, documented, and under
> configuration control.
>
>
The tool you are looking for does not and probably will never exist.
You said yourself very early on that you like the unix way. CM is a
smaller part of a bigger picture and the tools that you specifically
mentioned address configuration management solely right now. Some of
them have compliments and more tightly integrated add-on (i.e.
mcollective) but you'll find that most of the tools around address one
specific problem and do it very well.

> You ever notice that no two systems are alike?

Congrats. I figured it would get to this point. You've just described
why you need proper configuration management:

Puppet -

package { "screen": ensure => "installed" }

Chef -

package "screen" do
action :install
end

Guess what. That works the same across all distros. Doesn't matter if
those systems are alike. But here's the real crux of the matter: the
point of configuration management is not to make your special
snowflake systems behave the same. It's to make sure you don't have
special snowflake systems. Systems are cattle...pets. They aren't
artisan crafted, flame broiled, freshly brewed works of art.

I'm very sorry if you haven't gotten a very good welcome to the list.
However, while we may not have a good definition of what devops *IS*,
most people here feel that encouraging a pattern of brittle,
busted-ass inflexible and ultimately unscalable shell scripts is wrong
and downright irresponsible.

Even more telling is the fact that the questions you're asking are the
first sign that you've grown beyond those shell scripts.

SM (nee BDSM) is probably up your alley. Wayne is a solid guy and he
wrote SM mainly out of frustration with a poorly implemented Chef
setup. While I wouldn't suggest it for the reasons I listed above,
Wayne takes his community and his code seriously. Kudos if SM gets you
further along your journey but at some point, you are (and I would
argue already have) reached the limits of your shell addiction.


--
John E. Vincent
http://about.me/lusis

Chad Woolley

unread,
Mar 18, 2012, 10:47:12 PM3/18/12
to dev...@googlegroups.com
On Sun, Mar 18, 2012 at 6:39 PM, Sean OMeara <som...@gmail.com> wrote:
> Congratulations! You've broken the troll barrier!
>
> Try actually building your next project in a proper CM tool and come back for your prize!

No idea what you mean, but I'm pretty sure it's unnecessarily snarky.
Git/GitHub is a proper CM tool in my book.

Andrew Clay Shafer

unread,
Mar 18, 2012, 10:47:45 PM3/18/12
to devops

Chad Woolley

unread,
Mar 18, 2012, 11:03:34 PM3/18/12
to dev...@googlegroups.com
On Sun, Mar 18, 2012 at 7:47 PM, Andrew Clay Shafer
<andrew...@gmail.com> wrote:
>
> http://www.youtube.com/watch?v=1sONfxPCTU0

I love you, man...

John Vincent

unread,
Mar 18, 2012, 11:06:34 PM3/18/12
to dev...@googlegroups.com

Andrew Clay Shafer

unread,
Mar 18, 2012, 11:20:34 PM3/18/12
to devops

You can either consistently and predictably manage systems at scale or
you can't.

Theoretically, there are better bash solutions than some of the puppet/
chef implementations I have seen.

Sure, there are benefits to these frameworks, but if you don't
understand the foundation they are built on, then they just look like
a lot more problem than they are worth.

Often, telling someone a truth they aren't ready to hear is
indistinguishable from lying to them.

Also, for the record, Configuration Management in nearly all 'devops'
conversations is going to refer to tools for configuring servers and
not version control systems. There is a bit of a namespace collision
there.

What we really need are thunderdome style system management battles.

Miles Fidelman

unread,
Mar 18, 2012, 11:22:24 PM3/18/12
to dev...@googlegroups.com

Chad, you beat me to it :-)

Miles Fidelman

unread,
Mar 18, 2012, 11:29:40 PM3/18/12
to dev...@googlegroups.com
John Vincent wrote:
> On Sun, Mar 18, 2012 at 9:11 PM, Miles Fidelman
> <mfid...@meetinghouse.net> wrote:
>> Pardon me for being blunt as well, then. I don't see a particularly high
>> return on investment for investing time in converting to a tool (be it
>> puppet, chef, cfengine, or what have you) that addresses only a small piece
>> of an overall operations toolchain. I'd rather invest time and effort in
>> getting my existing toolchain more automated, documented, and under
>> configuration control.
>>
>>
> The tool you are looking for does not and probably will never exist.
> You said yourself very early on that you like the unix way. CM is a
> smaller part of a bigger picture and the tools that you specifically
> mentioned address configuration management solely right now. Some of
> them have compliments and more tightly integrated add-on (i.e.
> mcollective) but you'll find that most of the tools around address one
> specific problem and do it very well.

Seems to me that rundeck and rerun, and possibly bdsm come pretty
close. Tcl/tk + git as well.

Tcl has been around forever. Rundeck I knew about, rerun and bdsm I
found out in responses to my query.

>> You ever notice that no two systems are alike?
> Congrats. I figured it would get to this point. You've just described
> why you need proper configuration management:
>
> Puppet -
>
> package { "screen": ensure => "installed" }
>
> Chef -
>
> package "screen" do
> action :install
> end
>

No. Configuration is a fairly evolve discipline for defining,
controlling, accounting for, and auditing system configurations.

CVS, Git, Subversion are configuration control systems.

Puppet and Chef are engines for applying configurations - with arguably
cleaner syntax than pure bash scripts.

> I'm very sorry if you haven't gotten a very good welcome to the list.
> However, while we may not have a good definition of what devops *IS*,
> most people here feel that encouraging a pattern of brittle,
> busted-ass inflexible and ultimately unscalable shell scripts is wrong
> and downright irresponsible.

--

Miles Fidelman

unread,
Mar 18, 2012, 11:56:01 PM3/18/12
to dev...@googlegroups.com

For the record, people have been consistently and predictably managing
systems, at very large scale, for a really long time (at least by the
standards of "internet time"). People have been deploying distribution
systems (e.g., electric distribution grids), telecom. networks, large
distributed systems (such as those found in defense environments),
financial trading systems, etc., etc., etc. for decades. (By the way,
my background is large comms. networks, largely supporting distributed
command and control systems.)

I think its Alex Honor who's powerpoint decks that lead with a slide
along the lines of "people -> process -> tools" -- the point being that
tools have to serve the organization and process, not the other way around.

If the goal is to deploy 10,000 hadoop instances, quickly - sure chef is
going to do a great job. If you're managing something more complicated,
process discipline is what counts - you can manage huge huge and
complicated operations with little more than checklists and
whiteboards. The worst thing you want to do is have your tools impose
inappropriate disciplines in environments where operating procedures
have been evolved and tested over time.

John Vincent

unread,
Mar 19, 2012, 12:08:53 AM3/19/12
to dev...@googlegroups.com
On Sun, Mar 18, 2012 at 11:29 PM, Miles Fidelman

<mfid...@meetinghouse.net> wrote:
> John Vincent wrote:
>>
>> On Sun, Mar 18, 2012 at 9:11 PM, Miles Fidelman
>> <mfid...@meetinghouse.net>  wrote:
>>>
>>> Pardon me for being blunt as well, then.  I don't see a particularly high
>>> return on investment for investing time in converting to a tool (be it
>>> puppet, chef, cfengine, or what have you) that addresses only a small
>>> piece
>>> of an overall operations toolchain.  I'd rather invest time and effort in
>>> getting my existing toolchain more automated, documented, and under
>>> configuration control.
>>>
>>>
>> The tool you are looking for does not and probably will never exist.
>> You said yourself very early on that you like the unix way. CM is a
>> smaller part of a bigger picture and the tools that you specifically
>> mentioned address configuration management solely right now. Some of
>> them have compliments and more tightly integrated add-on (i.e.
>> mcollective) but you'll find that most of the tools around address one
>> specific problem and do it very well.
>
>
> Seems to me that rundeck and rerun, and possibly bdsm come pretty close.
>  Tcl/tk + git as well.
>
> Tcl has been around forever.  Rundeck I knew about, rerun and bdsm I found
> out in responses to my query.
>
>

I'm confused then. If you're talking about Tcl, why are we still
discussing bash scripts? I don't really see where Tk comes into play.
If you need to slap a gui on something, you'd be better served not
burdening your fellow sysadmins and just sticking it behind a web
interface. I guess AOLServer is still a thing?

>>> You ever notice that no two systems are alike?
>>
>> Congrats. I figured it would get to this point. You've just described
>> why you need proper configuration management:
>>
>> Puppet -
>>
>> package { "screen": ensure =>  "installed" }
>>
>> Chef -
>>
>> package "screen" do
>>   action :install
>> end
>>
> No.  Configuration is a fairly evolve discipline for defining, controlling,
> accounting for, and auditing system configurations.
>
> CVS, Git, Subversion are configuration control systems.
>
> Puppet and Chef are engines for applying configurations - with arguably
> cleaner syntax than pure bash scripts.
>
>

Andrew has already addressed this but we seem to have a semantic
confusion here. As he said, in "devops" discussions CM pretty much
always refers to a tool akin to puppet, chef or cfengine. Git, SVN,
CVS, Hg, Bazaar I have NEVER heard referred to as a "configuration
control system". Version control system (VCS) is the term I've always
heard.

Having said that:

"> No.  Configuration is a fairly evolve discipline for defining, controlling,
> accounting for, and auditing system configurations."

If that's your definition then pretty much every tool discussed up to
this point accomplishes that task 100%. The nice thing about a CM (my
definition) tool is that's what its DESIGNED to do. I can tell you
that any system, regardless of being managed with Puppet or Chef (I
haven't done cfengine stuff in at least 8 years) I could meet all of
those requirements with my chef or puppet repo. So color me confused
as to what exactly these tools seem to be missing that DOESN'T meet
that need.

Christopher Webber

unread,
Mar 19, 2012, 12:49:59 AM3/19/12
to dev...@googlegroups.com

Puppet and Chef are engines for applying configurations - with arguably cleaner syntax than pure bash scripts.


I just want to point this out because it drives me insane... Puppet and Chef don't apply configuration, per say, and if you think about it that way you will never use them effectively. They manage and document the state of a system. You are not managing the apache config. You are documenting that the system should have apache installed, the config should look like X and that apache is running and then applying that documentation to the system. And, because it is code, we can put it in a VCS. 

-- cwebber

Miles Fidelman

unread,
Mar 19, 2012, 12:59:27 AM3/19/12
to dev...@googlegroups.com
John Vincent wrote:
> On Sun, Mar 18, 2012 at 11:29 PM, Miles Fidelman
> <mfid...@meetinghouse.net> wrote:
>> John Vincent wrote:
>>> The tool you are looking for does not and probably will never exist.
>>> You said yourself very early on that you like the unix way. CM is a
>>> smaller part of a bigger picture and the tools that you specifically
>>> mentioned address configuration management solely right now. Some of
>>> them have compliments and more tightly integrated add-on (i.e.
>>> mcollective) but you'll find that most of the tools around address one
>>> specific problem and do it very well.
>>
>> Seems to me that rundeck and rerun, and possibly bdsm come pretty close.
>> Tcl/tk + git as well.
>>
>> Tcl has been around forever. Rundeck I knew about, rerun and bdsm I found
>> out in responses to my query.
>>
>>
> I'm confused then. If you're talking about Tcl, why are we still
> discussing bash scripts? I don't really see where Tk comes into play.
> If you need to slap a gui on something, you'd be better served not
> burdening your fellow sysadmins and just sticking it behind a web
> interface. I guess AOLServer is still a thing?

Tcl is pretty good at composing command line executables - particularly
when you use expect. (tcl, after all, stands for "tool control
language") Tk lets you put a gui on it if you wish, but that's really
extraneous.

if you think that puppet or chef can support the formal configuration
control requirements of a large distributed system, including hardware
components, software components, configuration files, network devices
and configuration, crypto configuration, ..... - you just haven't dealt
with serious systems -- for that matter, when it comes to definitions,
the best taxonomy of devops that I've seen (of the few that seem to be
around) is the one at
http://code.google.com/p/devops-toolchain/wiki/ArchitectureComponents -
of which puppet/chef addresses only one small piece of an overall tool
chain - NOT including inventory (a key piece of any reasonable
configuration management process)

not to imply that they're not useful tools - but they certainly are only
one piece of provisioning, configuring, monitoring and controlling systems

in any case, my original query was specifically about tools for managing
and composing shell scripts - NOT a philosophical discussion of
competing operations styles and philosophies

if you want some elaboration of what I'm looking for, is a tool that,
ideally, would:


- provide for editing/managing/versioning scripts (script = anything
that can be invoked at the command line)
- a library of control functions for use within scripts
- invoking scripts, combinations of scripts, pipelines of scripts (in
the Unix sense of pipes)
- locally, remotely, across multiple machines
- remote script execution via ssh, rather than some kind of agent
- providing a simple database for keeping track of variables used by
scripts (e.g., IP addresses, DNS records for use by a provisioning
script) - that can be accessed from scripts
- accessing the above via cli, RESTful API, GUI
- cross-platform
- (nice-to-have) minimal environmental requirements (i.e., a step above
the unix shell, say the gnu buildtool suite rather than the JVM + a mass
of libraries, or the ruby ecosystem -- a self-configuring basic
environment would be nice, like perl+cpan)

- tcl has been used this way for a long time, but has its limits (on the
other hand, it has some nice features for interacting with databases -
which makes it useful for, say, linking a virtual-machine provisioning
script to a database of IP address assignments)
- rundeck and rerun are relatively new and get very close
- bdsm also seems to be in the same ballpark

tcl and rundeck I knew about - I have to thank this list and the
devops-toolchain list for pointers to rerun and bdsm, and some
discussion thereof

Miles Fidelman

unread,
Mar 19, 2012, 1:02:22 AM3/19/12
to dev...@googlegroups.com

Ok, if you want to get pedantic about it - they are tools that "coerce"
a system to a specific state (using the technical definition of the word).

John Vincent

unread,
Mar 19, 2012, 1:10:44 AM3/19/12
to dev...@googlegroups.com
On Mon, Mar 19, 2012 at 12:59 AM, Miles Fidelman

<mfid...@meetinghouse.net> wrote:
> John Vincent wrote:
>>
> if you think that puppet or chef can support the formal configuration
> control requirements of a large distributed system, including hardware
> components, software components, configuration files, network devices and
> configuration, crypto configuration, ..... - you just haven't dealt with
> serious systems --

This is where I bow out of the conversation. I don't feel the need to
get into a measuring contest over who's managed bigger environments
and the relative merit therein. I'm perfectly happy with the size of
my infrastructures over the years.

> if you want some elaboration of what I'm looking for, is a tool that,
> ideally, would:
>
> - provide for editing/managing/versioning scripts (script = anything that
> can be invoked at the command line)
> - a library of control functions for use within scripts
> - invoking scripts, combinations of scripts, pipelines of scripts (in
> the Unix sense of pipes)
> - locally, remotely, across multiple machines
> - remote script execution via ssh, rather than some kind of agent
> - providing a simple database for keeping track of variables used by
> scripts (e.g., IP addresses, DNS records for use by a provisioning
> script) - that can be accessed from scripts
> - accessing the above via cli, RESTful API, GUI
> - cross-platform
> - (nice-to-have) minimal environmental requirements (i.e., a step above
> the unix shell, say the gnu buildtool suite rather than the JVM + a mass
> of libraries, or the ruby ecosystem -- a self-configuring basic
> environment would be nice, like perl+cpan)
>

Again, congratulations you just just described both Chef and Puppet.
The fact that you've not done enough research (or if you had, you
wouldn't still be using this as your baseline to DISPROVE those as
valid tools), suggests that others are right and you're most likely
trolling at this point.

Good luck with your search.

Christopher Webber

unread,
Mar 19, 2012, 1:14:31 AM3/19/12
to dev...@googlegroups.com

:-(

This isn't about being pedantic, it is about understanding tools and how you use them. Sure you can take a pair of pliers to remove a nut, but a wrench is a much better tool. All I was trying to say was that Puppet and Chef are often looked at as a way to replace shell scripts directly. They are not. Instead, they solve the problem in a completely different way that causes you to not need the shell scripts to begin with. I get it, you are happy with shell scripts. But just as there are times where you have to use some other language, generally perl in years past, there comes a time where you realize that managing with shell scripts is fundamentally flawed. Your arguments really are akin to telling me how amazing the Oracle installer is. I am sure that it was great compared to what was around when it was first conceived, but I would tell you that is the best example of how not to install software.

-- cwebber

ranjib dey

unread,
Mar 19, 2012, 1:22:25 AM3/19/12
to dev...@googlegroups.com
So you mean you have dealt with seriously large infrastructure with bash and tcl and tk (i dont really get it how tk helps anywhere?) , and chef or puppet can not bring network or crypto devices in the cm purview? I dont agree to that. I have delt with distributed infrastructure that spread across 18 offices, 4 different hybrid cloud setups, infrastructure of a large travelling and an mobile ad agency that consumes 100 transactions / second. I do think these are significantly large installations. Also there are ample example of people using chef or puppet at scale. There are patterns when you used then passively, (like  you can generate your cisco asa's configs via chef and then version control it, but you apply it manually ), but those are situation specific adaptation. And chef/puppet both are pretty effective.

You might have dealt with more serious installations.. but there might be a chance you are living in a shell and the shell became your reality.. u just cant see or experience the change.

Andrew Clay Shafer

unread,
Mar 19, 2012, 1:44:27 AM3/19/12
to devops

Miles, juju is what you want.

Miles Fidelman

unread,
Mar 19, 2012, 1:51:48 AM3/19/12
to dev...@googlegroups.com
ranjib dey wrote:
> So you mean you have dealt with seriously large infrastructure with
> bash and tcl and tk (i dont really get it how tk helps anywhere?) ,
> and chef or puppet can not bring network or crypto devices in the cm
> purview? I dont agree to that. I have delt with distributed
> infrastructure that spread across 18 offices, 4 different hybrid cloud
> setups, infrastructure of a large travelling and an mobile ad agency
> that consumes 100 transactions / second. I do think these are
> significantly large installations. Also there are ample example of
> people using chef or puppet at scale. There are patterns when you used
> then passively, (like you can generate your cisco asa's configs via
> chef and then version control it, but you apply it manually ), but
> those are situation specific adaptation. And chef/puppet both are
> pretty effective.

No. I've dealt with seriously large infrastructure that uses LOTS of
different tools, of which scripts are a big piece. So is Nagios, so are
trouble ticket systems.

And no, 18 offices and 100 transactions/second is a trivial sized
operation. 10,000 spread across a university, or a battlefield, starts
to be a large infrastructure. Bank of America, NASDAQ, Fedwire,
MasterCard's transaction network - those are large systems. Now Google
is an example of a truly huge system.


>
> You might have dealt with more serious installations.. but there might
> be a chance you are living in a shell and the shell became your
> reality.. u just cant see or experience the change.
>

No. I'm asking about tools for managing shell scripts. You, on the
other hand, seem to be living in a reality that consists of chef,
puppet, and nothing else. Seems like a case of "if the only tool you
have is a hammer, every problem looks like a nail."

Miles Fidelman

unread,
Mar 19, 2012, 2:01:24 AM3/19/12
to dev...@googlegroups.com
Andrew Clay Shafer wrote:
> Miles, juju is what you want.
If you're referring to Ubuntu juju - it looks interesting (if a bit
cutesy - I mean "charms", really? :-). Seems to only work for deploying
to EC2 instances though - I'm focused on real hardware for the moment,
with cloud VMs somewhere down the line. Ubuntu Orchestra looks pretty
cool - though specific to Ubuntu server.

Erik Hollensbe

unread,
Mar 19, 2012, 2:39:02 AM3/19/12
to dev...@googlegroups.com

On Mar 18, 2012, at 10:51 PM, Miles Fidelman wrote:
>> You might have dealt with more serious installations.. but there might be a chance you are living in a shell and the shell became your reality.. u just cant see or experience the change.
>>
> No. I'm asking about tools for managing shell scripts. You, on the other hand, seem to be living in a reality that consists of chef, puppet, and nothing else. Seems like a case of "if the only tool you have is a hammer, every problem looks like a nail."

Just to take this argument to the next level of absurdity -- why not scrap your shell scripts then, and just write everything in C?

-Erik

Benjamin Krueger

unread,
Mar 19, 2012, 3:08:45 AM3/19/12
to dev...@googlegroups.com
In my humble career, I've found that those who have experience, competence, or both don't need to or tend to brag. And those who are arrogant back their boasts up with evidence, rather than hinting, beating around the bush at how amazing they and the things they've worked on are, and dropping names. "You know the Army and Google? Yeah, I totally know those guys." Yeah, ok. That's nice.

You've made your point. You have descended from heaven to bless us mortals with your wisdom. We reject it because it's just beyond the ability of our tiny brains to comprehend. Etc etc. You can go back to your palace in the sky now please. Come back when you can respect even the "little guys" and the rest of us who are more interested in learning, building, and sharing what we find than throwing our giant egos around.

On Mar 18, 2012, at 10:51 PM, Miles Fidelman wrote:

Paul Graydon

unread,
Mar 19, 2012, 3:21:14 AM3/19/12
to dev...@googlegroups.com
On 3/18/2012 7:51 PM, Miles Fidelman wrote:
>>
>> You might have dealt with more serious installations.. but there
>> might be a chance you are living in a shell and the shell became your
>> reality.. u just cant see or experience the change.
>>
> No. I'm asking about tools for managing shell scripts. You, on the
> other hand, seem to be living in a reality that consists of chef,
> puppet, and nothing else. Seems like a case of "if the only tool you
> have is a hammer, every problem looks like a nail."
>
Pot, kettle, black?
ou're living in a reality in which apparently consists of shell scripts,
SSH and nothing else.

You've outright written off other tools, apparently without
understanding them given that you're telling people they can't do what
those people are using them for.
On the other hand, many of us here have been down the shell script + SSH
route, and dealt with infrastructure heavily based upon it. We know
what it's like, and many of us probably know what it's like to inherit
such an infrastructure. (For myself, I worked in a good sized ISP who's
complex infrastructure heavily relied on such techniques, last I heard
they were migrating to Puppet.)

You came in and asked a question, people took time out of their day to
give you answers based on their real world experience managing
infrastructures small and large and you have basically spent the entire
thread calling them all idiots and liars.

I'm assuming it'll therefore come as no surprise that there has been a
bit of discussion off-list about whether or not you're trolling. From
what I've seen the prevailing opinion is that you are, and some of those
opinions are coming from people who are managing infrastructures larger
than what you're talking about who haven't bothered participating once
they saw your attitude.

As far as I'm concerned: Take it, or leave it.

You came and asked for advice, you got it. I'm sorry it doesn't appear
to be the advice that you were seeking which seems to be reassurance
that your way of doing this is the one true and holy way if only you
could find ways to make it work better.

Paul

Ernest Mueller

unread,
Mar 19, 2012, 8:35:31 AM3/19/12
to dev...@googlegroups.com
At 02:21 AM 3/19/2012, Paul Graydon wrote:
>Pot, kettle, black?
>ou're living in a reality in which apparently consists of shell
>scripts, SSH and nothing else.

I'm not sure someone's preference for shell scripts requires us to
become assholes to them.

Not everyone has drunk the chef/puppet kool-aid and that's fine. I
think he has some good points about wanting to initially invest his
time in something else and that these tools only form a small part of
the toolchain. That's totally correct. At NI we spent our time
writing PIE instead of uptaking chef or puppet, because provisioning
(and frankly those only do the middle layer of provisioning well,
leaving system spinup and app deployment to other components) wasn't
our most pressing need.

There's also some good points from Andrew about chef/puppet not being
a magical solution. We have a puppet implementation here at BV that
is frustrating to us because in retrospect, it was implemented in a
pretty convoluted and fragile way (in good faith, by following the
random Internet help out there) and isn't all that much better at the
moment (It has promise, yadda yadda) than the old
nest-of-shell-scripts deployment pipeline we have.

Let's not all confuse our preferred approach with the One True Way,
and let's not confuse DevOps with using the shiniest tools.

Ernest

Miles Fidelman

unread,
Mar 19, 2012, 9:20:35 AM3/19/12
to dev...@googlegroups.com
Paul Graydon wrote:
> On 3/18/2012 7:51 PM, Miles Fidelman wrote:
>>>
>>> You might have dealt with more serious installations.. but there
>>> might be a chance you are living in a shell and the shell became
>>> your reality.. u just cant see or experience the change.
>>>
>> No. I'm asking about tools for managing shell scripts. You, on the
>> other hand, seem to be living in a reality that consists of chef,
>> puppet, and nothing else. Seems like a case of "if the only tool you
>> have is a hammer, every problem looks like a nail."
>>
> Pot, kettle, black?
> ou're living in a reality in which apparently consists of shell
> scripts, SSH and nothing else.
>
> You've outright written off other tools, apparently without
> understanding them given that you're telling people they can't do what
> those people are using them for.
<snip>

> On the other hand, many of us here have been down the shell script +
> SSH route, and dealt with infrastructure heavily based upon it. We
> know what it's like, and many of us probably know what it's like to
> inherit such an infrastructure. (For myself, I worked in a good sized
> ISP who's complex infrastructure heavily relied on such techniques,
> last I heard they were migrating to Puppet.)
>
> You came in and asked a question, people took time out of their day to
> give you answers based on their real world experience managing
> infrastructures small and large and you have basically spent the
> entire thread calling them all idiots and liars.

No... I asked for suggestions and experiences re. tools for managing
shell scripts. I got precious few answers (a few good ones). If I
asked for help with a Debian boot problem, I wouldn't expect answers
about CentOS boot problems, or why I should be using OS X, or writing my
code in C.

> You came and asked for advice, you got it. I'm sorry it doesn't
> appear to be the advice that you were seeking which seems to be
> reassurance that your way of doing this is the one true and holy way
> if only you could find ways to make it work better.

Opinions and mileage may vary. Obviously some well-respected folks in
the devops world don't find shell scripts and SSH completely obsolete,
or easy to abandon - or we wouldn't have rundeck and rerun floating around.

I asked for input on something specific, and got religious arguments and
suggestions that I'm a troll. Sheesh. (along with a few good
suggestions - for which I thank people).

Miles Fidelman

unread,
Mar 19, 2012, 9:21:15 AM3/19/12
to dev...@googlegroups.com
Ernest,

Thank you!

Miles

Joe McDonagh

unread,
Mar 19, 2012, 10:08:00 AM3/19/12
to dev...@googlegroups.com
On 3/19/12 12:59 AM, Miles Fidelman wrote:
>
> if you think that puppet or chef can support the formal configuration
> control requirements of a large distributed system, including hardware
> components, software components, configuration files, network devices
> and configuration, crypto configuration, ..... - you just haven't
> dealt with serious systems -- for that matter, when it comes to
> definitions, the best taxonomy of devops that I've seen (of the few
> that seem to be around) is the one at
> http://code.google.com/p/devops-toolchain/wiki/ArchitectureComponents
> - of which puppet/chef addresses only one small piece of an overall
> tool chain - NOT including inventory (a key piece of any reasonable
> configuration management process)
>
Puppet and Chef are infrastructure automation *frameworks*. You seem to
be under the impression they are simply tools. Thinking about those two
pieces of software in this way is really limiting. And I'm not entirely
sure what you're getting at in that last comment, since Puppet and Chef
have features in mind to satisfy inventory especially.

> not to imply that they're not useful tools - but they certainly are
> only one piece of provisioning, configuring, monitoring and
> controlling systems
Again, each of those activities can be setup and automated by the
previously mentioned pieces of software. Therefore, stop thinking about
them as a tiny piece of the pie and actually an ingredient in each slice.

>
> in any case, my original query was specifically about tools for
> managing and composing shell scripts - NOT a philosophical discussion
> of competing operations styles and philosophies
>
> if you want some elaboration of what I'm looking for, is a tool that,
> ideally, would:
> - provide for editing/managing/versioning scripts (script = anything
> that can be invoked at the command line)
> - a library of control functions for use within scripts
> - invoking scripts, combinations of scripts, pipelines of scripts (in
> the Unix sense of pipes)
> - locally, remotely, across multiple machines
> - remote script execution via ssh, rather than some kind of agent
> - providing a simple database for keeping track of variables used by
> scripts (e.g., IP addresses, DNS records for use by a provisioning
> script) - that can be accessed from scripts
> - accessing the above via cli, RESTful API, GUI
> - cross-platform
> - (nice-to-have) minimal environmental requirements (i.e., a step above
> the unix shell, say the gnu buildtool suite rather than the JVM + a mass
> of libraries, or the ruby ecosystem -- a self-configuring basic
> environment would be nice, like perl+cpan)
>
>
Why wouldn't you just develop a solution built around an existing
popular automation framework? I can tell you that it sucks to inherit
special snowflake systems and scripts, because if you're like about the
lion's share of sys admins in this field, your speshul codez give me a
migraine and sometimes make me weep tears of blood.

--
Joe McDonagh

AIM: YoosingYoonickz Google: Joseph.E.McDonagh
IRC: joe-mac Skype: therealjoemac

jtimberman

unread,
Mar 19, 2012, 10:19:55 AM3/19/12
to dev...@googlegroups.com
On Monday, March 19, 2012 7:20:35 AM UTC-6, Miles Fidelman wrote:

No... I asked for suggestions and experiences re. tools for managing
shell scripts.  I got precious few answers (a few good ones). 

I asked for input on something specific, and got religious arguments and

suggestions that I'm a troll.  Sheesh.  (along with a few good
suggestions - for which I thank people).

So let us know how it goes, which tool you pick, and detail about how well it works with your shell code, and stop with the grandstanding "my infrastructure is bigger than yours so those other tools won't work for me."

 

damone...@gmail.com

unread,
Mar 19, 2012, 5:38:35 PM3/19/12
to dev...@googlegroups.com
Ernest's points deserve highlighting. Not acknowledging the realities of the world some people live in and expecting everyone to conform to ours seems like a sure fire way to kill the DevOps momentum and goodwill.  

Most of the time DTO get's called into "brownfield" organizations (or "greenpatch" where it's a "lets do it right" project inside a legacy mess). There are years worth of people, politics, processes, entrenched habits, and general organizational spaghetti to unravel.  Automation toolchains are just one part of a much broader organizational transformation endeavor.

If it's a true greenfield (tooling AND people) or a small enough org where you can get everyone in the same classroom, it's simple enough just to say "for the configuration management and modular automation portion of the toolchain pick Puppet, Chef, or Cfengine", show them what to do with it, and move on since it's likely that there are way bigger problems hurting the organization. But for brownfield and greenpatch there are a myriad of reasons at the organization level as to why they often can't make that wholesale jump to the new "right way" to do automation.

So what should we do? How do you help an organization who has the desire to move down the path towards DevOps enlightenment but for political, logistical, or budgetary constraints can't make that quantum leap to what Botchagalupe coined the third generation configuration management tools? 

This is where rerun came in. The specific scenario we kept coming across was large organizations who had massive investment in shell scripts (scripts of all flavors really) both in terms of working code and the skills of dozens or even hundreds of people. The individual DevOps thought leaders inside these companies saw the benefit of tools like Puppet and Chef but the non-technical constraints placed on them were holding them back. Someday they knew they could go there, but what were they to do in the meantime?

There is nothing inherently wrong with shell. It is the unruly and ad-hoc ways in which it is used that causes the problems. Rerun was designed to provide "just enough framework" so that those unruly scripts could become reusable modular automation with standard interfaces. This allows those DevOps pioneers in these companies to easily teach people how to shoehorn their mountain of perfectly adequate shell scripts into a broader, modern toolchain (which could already include Puppet or Chef in other parts).  It allows them to start teaching modern techniques to their workforce without a costly and high-stakes rip and replacement of how everyone works. 

Rerun is meant to be a temporary step on a longer journey. It's a stub that allows the old to be part of the toolchain of the new. It's not supposed to replace Chef or Puppet, nor could it. In fact it speeds the adoption of Chef or Puppet since it helps to teach the basics of modularity, reuse, and a shared framework. Once you get a taste for that powerful sauce, the next step is move up to Chef or Puppet to get your fix. 

Read more about the "why?" behind rerun:

Is rerun still appropriate for "DevOps" because it's made to tame shell scripts? Of course it is. Is it the ideal fresh and shiny power tool? No, its a practical fix that gets you around serious non-technical constraints. 

In every situation where rerun went in, if you stand back far enough the benefits of the solution look the same as if the heart of the toolchain was Puppet or Chef. Sure the benefits might not have been nearly as great and upon closer observation there is still lingering technical debt that will have to be paid off in the future when they do move to Puppet or Chef. But in the meantime their DevOps problems did get better in the face of serious constraints and, hopefully, we all gained another set of converts to the DevOps way of thinking.

So if people are coming to this list… the DevOps list… not Puppet/Chef/InfrastructureAsCode list... with questions about how they can move along the DevOps path given their current constraints (perceived or real), what do we gain by ignoring their constraints and shouting them down? 


-Damon

Miles Fidelman

unread,
Mar 19, 2012, 6:39:59 PM3/19/12
to dev...@googlegroups.com
Damon,

Thank you very much for your measured comments - and for both rerun and
rundeck, one or both of which I expect I might be using in the near future.

I've also become a big fan of the various presentations from you, Alex
Honor, and the crew at DTO Solutions - I've found them very insightful
vis-a-vis real-world approaches to operational systems management.
Particularly DTO's standard slide on process coming before tools (one of
my personal peeves, over the years, are people who design tools that
require massive changes in operations processes - there's a reason that
"groupware" has never really caught on, but that's a discussion for
another time and place).

A couple of questions and comments below, inline:

damone...@gmail.com wrote:
>
> This is where rerun came in. The specific scenario we kept coming
> across was large organizations who had massive investment in shell
> scripts (scripts of all flavors really) both in terms of working code
> and the skills of dozens or even hundreds of people. The individual
> DevOps thought leaders inside these companies saw the benefit of tools
> like Puppet and Chef but the non-technical constraints placed on them
> were holding them back. Someday they knew they could go there, but
> what were they to do in the meantime?

Not just large organizations. I'm currently in a tiny one, wearing
multiple hats. In larger settings, we had the luxury of turning someone
loose to learn and implement new tools. That's a bit harder to do when
everyone is wearing too many hats. Quick and dirty tends to be very
efficient, but not scalable - and that's the situation I find myself
in. Putting scripts, and scraps of paper, and notes on whiteboards, and
spreadsheets containing IP address assignments and DNS under some
measure of configuration control is a good first step toward being able
to replicate/restore systems, and toward sharing knowledge and
responsibilities across a group.

> <snip>


>
> In every situation where rerun went in, if you stand back far enough
> the benefits of the solution look the same as if the heart of the
> toolchain was Puppet or Chef. Sure the benefits might not have been
> nearly as great and upon closer observation there is still lingering
> technical debt that will have to be paid off in the future when they
> do move to Puppet or Chef. But in the meantime their DevOps problems
> did get better in the face of serious constraints and, hopefully, we
> all gained another set of converts to the DevOps way of thinking.

It strikes me that an orchestration framework - be it rundeck, or
capistrano, or mcollective, or what have you - can be a more effective
starting point for automation than something like puppet or chef. I
wonder if you might have an opinion on that?

And... I find that one of the most time consuming operational headaches
is inventory tracking - particularly of things like IP address
assignments, DNS records, user accounts and privileges, and so forth.
It's one thing to use a tool to build and configure a system (say FAI +
chef), but there's a manual step involved in pulling a bunch of IP
addresses off a list and plugging them into the configuration files (or
recipes, or what have you). On the devops-toolchain list, Noah Campbell
reported some shortcoming of CMDB approaches (notably keeping them in
sync with reality), and discussed some of the issues sourcing
information when running stuff under Rundeck control. I'd also be very
interested if you have something to add in this regard.

Thanks Again,

Miles Fidelman

Chad Woolley

unread,
Mar 19, 2012, 6:53:25 PM3/19/12
to dev...@googlegroups.com
On Mon, Mar 19, 2012 at 2:38 PM, <damone...@gmail.com> wrote:
In every situation where rerun went in, if you stand back far enough the benefits of the solution look the same as if the heart of the toolchain was Puppet or Chef. Sure the benefits might not have been nearly as great and upon closer observation there is still lingering technical debt that will have to be paid off in the future when they do move to Puppet or Chef. But in the meantime their DevOps problems did get better in the face of serious constraints and, hopefully, we all gained another set of converts to the DevOps way of thinking.

 
Good point.  To further illustrate the point, consider the following continuum of scenarios:

* Manually executed commands (yee-haw I'm a cowboy)
* Homegrown shell scripts run manually
* Homegrown shell scripts run via SSH, by other homegrown shell scripts
* Homegrown shell scripts + ssh, with some shell-based framework like rerun/rundeck
* Chef-solo, run via SSH, by shell scripts
* Chef-solo, run via SSH, by Capistrano
* Chef client-server (i can haz 1337 automation skillz)

Feel free to replace the bottom bullets with whatever you feel is more 1337 than Chef.

Point being, as long as people are trying to improve their organization's operations and CM(tm), that's a Good Thing.

Should we discourage them if they want to (or are forced to) only use shell scripts to achieve those goals?  Or if they don't use or CM(tm) tools of choice?  Of course not.

At what point do are these activities blessed and called "DevOps"?  Who cares...

-- Chad

Eric Shamow

unread,
Mar 19, 2012, 6:58:49 PM3/19/12
to dev...@googlegroups.com

Should we discourage them if they want to (or are forced to) only use shell scripts to achieve those goals?  Or if they don't use or CM(tm) tools of choice?  Of course not.

At what point do are these activities blessed and called "DevOps"?  Who cares...

I think we're losing track of what went on here.  The gentleman in question espoused the viewpoint that none of the tools suggested to him were possibly capable of the sophistication necessary to manage his "complex" environment, and he suggested that those who disagreed clearly didn't have experience with sufficiently complex infrastructures.

While we might have done better as an advocacy group in sheparding him down the right path, I think the superior "hand me a solution, and not any of the ones you have all figured out already, because they clearly aren't special enough for me" attitude put folks, myself included, off.  For what it's worth there are more productive ways to start a conversation.

So we should have been more mature in our response, and you are right that we should be bringing people along the route we've all discovered…but I don't think this was a case of someone being hounded out for not buying the One True Way so much as someone who dismissed the collective expertise of the group as a whole.  I suspect this is why there was widespread suspicion that it was a troll.

-Eric

damone...@gmail.com

unread,
Mar 19, 2012, 7:24:19 PM3/19/12
to dev...@googlegroups.com
Ah I see. I guess coming late to the party I missed that nuance across all of the different threads. That's a different matter all together.

Sidebar: how did this conversation jump from devops-t...@googlegroups.com, where is started and seemed to belong, to dev...@googlegroups.com?

Eric Shamow

unread,
Mar 19, 2012, 7:26:04 PM3/19/12
to dev...@googlegroups.com
Not to be overly snarky, but Mr My Environment is Bigger Than Yours……cross-posted.  I had lost track of it too until folks pointed it out to me off-channel.

-Eric

-- 
Eric Shamow
Sent with Sparrow

damone...@gmail.com

unread,
Mar 19, 2012, 7:35:36 PM3/19/12
to dev...@googlegroups.com
That's exactly the kind of progression we normally see. 

Although I should point out that rerun and Rudeck are different projects and completely separate tools. While I suspect a lot of rerun users are Rundeck users, the majority of Rundeck users are not using rerun. Rundeck with Puppet or Chef is the most common usage and what we generally recommend (hence James Turnbull's surprise earlier in the devops-toolchain part of thread that there were Rundeck users who weren't either Puppet of Chef users).

Miles Fidelman

unread,
Mar 19, 2012, 8:37:43 PM3/19/12
to dev...@googlegroups.com
Eric Shamow wrote:
>
>> Should we discourage them if they want to (or are forced to) only use
>> shell scripts to achieve those goals? Or if they don't use or CM(tm)
>> tools of choice? Of course not.
>>
>> At what point do are these activities blessed and called "DevOps"?
>> Who cares...
>
> I think we're losing track of what went on here. The gentleman in
> question espoused the viewpoint that none of the tools suggested to
> him were possibly capable of the sophistication necessary to manage
> his "complex" environment, and he suggested that those who disagreed
> clearly didn't have experience with sufficiently complex infrastructures.

Actually, no. I asked a very specific question about tools for managing
shell scripts. When things degenerated into religious arguments, I
ventured the observation that complex environments have been around a
lot longer than chef or puppet, and the opinion that I'm not quite sure
that the newest, shiniest tools address various complex environments
I've seen over the years. Perhaps I over-reacted, but then I kind of
have that reaction to those who preach the "one true way."

Miles Fidelman

Miles Fidelman

unread,
Mar 19, 2012, 8:43:28 PM3/19/12
to dev...@googlegroups.com
damone...@gmail.com wrote:
> Ah I see. I guess coming late to the party I missed that nuance across all of the different threads. That's a different matter all together.
>
> Sidebar: how did this conversation jump from devops-t...@googlegroups.com, where is started and seemed to belong, to dev...@googlegroups.com?

My fault. Frankly, I didn't know which was a more appropriate list.
Devops seems like the arena where sys admin tools are discussed these
days, but I couldn't quite tell from the archives which list was more
appropriate, so I posted to both. (Now if someone knows a good sys
admin list, rather than a devops list, I'd welcome a pointer. I know
where the network operations folks hang out, but not the sysops.)

Paul Graydon

unread,
Mar 19, 2012, 8:57:46 PM3/19/12
to dev...@googlegroups.com
On Mon, Mar 19, 2012 at 2:43 PM, Miles Fidelman
<mfid...@meetinghouse.net> wrote:
>I know where the network operations
> folks hang out, but not the sysops.)

NANOG, where even saying "Good morning" results in an argument that
lasts several days.

For straight sysadmin stuff, you could try the LOPSA mailing lists:
https://lopsa.org/mailinglists

David Symons

unread,
Mar 20, 2012, 10:16:16 AM3/20/12
to dev...@googlegroups.com
Hi all,

So given the above - thoughts/opinions aside, who has really s