tools for managing and composing shell scripts?

396 views
Skip to first unread message

Miles Fidelman

unread,
Mar 18, 2012, 10:03:52 AM3/18/12
to dev...@googlegroups.com
Over the years, I've built up a lot of shell scripts for provisioning
and configuration - which makes converting to something like chef or
puppet a somewhat daunting task (or even something like FAI).

I'm also a big fan of composing tool chains from piece parts - scripts
linked by pipes, mashups of functions accessed via RESTful APIs, etc.

Rundeck looks pretty interesting - given that it can manage shell
scripts and their execution via ssh. I'm sort of wondering what other
kinds of tools people are using for adding a job and configuration
control overlay to shell scripts. And what about orchestration tools
that that take a "mashup maker" approach? Pointers, thoughts, ....

Thanks,

Miles Fidelman

--
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra


Bryan Berry

unread,
Mar 18, 2012, 11:01:34 AM3/18/12
to dev...@googlegroups.com
miles,

we've all been there and we have all tried to shoehorn our shell scripts into a modular system. Spare yourself the pain, it's a losing game. You have to bite the bullet and learn a configuration management system, be it cfengine, puppet, chef, or foobarbaz

It may seem daunting but let me tell you it is absolutely worth the effort

I have actually recorded two podcasts on related topics and it may be worth your time to listen:
http://www.foodfightshow.org/2012/03/episode-5-getting-started-with-chef.html   about getting started w/ chef but most of the concepts also apply to puppet

Zack Williams

unread,
Mar 18, 2012, 11:03:34 AM3/18/12
to dev...@googlegroups.com
On Sun, Mar 18, 2012 at 7:03 AM, Miles Fidelman
<mfid...@meetinghouse.net> wrote:
> Over the years, I've built up a lot of shell scripts for provisioning and
> configuration - which makes converting to something like chef or puppet a
> somewhat daunting task (or even something like FAI).
>
> I'm also a big fan of composing tool chains from piece parts - scripts
> linked by pipes, mashups of functions accessed via RESTful APIs, etc.
>
> Rundeck looks pretty interesting - given that it can manage shell scripts
> and their execution via ssh.  I'm sort of wondering what other kinds of
> tools people are using for adding a job and configuration control overlay to
> shell scripts.  And what about orchestration tools that that take a "mashup
> maker" approach?  Pointers, thoughts, ....

Are you looking for a script organization tool, or a one-to-many
execution tool?

I'm a fan of using Rakefiles to organize a bunch of one- or few- liner
scripts into a namespaced whole. I generally wrap these with small
boilerplate shell script that makes them filesystem location
independent and allows me to pass variables in as required. I keep
these in git, for replication/backup.

You also get the full power of ruby so it's a good gateway drug to
other systems...

- Zack

gareth rushgrove

unread,
Mar 18, 2012, 12:10:26 PM3/18/12
to dev...@googlegroups.com
> other systems…
>

I'd echo Bryan with regards the worth of looking at the (sometimes
large and potentially intimidating) configuration management tools.

On the other hand I do think there is a place for more adhoc script
execution things, especially in smaller but still dynamic environments
and mainly for asking questions of the systems. I've used Fabric
(http://fabfile.org/) successfully here, partly because it provides
the framework for managing hosts and roles and doing parallel
execution, and the arguments can just be shell scripts. Once it's
setup anyone who's happy with bash can easily jump in, and it's a good
way of collecting useful one-liners. Past a certain number of hosts
something like mcollective or salt are probably more useful, they use
(different) message brokers to avoid problems with lots of ssh
sessions.

G

> - Zack

--
Gareth Rushgrove
Web Geek

morethanseven.net
garethrushgrove.com

Bryan Berry

unread,
Mar 18, 2012, 12:22:31 PM3/18/12
to dev...@googlegroups.com
+1 for fabric, its an awesome adhoc scripting tool

that said, I rarely reach for it as knife scripts are even easier to write w/ the knife exec sub-command. 

Does puppet has something like knife exec?

My $0.02, it is worth diving into puppet/chef/foobarbaz and later filling in the gaps w/ fabric/capistrano/mcollective. However, fabric and capistrano have much lower learning curves. I just found that I had written fabfiles that would have been easier to implement (and idempotent) w/ chef recipes.

Miles Fidelman

unread,
Mar 18, 2012, 1:52:33 PM3/18/12
to dev...@googlegroups.com
Bryan Berry wrote:
> miles,
>
> we've all been there and we have all tried to shoehorn our shell
> scripts into a modular system. Spare yourself the pain, it's a losing
> game. You have to bite the bullet and learn a configuration management
> system, be it cfengine, puppet, chef, or foobarbaz
>
> It may seem daunting but let me tell you it is absolutely worth the effort
>

Well you've obviously drunk the koolaid :-)

But that's not the problem I'm trying to address. As far as I can tell,
pretty much any configuration management system is only one part of a
toolchain - and right now, I don't see any complete toolchains, many of
the tools seem don't seem to compose very well (e.g., point me at
something that ties together provisioning with IP address management
with DNS record management), and a lot of pieces seem to be distribution
specific (e.g., FIA for debian, kickstarter for redhat-like distros).

Re. configuration tools specifically: I don't find chef, or puppet, or
bcg2, or cfengine recipes/rules all that clearer or more powerful than a
shell script, or perhaps a tcl script with expect statements - while I
find the restrictions associated with picking one of these something I'm
not ready to do.

Hence my interest in something like rundeck, that lets you manage and
compose scripts, AND configurations stored in other systems.

Miles Fidelman

unread,
Mar 18, 2012, 2:02:23 PM3/18/12
to dev...@googlegroups.com
Zack,

Rake looks pretty interesting. Not sure I want to bit off ruby (doing
most of our development in Erlang and JavaScript at the moment), but it
does seem to do some useful things.

Any thoughts on Rake vs. tcl/tk?

Thanks!

Miles

ranjib dey

unread,
Mar 18, 2012, 4:14:11 PM3/18/12
to dev...@googlegroups.com
tcl is a full blown scripting language (with tk being the ui part), you can compare it against ruby. While rake is a ruby library that gives nice wrapper around shell , introduces a notion of task and let you establish dependencies among tasks. Its like a build tool. consider it analogous of make or ant

AJ Christensen

unread,
Mar 18, 2012, 4:17:05 PM3/18/12
to dev...@googlegroups.com

Is this a joke thread? Still not quite gotten the hang of American sarcasm.

--AJ

Miles Fidelman

unread,
Mar 18, 2012, 4:25:13 PM3/18/12
to dev...@googlegroups.com
Zack Williams wrote:
> On Sun, Mar 18, 2012 at 7:03 AM, Miles Fidelman
> <mfid...@meetinghouse.net> wrote:
>> Over the years, I've built up a lot of shell scripts for provisioning and
>> configuration - which makes converting to something like chef or puppet a
>> somewhat daunting task (or even something like FAI).
>>
>> I'm also a big fan of composing tool chains from piece parts - scripts
>> linked by pipes, mashups of functions accessed via RESTful APIs, etc.
>>
>> Rundeck looks pretty interesting - given that it can manage shell scripts
>> and their execution via ssh. I'm sort of wondering what other kinds of
>> tools people are using for adding a job and configuration control overlay to
>> shell scripts. And what about orchestration tools that that take a "mashup
>> maker" approach? Pointers, thoughts, ....
> Are you looking for a script organization tool, or a one-to-many
> execution tool?
(realized I just jumped past this, right to your suggestion that I look
at Rakefiles)

Both, but more focused on organization. An ideal tool would:
- provide for editing/managing/versioning scripts (script = anything
that can be invoked at the command line)
- a library of control functions for use within scripts
- invoking scripts, combinations of scripts, pipelines of scripts (in
the Unix sense of pipes) - locally, remotely, across multiple machines
- remote script execution via ssh, rather than some kind of agent
- providing a simple database for keeping track of variables used by
scripts (e.g., IP addresses, DNS records for use by a provisioning
script) - that can be accessed from scripts
- accessing the above via cli, RESTful API, GUI
- cross-platform
- (nice-to-have) minimal environmental requirements (i.e., a step above
the unix shell, say the gnu buildtool suite rather than the JVM + a mass
of libraries, or the ruby ecosystem -- a self-configuring basic
environment would be nice, like perl+cpan)

I'm sure we've each cobbled something together along these lines - be it
through manual processes, standard installs, or something fancier. I'm
kind of wondering what different people have done, what pieces fit
together better than others, what's available that's more integrated but
that doesn't create a monolithic environment.

Rundeck seems to do most (all?) of this, running on Java. It looks
nice, but it's also really new.

tcl/tk provides a lot of this - but without the organizing functions
(tcl/tk + git, or maybe a tcl/tk IDE perhaps?)

I'm basically wondering what else might be out there - perhaps less
visible - newer than tcl/tk, more mature than Rundeck perhaps.
Essentially doing some research before provisioning several new machines
and a bunch of VMs (easy enough to do from manual checklists, but I'm
setting the stage for some future scaling).

Miles

AJ Christensen

unread,
Mar 18, 2012, 4:29:39 PM3/18/12
to dev...@googlegroups.com

The clojure 'pallet' tool (cm) has some of these principles.

I've seen sloppy admins compose and reuse Bash snippets with Chef all the time. Easy trap for beginners.

--AJ

Miles Fidelman

unread,
Mar 18, 2012, 4:44:57 PM3/18/12
to dev...@googlegroups.com
AJ Christensen wrote:
>
> Is this a joke thread? Still not quite gotten the hang of American
> sarcasm.
>

Why would you think that?

and later, AJ Christensen wrote:

> The clojure 'pallet' tool (cm) has some of these principles.
>

seems interesting, if a little java and cloud centric

> I've seen sloppy admins compose and reuse Bash snippets with Chef all
> the time. Easy trap for beginners.
>

Please elaborate --- are you saying that composing bash snippets is a
trap, or using Chef to do it is a trap? (If the former, who are you
calling a beginner? If the latter, I agree - not what Chef is built
for, and not the tool I'd use for managing shell scripts - hence my query.)

Scott Likens

unread,
Mar 18, 2012, 7:15:05 PM3/18/12
to dev...@googlegroups.com
On Sunday, March 18, 2012 7:03:52 AM UTC-7, Miles Fidelman wrote:
Over the years, I've built up a lot of shell scripts for provisioning
and configuration - which makes converting to something like chef or
puppet a somewhat daunting task (or even something like FAI).

I'm also a big fan of composing tool chains from piece parts - scripts
linked by pipes, mashups of functions accessed via RESTful APIs, etc.

Rundeck looks pretty interesting - given that it can manage shell
scripts and their execution via ssh.  I'm sort of wondering what other
kinds of tools people are using for adding a job and configuration
control overlay to shell scripts.  And what about orchestration tools
that that take a "mashup maker" approach?  Pointers, thoughts, ....

Thanks,


If your looking for a middle-ground that you can force as you will  you might find SM appealing https://bdsm.beginrescueend.com/

Miles Fidelman

unread,
Mar 18, 2012, 7:27:15 PM3/18/12
to dev...@googlegroups.com
Scott Likens wrote:
>
>
> If your looking for a middle-ground that you can force as you will
> you might find SM appealing https://bdsm.beginrescueend.com/

Two people have pointed at this, now, and the description sure sounds
interesting. Anybody know how active the project is - there doesn't
seem to be a mailing list or FAQ, and it seems to have one primary
developer.

Scott Likens

unread,
Mar 18, 2012, 8:22:25 PM3/18/12
to dev...@googlegroups.com
Miles,

It has 2 or 3 active developers to the best of my knowledge.  As far as the mailing list... See This response

I really can't speak for how active it is or anything less as I don't talk to Wayne.

Paul Graydon

unread,
Mar 18, 2012, 8:25:59 PM3/18/12
to dev...@googlegroups.com
On 3/18/2012 4:03 AM, Miles Fidelman wrote:
Over the years, I've built up a lot of shell scripts for provisioning and configuration - which makes converting to something like chef or puppet a somewhat daunting task (or even something like FAI).

Excuse me, I'm going to go for the utterly blunt response:

So what..?  I know as a sysadmin we try our best to be "appropriately lazy", but that means evaluating the benefits too compared to the work put in.  The positives of a config management system outweigh the negatives by a big enough margin to make any desire not to implement it border on pure laziness, not appropriate laziness.  Just being daunted by it is not a good enough excuse.

Yes, it's a pain in the ass trying to move from what is for you a smooth working set of scripts, but you gain a system that works easily, and that's going to be both understandable and enhanceable by anyone else that comes along after you (or should you be 'run over by a bus'.)*

No one is requiring you to convert the entire thing on one go.  I've a long way to go with the infrastructure I've inherited (an absolute mess of scripts and inconsistencies), but I've already got implemented a recipes in Chef that handle all the baseline common stuff, e.g. applications that should be on all (ntp etc), their configuration and so on, and am slowly adding more complex aspects of it.  Yes it's a pain in the arse, but it's important.  Just like all those niggling security requirements are a pain in the arse but you wouldn't be seen dead not implementing them.

Paul

* Every time I've gone into a sysadmin job, or picked up consulting work, I've found I've inherited a bunch of unusual bash and perl scripts, undocumented, inconsistent and bizarre in methodology (ever notice how no two sysadmins write perl / bash scripts alike?).  It means it takes a stupid amount of time to piece things together and understand quite how they interact.  Every time something breaks or goes wrong I'm left playing guessing games and tracking down whether it's a fringe case from some obscure aspect of a script or quite what (side note, it's amazing just how many problems I fix by enabling 'use strict; use warnings' and clearing up errors reported).

Miles Fidelman

unread,
Mar 18, 2012, 9:01:37 PM3/18/12
to dev...@googlegroups.com
Scott Likens wrote:
> Miles,
>
> It has 2 or 3 active developers to the best of my knowledge. As far
> as the mailing list... See This response
> <https://bdsm.beginrescueend.com/development/list>
>
>

You mean this one?

"None yet, stay tuned I might setup a librelist. "


That's what makes me wonder.

Miles

Chad Woolley

unread,
Mar 18, 2012, 9:10:22 PM3/18/12
to dev...@googlegroups.com
On Sun, Mar 18, 2012 at 6:01 PM, Miles Fidelman
<mfid...@meetinghouse.net> wrote:
> Scott Likens wrote:
>>
>> Miles,
>>
>> It has 2 or 3 active developers to the best of my knowledge.  As far as
>> the mailing list... See This response
>> <https://bdsm.beginrescueend.com/development/list>
>
> You mean this one?
>
> "None yet, stay tuned I might setup a librelist. "
>
>
> That's what makes me wonder.
>
> Miles

It's active on GitHub, and you can submit issues and pull requests
there (they accepted one 12 days ago):

https://bdsm.beginrescueend.com/development/github/

I do know Wayne, and he's usually very responsive to input, but I
think he prefers IRC, which is probably why he hasn't set up a mailing
list for this yet (they suck up valuable coding time, right?). Dunno
how far-baked BDSM is or if you should bet the farm on it, but Wayne
wrote RVM, which is a widely used project in the Ruby community
(written in shell script), and has actively maintained that for years
now.

If the lack of a mailing list really bothers you, open a github issue
asking for one ;)

-- Chad

Miles Fidelman

unread,
Mar 18, 2012, 9:11:19 PM3/18/12
to dev...@googlegroups.com
Paul Graydon wrote:
> On 3/18/2012 4:03 AM, Miles Fidelman wrote:
>> Over the years, I've built up a lot of shell scripts for provisioning
>> and configuration - which makes converting to something like chef or
>> puppet a somewhat daunting task (or even something like FAI).
>
> Excuse me, I'm going to go for the utterly blunt response:
>
> So what..? I know as a sysadmin we try our best to be "appropriately
> lazy", but that means evaluating the benefits too compared to the work
> put in. The positives of a config management system outweigh the
> negatives by a big enough margin to make any desire not to implement
> it border on pure laziness, not appropriate laziness. Just being
> daunted by it is not a good enough excuse.
>
> Yes, it's a pain in the ass trying to move from what is for you a
> smooth working set of scripts, but you gain a system that works
> easily, /and /that's going to be both understandable and enhanceable
> by anyone else that comes along after you (or should you be 'run over
> by a bus'.)*

Pardon me for being blunt as well, then. I don't see a particularly
high return on investment for investing time in converting to a tool (be
it puppet, chef, cfengine, or what have you) that addresses only a small
piece of an overall operations toolchain. I'd rather invest time and
effort in getting my existing toolchain more automated, documented, and
under configuration control.

>
> * Every time I've gone into a sysadmin job, or picked up consulting
> work, I've found I've inherited a bunch of unusual bash and perl
> scripts, undocumented, inconsistent and bizarre in methodology (ever
> notice how no two sysadmins write perl / bash scripts alike?). It
> means it takes a stupid amount of time to piece things together and
> understand quite how they interact. Every time something breaks or
> goes wrong I'm left playing guessing games and tracking down whether
> it's a fringe case from some obscure aspect of a script or quite what
> (side note, it's amazing just how many problems I fix by enabling 'use
> strict; use warnings' and clearing up errors reported).

You ever notice that no two systems are alike?

Miles Fidelman

unread,
Mar 18, 2012, 9:30:45 PM3/18/12
to dev...@googlegroups.com
I just sent Wayne an email. Main thing about presence/absence of a list
is as a way to build a user/developer community around a project.

Sean OMeara

unread,
Mar 18, 2012, 9:39:18 PM3/18/12
to dev...@googlegroups.com, dev...@googlegroups.com
Congratulations! You've broken the troll barrier!

Try actually building your next project in a proper CM tool and come back for your prize!

-s

John Vincent

unread,
Mar 18, 2012, 9:58:56 PM3/18/12
to dev...@googlegroups.com
On Sun, Mar 18, 2012 at 9:11 PM, Miles Fidelman
<mfid...@meetinghouse.net> wrote:
> Pardon me for being blunt as well, then.  I don't see a particularly high
> return on investment for investing time in converting to a tool (be it
> puppet, chef, cfengine, or what have you) that addresses only a small piece
> of an overall operations toolchain.  I'd rather invest time and effort in
> getting my existing toolchain more automated, documented, and under
> configuration control.
>
>
The tool you are looking for does not and probably will never exist.
You said yourself very early on that you like the unix way. CM is a
smaller part of a bigger picture and the tools that you specifically
mentioned address configuration management solely right now. Some of
them have compliments and more tightly integrated add-on (i.e.
mcollective) but you'll find that most of the tools around address one
specific problem and do it very well.

> You ever notice that no two systems are alike?

Congrats. I figured it would get to this point. You've just described
why you need proper configuration management:

Puppet -

package { "screen": ensure => "installed" }

Chef -

package "screen" do
action :install
end

Guess what. That works the same across all distros. Doesn't matter if
those systems are alike. But here's the real crux of the matter: the
point of configuration management is not to make your special
snowflake systems behave the same. It's to make sure you don't have
special snowflake systems. Systems are cattle...pets. They aren't
artisan crafted, flame broiled, freshly brewed works of art.

I'm very sorry if you haven't gotten a very good welcome to the list.
However, while we may not have a good definition of what devops *IS*,
most people here feel that encouraging a pattern of brittle,
busted-ass inflexible and ultimately unscalable shell scripts is wrong
and downright irresponsible.

Even more telling is the fact that the questions you're asking are the
first sign that you've grown beyond those shell scripts.

SM (nee BDSM) is probably up your alley. Wayne is a solid guy and he
wrote SM mainly out of frustration with a poorly implemented Chef
setup. While I wouldn't suggest it for the reasons I listed above,
Wayne takes his community and his code seriously. Kudos if SM gets you
further along your journey but at some point, you are (and I would
argue already have) reached the limits of your shell addiction.


--
John E. Vincent
http://about.me/lusis

Chad Woolley

unread,
Mar 18, 2012, 10:47:12 PM3/18/12
to dev...@googlegroups.com
On Sun, Mar 18, 2012 at 6:39 PM, Sean OMeara <som...@gmail.com> wrote:
> Congratulations! You've broken the troll barrier!
>
> Try actually building your next project in a proper CM tool and come back for your prize!

No idea what you mean, but I'm pretty sure it's unnecessarily snarky.
Git/GitHub is a proper CM tool in my book.

Andrew Clay Shafer

unread,
Mar 18, 2012, 10:47:45 PM3/18/12
to devops

Chad Woolley

unread,
Mar 18, 2012, 11:03:34 PM3/18/12
to dev...@googlegroups.com
On Sun, Mar 18, 2012 at 7:47 PM, Andrew Clay Shafer
<andrew...@gmail.com> wrote:
>
> http://www.youtube.com/watch?v=1sONfxPCTU0

I love you, man...

John Vincent

unread,
Mar 18, 2012, 11:06:34 PM3/18/12
to dev...@googlegroups.com

Andrew Clay Shafer

unread,
Mar 18, 2012, 11:20:34 PM3/18/12
to devops

You can either consistently and predictably manage systems at scale or
you can't.

Theoretically, there are better bash solutions than some of the puppet/
chef implementations I have seen.

Sure, there are benefits to these frameworks, but if you don't
understand the foundation they are built on, then they just look like
a lot more problem than they are worth.

Often, telling someone a truth they aren't ready to hear is
indistinguishable from lying to them.

Also, for the record, Configuration Management in nearly all 'devops'
conversations is going to refer to tools for configuring servers and
not version control systems. There is a bit of a namespace collision
there.

What we really need are thunderdome style system management battles.

Miles Fidelman

unread,
Mar 18, 2012, 11:22:24 PM3/18/12
to dev...@googlegroups.com

Chad, you beat me to it :-)

Miles Fidelman

unread,
Mar 18, 2012, 11:29:40 PM3/18/12
to dev...@googlegroups.com
John Vincent wrote:
> On Sun, Mar 18, 2012 at 9:11 PM, Miles Fidelman
> <mfid...@meetinghouse.net> wrote:
>> Pardon me for being blunt as well, then. I don't see a particularly high
>> return on investment for investing time in converting to a tool (be it
>> puppet, chef, cfengine, or what have you) that addresses only a small piece
>> of an overall operations toolchain. I'd rather invest time and effort in
>> getting my existing toolchain more automated, documented, and under
>> configuration control.
>>
>>
> The tool you are looking for does not and probably will never exist.
> You said yourself very early on that you like the unix way. CM is a
> smaller part of a bigger picture and the tools that you specifically
> mentioned address configuration management solely right now. Some of
> them have compliments and more tightly integrated add-on (i.e.
> mcollective) but you'll find that most of the tools around address one
> specific problem and do it very well.

Seems to me that rundeck and rerun, and possibly bdsm come pretty
close. Tcl/tk + git as well.

Tcl has been around forever. Rundeck I knew about, rerun and bdsm I
found out in responses to my query.

>> You ever notice that no two systems are alike?
> Congrats. I figured it would get to this point. You've just described
> why you need proper configuration management:
>
> Puppet -
>
> package { "screen": ensure => "installed" }
>
> Chef -
>
> package "screen" do
> action :install
> end
>

No. Configuration is a fairly evolve discipline for defining,
controlling, accounting for, and auditing system configurations.

CVS, Git, Subversion are configuration control systems.

Puppet and Chef are engines for applying configurations - with arguably
cleaner syntax than pure bash scripts.

> I'm very sorry if you haven't gotten a very good welcome to the list.
> However, while we may not have a good definition of what devops *IS*,
> most people here feel that encouraging a pattern of brittle,
> busted-ass inflexible and ultimately unscalable shell scripts is wrong
> and downright irresponsible.

--

Miles Fidelman

unread,
Mar 18, 2012, 11:56:01 PM3/18/12
to dev...@googlegroups.com

For the record, people have been consistently and predictably managing
systems, at very large scale, for a really long time (at least by the
standards of "internet time"). People have been deploying distribution
systems (e.g., electric distribution grids), telecom. networks, large
distributed systems (such as those found in defense environments),
financial trading systems, etc., etc., etc. for decades. (By the way,
my background is large comms. networks, largely supporting distributed
command and control systems.)

I think its Alex Honor who's powerpoint decks that lead with a slide
along the lines of "people -> process -> tools" -- the point being that
tools have to serve the organization and process, not the other way around.

If the goal is to deploy 10,000 hadoop instances, quickly - sure chef is
going to do a great job. If you're managing something more complicated,
process discipline is what counts - you can manage huge huge and
complicated operations with little more than checklists and
whiteboards. The worst thing you want to do is have your tools impose
inappropriate disciplines in environments where operating procedures
have been evolved and tested over time.

John Vincent

unread,
Mar 19, 2012, 12:08:53 AM3/19/12
to dev...@googlegroups.com
On Sun, Mar 18, 2012 at 11:29 PM, Miles Fidelman

<mfid...@meetinghouse.net> wrote:
> John Vincent wrote:
>>
>> On Sun, Mar 18, 2012 at 9:11 PM, Miles Fidelman
>> <mfid...@meetinghouse.net>  wrote:
>>>
>>> Pardon me for being blunt as well, then.  I don't see a particularly high
>>> return on investment for investing time in converting to a tool (be it
>>> puppet, chef, cfengine, or what have you) that addresses only a small
>>> piece
>>> of an overall operations toolchain.  I'd rather invest time and effort in
>>> getting my existing toolchain more automated, documented, and under
>>> configuration control.
>>>
>>>
>> The tool you are looking for does not and probably will never exist.
>> You said yourself very early on that you like the unix way. CM is a
>> smaller part of a bigger picture and the tools that you specifically
>> mentioned address configuration management solely right now. Some of
>> them have compliments and more tightly integrated add-on (i.e.
>> mcollective) but you'll find that most of the tools around address one
>> specific problem and do it very well.
>
>
> Seems to me that rundeck and rerun, and possibly bdsm come pretty close.
>  Tcl/tk + git as well.
>
> Tcl has been around forever.  Rundeck I knew about, rerun and bdsm I found
> out in responses to my query.
>
>

I'm confused then. If you're talking about Tcl, why are we still
discussing bash scripts? I don't really see where Tk comes into play.
If you need to slap a gui on something, you'd be better served not
burdening your fellow sysadmins and just sticking it behind a web
interface. I guess AOLServer is still a thing?

>>> You ever notice that no two systems are alike?
>>
>> Congrats. I figured it would get to this point. You've just described
>> why you need proper configuration management:
>>
>> Puppet -
>>
>> package { "screen": ensure =>  "installed" }
>>
>> Chef -
>>
>> package "screen" do
>>   action :install
>> end
>>
> No.  Configuration is a fairly evolve discipline for defining, controlling,
> accounting for, and auditing system configurations.
>
> CVS, Git, Subversion are configuration control systems.
>
> Puppet and Chef are engines for applying configurations - with arguably
> cleaner syntax than pure bash scripts.
>
>

Andrew has already addressed this but we seem to have a semantic
confusion here. As he said, in "devops" discussions CM pretty much
always refers to a tool akin to puppet, chef or cfengine. Git, SVN,
CVS, Hg, Bazaar I have NEVER heard referred to as a "configuration
control system". Version control system (VCS) is the term I've always
heard.

Having said that:

"> No.  Configuration is a fairly evolve discipline for defining, controlling,
> accounting for, and auditing system configurations."

If that's your definition then pretty much every tool discussed up to
this point accomplishes that task 100%. The nice thing about a CM (my
definition) tool is that's what its DESIGNED to do. I can tell you
that any system, regardless of being managed with Puppet or Chef (I
haven't done cfengine stuff in at least 8 years) I could meet all of
those requirements with my chef or puppet repo. So color me confused
as to what exactly these tools seem to be missing that DOESN'T meet
that need.

Christopher Webber

unread,
Mar 19, 2012, 12:49:59 AM3/19/12
to dev...@googlegroups.com

Puppet and Chef are engines for applying configurations - with arguably cleaner syntax than pure bash scripts.


I just want to point this out because it drives me insane... Puppet and Chef don't apply configuration, per say, and if you think about it that way you will never use them effectively. They manage and document the state of a system. You are not managing the apache config. You are documenting that the system should have apache installed, the config should look like X and that apache is running and then applying that documentation to the system. And, because it is code, we can put it in a VCS. 

-- cwebber

Miles Fidelman

unread,
Mar 19, 2012, 12:59:27 AM3/19/12
to dev...@googlegroups.com
John Vincent wrote:
> On Sun, Mar 18, 2012 at 11:29 PM, Miles Fidelman
> <mfid...@meetinghouse.net> wrote:
>> John Vincent wrote:
>>> The tool you are looking for does not and probably will never exist.
>>> You said yourself very early on that you like the unix way. CM is a
>>> smaller part of a bigger picture and the tools that you specifically
>>> mentioned address configuration management solely right now. Some of
>>> them have compliments and more tightly integrated add-on (i.e.
>>> mcollective) but you'll find that most of the tools around address one
>>> specific problem and do it very well.
>>
>> Seems to me that rundeck and rerun, and possibly bdsm come pretty close.
>> Tcl/tk + git as well.
>>
>> Tcl has been around forever. Rundeck I knew about, rerun and bdsm I found
>> out in responses to my query.
>>
>>
> I'm confused then. If you're talking about Tcl, why are we still
> discussing bash scripts? I don't really see where Tk comes into play.
> If you need to slap a gui on something, you'd be better served not
> burdening your fellow sysadmins and just sticking it behind a web
> interface. I guess AOLServer is still a thing?

Tcl is pretty good at composing command line executables - particularly
when you use expect. (tcl, after all, stands for "tool control
language") Tk lets you put a gui on it if you wish, but that's really
extraneous.

if you think that puppet or chef can support the formal configuration
control requirements of a large distributed system, including hardware
components, software components, configuration files, network devices
and configuration, crypto configuration, ..... - you just haven't dealt
with serious systems -- for that matter, when it comes to definitions,
the best taxonomy of devops that I've seen (of the few that seem to be
around) is the one at
http://code.google.com/p/devops-toolchain/wiki/ArchitectureComponents -
of which puppet/chef addresses only one small piece of an overall tool
chain - NOT including inventory (a key piece of any reasonable
configuration management process)

not to imply that they're not useful tools - but they certainly are only
one piece of provisioning, configuring, monitoring and controlling systems

in any case, my original query was specifically about tools for managing
and composing shell scripts - NOT a philosophical discussion of
competing operations styles and philosophies

if you want some elaboration of what I'm looking for, is a tool that,
ideally, would:


- provide for editing/managing/versioning scripts (script = anything
that can be invoked at the command line)
- a library of control functions for use within scripts
- invoking scripts, combinations of scripts, pipelines of scripts (in
the Unix sense of pipes)
- locally, remotely, across multiple machines
- remote script execution via ssh, rather than some kind of agent
- providing a simple database for keeping track of variables used by
scripts (e.g., IP addresses, DNS records for use by a provisioning
script) - that can be accessed from scripts
- accessing the above via cli, RESTful API, GUI
- cross-platform
- (nice-to-have) minimal environmental requirements (i.e., a step above
the unix shell, say the gnu buildtool suite rather than the JVM + a mass
of libraries, or the ruby ecosystem -- a self-configuring basic
environment would be nice, like perl+cpan)

- tcl has been used this way for a long time, but has its limits (on the
other hand, it has some nice features for interacting with databases -
which makes it useful for, say, linking a virtual-machine provisioning
script to a database of IP address assignments)
- rundeck and rerun are relatively new and get very close
- bdsm also seems to be in the same ballpark

tcl and rundeck I knew about - I have to thank this list and the
devops-toolchain list for pointers to rerun and bdsm, and some
discussion thereof

Miles Fidelman

unread,
Mar 19, 2012, 1:02:22 AM3/19/12
to dev...@googlegroups.com

Ok, if you want to get pedantic about it - they are tools that "coerce"
a system to a specific state (using the technical definition of the word).

John Vincent

unread,
Mar 19, 2012, 1:10:44 AM3/19/12
to dev...@googlegroups.com