Test driven sysadmin

[Stephen Nelson-Smith]

So Julian and I were talking at LRUG this week, and decided that we
felt cucumber is becoming, or indeed will become, or perhaps already
is the tool of choice to implement test or behaviour driven system
administration.

Julian tweeted it, and shortly afterwards I was challenged by an old
colleague of mine:

@LordCope This sysadmin testing thing - I'm not really sure I get it.
Certainly not the cucumber bit; who needs to read the cuc scenarios?

Well Twitter's not really the best vehicle for a complex conversation,
but I made a stab at a quick summary:

#cucumber-nagios is ideal for capturing nonfunctional requirements.
Define system behaviour; Iterate until green.

On my return I notice the following response:

What's a nonfunctional requirement? And why is cucumber better than
well written code? The extra layer of indirection is a smell.

I thought I'd throw the subject open to discussion - we're going to
meet opposition as we try to move these kinds of ideas forward, so
it's probably worth having a good think about it.

S.

[Chad Woolley]

On Sat, Nov 14, 2009 at 10:03 AM, Stephen Nelson-Smith
<atalanta...@googlemail.com> wrote:
> #cucumber-nagios is ideal for capturing nonfunctional requirements.
> Define system behaviour; Iterate until green.
>

> I thought I'd throw the subject open to discussion - we're going to
> meet opposition as we try to move these kinds of ideas forward, so
> it's probably worth having a good think about it.

Can you provide links to examples, articles, or presentations?

[Matthias Marschall]

> Certainly not the cucumber bit; who needs to read the cuc scenarios?

> And why is cucumber better than
> well written code? The extra layer of indirection is a smell.

It really depends on who needs to read the tests. If they're only read
by sysadmins and developers, the extra layer is indeed not necessary.
If non technical guys need to read and understand the tests, you need
cucumber to make it possible for them to understand them.

In my team we decided to drop the cucumber indirection in favor of
(similarly structured) rspecs as only developers read the stuff. But
if we would have included the gold owners, we definitely would have
stuck to cucumber.

You can read the full story here: http://www.agileweboperations.com/webrat-automated-acceptance-testing-with-rspec-or-cucumber/

Best,
Matthias

[Steve Conover]

Agreed about the unnecessary layer of indirection. If you audience is
ops and dev then test-driven, well-written bash/perl/ruby code is
ideal for checks IMO.

> --
>
> You received this message because you are subscribed to the Google Groups "Agile System Administration" group.
> To post to this group, send email to agile-system-...@googlegroups.com.
> To unsubscribe from this group, send email to agile-system-admini...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/agile-system-administration?hl=.
>
>
>

[Lindsay Holmwood]

Don't disagree about indirection if your audience is technical,
however there are a few things Cucumber gives technical teams which
can make it useful:

1. Low barrier of entry for people unfamiliar with your setup.

Being able to sit a new team member down and show them the high level
functional or non-functional requirements of your environment without
them understanding the programming language your checks are written in
can be extremely useful. Your checks are essentially self-documenting.

2. Code reuse.

By writing Cucumber features, you can very easily reuse the same steps
across multiple scenarios. So much time is wasted writing the same
checks again and again and again.

In the software development world, we refactor common things out into
libraries. In the sysadmin world we're very happy to keep plugging
away, making the same mistakes, creating the same maintenance
headaches for ourselves.

Being able to utilise a common library of checks reduces the amount of
code you have to maintain, and thus there will be less bugs (and less
monitoring alerts at 3am).

3. (really 2.5) That "well-written bash/perl/ruby code" is already
done for you.

People will have different styles of coding. With cucumber-nagios at
least, i'm exceptionally picky about the style of code that is
released. All code in cucumber-nagios should obvious and easy to
understand to someone with a smattering of Ruby knowledge, and if it
isn't, I consider it a bug.

I aim to provide a base library of well-tested steps that will do 90%
of what you need in your checks, so the amount of code you have to
write (and maintain) is kept to a minimum.

Hope that's clear!

Lindsay

2009/11/14 Steve Conover <scon...@gmail.com>:

--
http://holmwood.id.au/~lindsay/ (me)

[Steve Conover]

- I'm wondering who this new team member is...are you imagining that
this person is not either ops or dev?

- Regarding teaching your sysads to refactor and reuse - you could
just do that directly. "People, we're going to build a library. And
write tests.". One could imagine picking bash or ruby and building up
a library of unit-tested nagios check support.

In addition, I kind of see cucumber as an unnecessary level of
abstraction away from where you want to be. I don't doubt that it
could be made to work by someone determined enough. But if you're
going to check return codes from commands, and mess with /proc or kill
-3 or curl localhost, why not use something like bash/perl/ruby (some
combo, even)? That environment seems to me to be most appropriate to
the "domain" - making assertions about the state of a system - where
by contrast cucumber (I would argue) is most appropriate for
expressing acceptance tests for a web application that someone less
technical might want to read.

I'm guessing we simply disagree about that last part. It's an
interesting concept nonetheless.

-Steve

[Lindsay Holmwood]

2009/11/15 Steve Conover <scon...@gmail.com>:

> - I'm wondering who this new team member is...are you imagining that
> this person is not either ops or dev?

They could be dev, ops, a new hire, someone from the business - it
doesn't really matter.

The self documenting nature of the Cucumber steps makes it immediately
obvious what the check is doing, without prior knowledge of the
programming language. Technical users (dev or ops) can drill down
further into the implementation of the check, and people from the
business can be content with a high-level understanding.

>
> - Regarding teaching your sysads to refactor and reuse - you could
> just do that directly.  "People, we're going to build a library.  And
> write tests.".  One could imagine picking bash or ruby and building up
> a library of unit-tested nagios check support.

There's a big difference between encouraging a culture of code sharing
and reuse, and using a tool that actively promotes it. With Cucumber
you really have no choice but to write reusable steps, whereas it
takes a lot of discipline to do it by yourself, and even more across a
team.

If you have high quality sysadmins who are also knowledgeable and
disciplined developers on your team you can probably pull it off, but
a lot of us aren't that blessed.

I hold no illusions that cucumber-nagios is the only way of doing
this. If you manage to build or find another tool that promotes that
disciplined workflow without the use of the Cucumber language, that
would be awesome too.

>
> In addition, I kind of see cucumber as an unnecessary level of
> abstraction away from where you want to be.  I don't doubt that it
> could be made to work by someone determined enough.  But if you're
> going to check return codes from commands, and mess with /proc or kill
> -3 or curl localhost, why not use something like bash/perl/ruby (some
> combo, even)?  That environment seems to me to be most appropriate to
> the "domain" - making assertions about the state of a system - where
> by contrast cucumber (I would argue) is most appropriate for
> expressing acceptance tests for a web application that someone less
> technical might want to read.

To your first point, you are just writing Ruby. Just think of a step
as a Ruby method or a Python function, with Cucumber as pattern
matching thrown on top.

You're not writing your checks in some crazy meta-language - you're
constructing checks by stringing Ruby methods together.

Secondly, there's no denying Cucumber is a great fit for acceptance
testing of web applications, but I think the Gherkin language (what
you're writing your Cucumber features/scenarios in) has suitably
evolved to be useful in a lot of other situations (monitoring checks
come to mind :-).

>
> I'm guessing we simply disagree about that last part.  It's an
> interesting concept nonetheless.

Cucumber is definitely *not* a silver bullet. There are plenty of
reasons why you wouldn't use it, which you have enumerated well.

Talking about it teases out the pros and cons of both approaches,
which is a good thing.

Lindsay

[Chad Woolley]

On Sun, Nov 15, 2009 at 12:58 PM, Lindsay Holmwood <aux...@gmail.com> wrote:
> The self documenting nature of the Cucumber steps makes it immediately
> obvious what the check is doing, without prior knowledge of the
> programming language. Technical users (dev or ops) can drill down
> further into the implementation of the check, and people from the
> business can be content with a high-level understanding.
>

> There's a big difference between encouraging a culture of code sharing
> and reuse, and using a tool that actively promotes it. With Cucumber
> you really have no choice but to write reusable steps, whereas it
> takes a lot of discipline to do it by yourself, and even more across a
> team.
>
> If you have high quality sysadmins who are also knowledgeable and
> disciplined developers on your team you can probably pull it off, but
> a lot of us aren't that blessed.

I was on the "Cucumber is overkill" side, mainly because I don't have
experience with it, but these are compelling arguments to use it more
cases, or at least give it a fair shot.

Are there repositories of reusable cucumber tests, like Opscode is
doing for Chef? This would lower the barrier, which may be the root
of many 'overkill' arguments, as it was for me. A repository approach
seems especially suited for sysadmin-type tests, as opposed to
Cucumber tests for applications, which would rarely be reusable.

-- Chad

[Julian Simpson]

Hello,

On 16 Nov 2009, at 05:25, Chad Woolley <thewoo...@gmail.com> wrote:

> On Sun, Nov 15, 2009 at 12:58 PM, Lindsay Holmwood
> <aux...@gmail.com> wrote:
>> The self documenting nature of the Cucumber steps makes it
>> immediately
>> obvious what the check is doing, without prior knowledge of the
>> programming language. Technical users (dev or ops) can drill down
>> further into the implementation of the check, and people from the
>> business can be content with a high-level understanding.
>>
>> There's a big difference between encouraging a culture of code
>> sharing
>> and reuse, and using a tool that actively promotes it. With Cucumber
>> you really have no choice but to write reusable steps, whereas it
>> takes a lot of discipline to do it by yourself, and even more
>> across a
>> team.
>>
>> If you have high quality sysadmins who are also knowledgeable and
>> disciplined developers on your team you can probably pull it off, but
>> a lot of us aren't that blessed.
>
> I was on the "Cucumber is overkill" side, mainly because I don't have
> experience with it, but these are compelling arguments to use it more
> cases, or at least give it a fair shot.
>

I think the BDD style of testing makes it much easier to work out the
ontent in someone else's code. You can write old-school test/unit
style unit tests in a similar style, but I find that cucumber and
rspec make it easier.

YMMV however :)

Julian.

[John Arundel]

On Sun, Nov 15, 2009 at 7:58 PM, Lindsay Holmwood <aux...@gmail.com> wrote:
> The self documenting nature of the Cucumber steps makes it immediately
> obvious what the check is doing, without prior knowledge of the
> programming language. Technical users (dev or ops) can drill down
> further into the implementation of the check, and people from the
> business can be content with a high-level understanding.
>

> There's a big difference between encouraging a culture of code sharing
> and reuse, and using a tool that actively promotes it. With Cucumber
> you really have no choice but to write reusable steps, whereas it
> takes a lot of discipline to do it by yourself, and even more across a
> team.

When Stephen and I were working at Wordtracker, we started out writing
custom Nagios plugins in Ruby for monitoring various things, and they
gradually became more high-level, so that the plugin might do things
like:

1. Hit the URL for the Wordtracker keyword tool
2. Log in with monitoring credentials
3. Run a keyword search
4. Parse the results and check that the numbers are higher than a
certain 'sanity value'.

We refactored out the Nagios plugin framework into a library, and the
next step would have been the 'logging in to sites and clicking
buttons' type stuff, but at that point cucumber-nagios came along and
we didn't need to! Also, we could never have shared that code with
anyone outside of the organisation, because they would have needed our
various support libraries and to have the same Nagios setup. With the
cucumber-nagios tests, we could sit down and pair with developers to
write tests with a framework they already understand, to verify that
the app is behaving as designed.

This is similar to the way Puppet lets you replace bespoke,
site-specific automation tools with generic, high-level, shareable,
cross-platform code that can be published and that other people can
contribute to.

John
--
Bitfield Consulting: we make software that makes things work
http://bitfieldconsulting.com/

[Lindsay Holmwood]

2009/11/16 Chad Woolley <thewoo...@gmail.com>:

I've spoken with Adam Jacob of Opscode about pooling our efforts, and
he's recently submitted a bunch of additional steps extracted from
Chef. I'm going to be integrating these in the next day and push out
another release.

The idea is to build up a solid library of steps in cucumber-nagios,
and then spin them out into a separate repository/library/project that
other systems management applications can use.

Lindsay

--
http://holmwood.id.au/~lindsay/ (me)

[Patrick Debois]

Hi Lindsay , that's great news! Two great minds combining their forces!

In an effort to get the TDD, BDD terminology straight I made a blog post
http://www.jedi.be/blog/2009/11/16/test-driven-automation-and-administration/

Feel free to comment.

[Nikolay Sturm]

On Nov 15, 8:58 pm, Lindsay Holmwood <auxe...@gmail.com> wrote:
> The self documenting nature of the Cucumber steps makes it immediately
> obvious what the check is doing, without prior knowledge of the
> programming language. Technical users (dev or ops) can drill down
> further into the implementation of the check, and people from the
> business can be content with a high-level understanding.

(Let me first say, that I have zero experience with cucumber or BDD, I
am just
an interested observer.)

I beg to differ. Human language is certainly more understandable to
most people
than code, but you have to pay a price as well, it's inaccuracy. Take
this
snippet for example:

Scenario: Valid Apache config
Given a config file template apache2.conf in /etc/apache2
When I generate it
Then there should be a file called apache2.conf in etc/apache2
And it should be a valid apache file

What does "valid apache file" mean? Does it imply a test with the
apache2
that actually runs in your production environment? Of just the one on
the
local machine, where the test is run? Or what? If you don't look at
the step
definition and thus at the code, you have no idea what this means and
thus you are prone to misinterpretation.

No diss'ing intended, I just want to point out some pitfalls.

cheers,

Nikolay

[Lindsay Holmwood]

2009/11/16 Nikolay Sturm <goo...@erisiandiscord.de>:

Totally agree - that's pretty much my biggest problem with Cucumber.

Language can be subjective, and meaning isn't always deducible without
diving into the code. Does "valid" mean syntactically correct, or
behaving how I would expect?

For that particular step, I think "And it should be a syntactically
valid apache config" would be more accurate.

Maintaining clarity in the language of your steps is probably the
hardest thing when using Cucumber.