Behavior Driven Infrastructure

[Martin Englund]

Folks,

I've been struggling a bit with how we're using puppet (at my job):
how do you validate that puppet has done what it is supposed to, and
even troublesome, how you validate that it has done what you intended
it to do?

Since I'm the only one who is writing the puppet profiles and working
with it on a daily basis, I'm the only one who can decipher the puppet
logs. I often get the question: how do we know when the system is
ready for production?

I've been playing around[1] with cucumber & webrat, and have pieced
together a way to do behavior driven infrastructure testing. Puppet
takes care of getting the system configured correctly, but there are
often other pieces involved, like opening firewall ports, adding DNS
entries, sendmail routing, etc. Which must be done outside of puppet
to get the system ready for release.

When you write code, you always use unit testing & integration testing
to verify that the application is working as expected, but why don't
we use that when we install a system?

What are you using to verify that your system is correctly configured
and behaves the way you want?

[1] <http://blogs.sun.com/martin/entry/behavior_driven_infrastructure>

cheers,
/Martin

[Lindsay Holmwood]

2009/11/6 Martin Englund <martin....@sun.com>:

[although i've commented on the excellent blog entry, i'm posting here]

Hi Martin,
It looks like there's a bit of crossover here with a project i've been
working on the last few months called cucumber-nagios[0]. It takes the
result of a Cucumber run and outputs it in the Nagios plugin format.
Essentially you use it to express your intentions in plain language,
and verify your intentions periodically through your monitoring
system. Just like what you've posted about. :-)

Anyhow, I spoke about cucumber-nagios at the excellent Devopsdays in
Belgium last weekend, and I got talking with people about expanding
the library of steps to cover things like logins over SSH, file
manipulation, and mail delivery. It would be cool if we could
centralise our efforts and focus on building an awesome library of
reusable steps to test our infrastructure.

Your point about doing behaviour driven development when writing
software is right on the mark. From an infrastructure perspective, I
like to think of Cucumber as the testing tool, and Puppet as the
programming language.

Anyhow, i'd be interested to hear what other people think about this idea!

Cheers,
Lindsay

[0] http://auxesis.github.com/cucumber-nagios

--
http://holmwood.id.au/~lindsay/ (me)

[Martin Englund]

Hi Lindsay!

On Nov 6, 3:56 pm, Lindsay Holmwood <lind...@holmwood.id.au> wrote:

> It looks like there's a bit of crossover here with a project i've been
> working on the last few months called cucumber-nagios[0]. It takes the
> result of a Cucumber run and outputs it in the Nagios plugin format.
> Essentially you use it to express your intentions in plain language,
> and verify your intentions periodically through your monitoring
> system. Just like what you've posted about. :-)
>

It sure does look like there is an overlap :)

> Anyhow, I spoke about cucumber-nagios at the excellent Devopsdays in
> Belgium last weekend, and I got talking with people about expanding
> the library of steps to cover things like logins over SSH, file
> manipulation, and mail delivery. It would be cool if we could
> centralise our efforts and focus on building an awesome library of
> reusable steps to test our infrastructure.
>

It is indeed better to work on this jointly. You are much further
along the way, so I'll clone your cucumber-nagios repository and work
in integrating the stuff I've done into your code. You'll be hearing
from me shortly...

I'm also talking to Ohad about integrating this into the foreman[1].
Then it will become the IDE of system/infrastructure development :)

[1] <http://theforeman.org/>

cheers,
/Martin

[william pink]

I am certainly interested from a infrastructure point of view having like you say a library of
reusable steps which would fit the bill nicely, The ability to build with Puppet and check that build with Cucumber-Nagios would be awesome and more in line in which I would consider Test Driven Infrastructure.

Will

[Patrick Debois]

for what it's worth:

for a while I was experimenting with cucumber to test scripts running
on linux machines. And yes , IMHO it is the way to go .

It worked well with the behavior testing just a you guys described.
still just as in regular development, bdd can be complemented with
tdd.
The way i did the tdd, was by using the exit codes of the shell
commands. I created an abstraction in ruby for execute (local, remote)
and upload, download of files.

If the command did not have exit 0 (the default) then i imagined the
test being failed.
Using snapshots of the virtual machines, I ran the scripts against a
virtual machine (used virtualbox), with the ability to do a rollback.
I know this is not the way puppet normally operates, but I used the
(if i recall correctly) the puppetrun command to execute it and see
what happened
If all of these exit codes where ok (similar to unit tests), i ran the
bdd tests to see if everything was correct.

By integrating the tdd, and the bdd within a CI-system, I was able to
have it run continuously and rebuild the system over and over again.

Another thing I want to mention is that you can easily abstract
commands in a custom dsl, but I really like to see what actual
commands got executed.
This is slightly off topic maybe, but by logging/running the actual
commands and have the whole environment build like that (including
making a floppy, booting the vms)
it allowed me to create a kind of install document with the actual
commands instead of pointing to a newly invented DSL.
So I say yes to an abstraction layer, but no to hiding the actual
commands.

just my 2 c

On Nov 6, 3:56 pm, Lindsay Holmwood <lind...@holmwood.id.au> wrote:

> 2009/11/6 Martin Englund <martin.engl...@sun.com>:

[Isaac Christoffersen]

I'd be interested in hearing more about this. I've been working on a
project in which we do a lot of automated server builds and
configuration using various OSS tools, including puppet. However, one
thing that I've been looking for is a way to write a set of test suites
to validate the server configuration. The combination of TDD and BDD
with server rollouts is very attractive.

I'd also be curious to know what CI server people are using to help
coordinate these intermediate builds and tests.

>>> I've been playing around[1] with cucumber& webrat, and have pieced

>>> together a way to do behavior driven infrastructure testing. Puppet
>>> takes care of getting the system configured correctly, but there are
>>> often other pieces involved, like opening firewall ports, adding DNS
>>> entries, sendmail routing, etc. Which must be done outside of puppet
>>> to get the system ready for release.
>>>
>>

>>> When you write code, you always use unit testing& integration testing

[Julian Simpson]

I use Hudson.

I've used Test::Unit to wrap validations of generated config files in
the past. I want to know as early as possible if something isn't
going to work.

Functionally testing the config from the outside in might be easy but
slow. You can deploy to a VM and actually test it using the goodness
of cucumber. I'm still looking for the equivalent of unit tests: how
do you quickly test function from the inside to give enough confidence
and a fast feedback loop? Maybe we need to build out Patrick's
approach with a library of command-line tools to verify that
everything works.

J.


2009/11/7 Isaac Christoffersen <isaac.chri...@gmail.com>:

--
Julian Simpson
Software Build and Deployment
http://www.build-doctor.com

[Isaac Christoffersen]

I've been thinking about using Hudson for this, so its good to know that
there are others out there doing this.

3 projects that I've bookmarked to look at for unit testing frameworks
for the OS are:

https://fedoraproject.org/wiki/AutoQA
http://autotest.kernel.org/
http://www.linux-kvm.org/page/Main_Page

I haven't had time to do much more than scan the web site, but they seem
to be worth watching.

I've also started including test.pp scripts as part of each module.
These scripts check directories or do some execs (i.e. wgets, pings,
lookups). These scripts then run with every puppet update push.
However, this isn't ideal for production and there needs to be a better
way to trigger the running of unit tests. But it has been helpful in
validating iptables configuration and ensuring services and their
clients are on the same network.

[Christian Kauhaus]

Isaac Christoffersen schrieb:

> to validate the server configuration. The combination of TDD and BDD
> with server rollouts is very attractive.

I think the whole topic is really hot. :-)

It would probably a good idea to expand what the difference between TDD
and BDD in systems management actually is about.

Not everyone here on the list is fine with the subtle differences in
terminology (for example, me ;-) )

Regards

Christian

--
Dipl.-Inf. Christian Kauhaus <>< · k...@gocept.com · systems administration
gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany
http://gocept.com · tel +49 345 1229889 11 · fax +49 345 1229889 1
Zope and Plone consulting and development

[Julian Simpson]

My understanding is that Test Driven Development hit the mainstream
(did smalltalk have a unit test framework?) with JUnit.[1] Junit
allowed developers to automate their unit tests. All the unit tests
for your project should run in seconds, if your codebase and tests are
loosely coupled.

e.g. testThatTheFooObjectReturnsABarWhenPassedABaz

Those TDD style tests can be pretty ugly. In theory you can use them
as a spec to describe what code does, but in practice, no. So a guy
called Dan North coined the term Behaviour Driven Development, and
focussed on the language used to help Business Analysts or other guys
in suits to specify. This is where the Given/When/Then syntax comes
from.

He worked on JBehave as an alternative to JUnit so you can specify
behavior. But Rspec and Cucumber are where the notion has taken off.
I'd write more but I have to make my kids' dinner :)

J.


[1] http://www.junit.org/

2009/11/9 Christian Kauhaus <k...@gocept.com>:

[Lindsay Holmwood]

2009/11/6 Martin Englund <martin....@sun.com>:

>>
> It is indeed better to work on this jointly. You are much further
> along the way, so I'll clone your cucumber-nagios repository and work
> in integrating the stuff I've done into your code. You'll be hearing
> from me shortly...

So i've added some SSH steps to cucumber-nagios, along the lines of
what Martin blogged about a few days ago.

I've published a blog post about the new feature, and how this could
potentially be used withina Behaviour Driven Infrastructure
techniques:

http://holmwood.id.au/~lindsay/2009/11/09/behaviour-driven-infrastructure-through-cucumber/

Cheers,
Lindsay

--
http://holmwood.id.au/~lindsay/ (me)

[Christian Kauhaus]

Julian Simpson schrieb:

> Those TDD style tests can be pretty ugly. In theory you can use them
> as a spec to describe what code does, but in practice, no. So a guy
> called Dan North coined the term Behaviour Driven Development, and
> focussed on the language used to help Business Analysts or other guys
> in suits to specify. This is where the Given/When/Then syntax comes
> from.

We use Puppet at our site quite extensively and try to start
implementing a feature by writing a custom Nagios check[1]. This starts
as a red one. After that we modify the manifests so that the check
passes. I think this is pretty much a TDD style of development. What
part of the procedure would I need to change in a BDD style of
development? Is this just another representation of the checks? Are
there differences in the process?

Regards

Christian

[1] With the Nagios::Plugin Perl library, this is just a breeze.

[Patrick Debois]

FYI: for those looking for examples and more discussions.

I've put my collection of infrastructure testing links online:
http://www.jedi.be/blog/2009/11/12/collection-of-test-driven-infrastructure-links/

[Christian Kauhaus]

Patrick Debois schrieb:

> I've put my collection of infrastructure testing links online:
> http://www.jedi.be/blog/2009/11/12/collection-of-test-driven-infrastructure-links/

Great compilation. Thank you for sharing it with us. :-)

Regards

Christian

[Ryan Dooley]

Agreed. Great collection of testing links.

Cheers,
Ryan