I've been struggling a bit with how we're using puppet (at my job):
how do you validate that puppet has done what it is supposed to, and
even troublesome, how you validate that it has done what you intended
it to do?
Since I'm the only one who is writing the puppet profiles and working
with it on a daily basis, I'm the only one who can decipher the puppet
logs. I often get the question: how do we know when the system is
ready for production?
I've been playing around[1] with cucumber & webrat, and have pieced
together a way to do behavior driven infrastructure testing. Puppet
takes care of getting the system configured correctly, but there are
often other pieces involved, like opening firewall ports, adding DNS
entries, sendmail routing, etc. Which must be done outside of puppet
to get the system ready for release.
When you write code, you always use unit testing & integration testing
to verify that the application is working as expected, but why don't
we use that when we install a system?
What are you using to verify that your system is correctly configured
and behaves the way you want?
> I've been struggling a bit with how we're using puppet (at my job): > how do you validate that puppet has done what it is supposed to, and > even troublesome, how you validate that it has done what you intended > it to do?
> Since I'm the only one who is writing the puppet profiles and working > with it on a daily basis, I'm the only one who can decipher the puppet > logs. I often get the question: how do we know when the system is > ready for production?
> I've been playing around[1] with cucumber & webrat, and have pieced > together a way to do behavior driven infrastructure testing. Puppet > takes care of getting the system configured correctly, but there are > often other pieces involved, like opening firewall ports, adding DNS > entries, sendmail routing, etc. Which must be done outside of puppet > to get the system ready for release.
> When you write code, you always use unit testing & integration testing > to verify that the application is working as expected, but why don't > we use that when we install a system?
> What are you using to verify that your system is correctly configured > and behaves the way you want?
[although i've commented on the excellent blog entry, i'm posting here]
Hi Martin, It looks like there's a bit of crossover here with a project i've been working on the last few months called cucumber-nagios[0]. It takes the result of a Cucumber run and outputs it in the Nagios plugin format. Essentially you use it to express your intentions in plain language, and verify your intentions periodically through your monitoring system. Just like what you've posted about. :-)
Anyhow, I spoke about cucumber-nagios at the excellent Devopsdays in Belgium last weekend, and I got talking with people about expanding the library of steps to cover things like logins over SSH, file manipulation, and mail delivery. It would be cool if we could centralise our efforts and focus on building an awesome library of reusable steps to test our infrastructure.
Your point about doing behaviour driven development when writing software is right on the mark. From an infrastructure perspective, I like to think of Cucumber as the testing tool, and Puppet as the programming language.
Anyhow, i'd be interested to hear what other people think about this idea!
On Nov 6, 3:56 pm, Lindsay Holmwood <lind...@holmwood.id.au> wrote:
> It looks like there's a bit of crossover here with a project i've been
> working on the last few months called cucumber-nagios[0]. It takes the
> result of a Cucumber run and outputs it in the Nagios plugin format.
> Essentially you use it to express your intentions in plain language,
> and verify your intentions periodically through your monitoring
> system. Just like what you've posted about. :-)
It sure does look like there is an overlap :)
> Anyhow, I spoke about cucumber-nagios at the excellent Devopsdays in
> Belgium last weekend, and I got talking with people about expanding
> the library of steps to cover things like logins over SSH, file
> manipulation, and mail delivery. It would be cool if we could
> centralise our efforts and focus on building an awesome library of
> reusable steps to test our infrastructure.
It is indeed better to work on this jointly. You are much further
along the way, so I'll clone your cucumber-nagios repository and work
in integrating the stuff I've done into your code. You'll be hearing
from me shortly...
I'm also talking to Ohad about integrating this into the foreman[1].
Then it will become the IDE of system/infrastructure development :)
> 2009/11/6 Martin Englund <martin.engl...@sun.com>:
> > Folks,
> > I've been struggling a bit with how we're using puppet (at my job):
> > how do you validate that puppet has done what it is supposed to, and
> > even troublesome, how you validate that it has done what you intended
> > it to do?
> > Since I'm the only one who is writing the puppet profiles and working
> > with it on a daily basis, I'm the only one who can decipher the puppet
> > logs. I often get the question: how do we know when the system is
> > ready for production?
> > I've been playing around[1] with cucumber & webrat, and have pieced
> > together a way to do behavior driven infrastructure testing. Puppet
> > takes care of getting the system configured correctly, but there are
> > often other pieces involved, like opening firewall ports, adding DNS
> > entries, sendmail routing, etc. Which must be done outside of puppet
> > to get the system ready for release.
> > When you write code, you always use unit testing & integration testing
> > to verify that the application is working as expected, but why don't
> > we use that when we install a system?
> > What are you using to verify that your system is correctly configured
> > and behaves the way you want?
> [although i've commented on the excellent blog entry, i'm posting here]
> Hi Martin,
> It looks like there's a bit of crossover here with a project i've been
> working on the last few months called cucumber-nagios[0]. It takes the
> result of a Cucumber run and outputs it in the Nagios plugin format.
> Essentially you use it to express your intentions in plain language,
> and verify your intentions periodically through your monitoring
> system. Just like what you've posted about. :-)
> Anyhow, I spoke about cucumber-nagios at the excellent Devopsdays in
> Belgium last weekend, and I got talking with people about expanding
> the library of steps to cover things like logins over SSH, file
> manipulation, and mail delivery. It would be cool if we could
> centralise our efforts and focus on building an awesome library of
> reusable steps to test our infrastructure.
> Your point about doing behaviour driven development when writing
> software is right on the mark. From an infrastructure perspective, I
> like to think of Cucumber as the testing tool, and Puppet as the
> programming language.
> Anyhow, i'd be interested to hear what other people think about this idea!
I am certainly interested from a infrastructure point of view having like
you say a library of
reusable steps which would fit the bill nicely, The ability to build with
Puppet and check that build with Cucumber-Nagios would be awesome and more
in line in which I would consider Test Driven Infrastructure.
for a while I was experimenting with cucumber to test scripts running
on linux machines. And yes , IMHO it is the way to go .
It worked well with the behavior testing just a you guys described.
still just as in regular development, bdd can be complemented with
tdd.
The way i did the tdd, was by using the exit codes of the shell
commands. I created an abstraction in ruby for execute (local, remote)
and upload, download of files.
If the command did not have exit 0 (the default) then i imagined the
test being failed.
Using snapshots of the virtual machines, I ran the scripts against a
virtual machine (used virtualbox), with the ability to do a rollback.
I know this is not the way puppet normally operates, but I used the
(if i recall correctly) the puppetrun command to execute it and see
what happened
If all of these exit codes where ok (similar to unit tests), i ran the
bdd tests to see if everything was correct.
By integrating the tdd, and the bdd within a CI-system, I was able to
have it run continuously and rebuild the system over and over again.
Another thing I want to mention is that you can easily abstract
commands in a custom dsl, but I really like to see what actual
commands got executed.
This is slightly off topic maybe, but by logging/running the actual
commands and have the whole environment build like that (including
making a floppy, booting the vms)
it allowed me to create a kind of install document with the actual
commands instead of pointing to a newly invented DSL.
So I say yes to an abstraction layer, but no to hiding the actual
commands.
just my 2 c
On Nov 6, 3:56 pm, Lindsay Holmwood <lind...@holmwood.id.au> wrote:
> 2009/11/6 Martin Englund <martin.engl...@sun.com>:
> > Folks,
> > I've been struggling a bit with how we're using puppet (at my job):
> > how do you validate that puppet has done what it is supposed to, and
> > even troublesome, how you validate that it has done what you intended
> > it to do?
> > Since I'm the only one who is writing the puppet profiles and working
> > with it on a daily basis, I'm the only one who can decipher the puppet
> > logs. I often get the question: how do we know when the system is
> > ready for production?
> > I've been playing around[1] with cucumber & webrat, and have pieced
> > together a way to do behavior driven infrastructure testing. Puppet
> > takes care of getting the system configured correctly, but there are
> > often other pieces involved, like opening firewall ports, adding DNS
> > entries, sendmail routing, etc. Which must be done outside of puppet
> > to get the system ready for release.
> > When you write code, you always use unit testing & integration testing
> > to verify that the application is working as expected, but why don't
> > we use that when we install a system?
> > What are you using to verify that your system is correctly configured
> > and behaves the way you want?
> [although i've commented on the excellent blog entry, i'm posting here]
> Hi Martin,
> It looks like there's a bit of crossover here with a project i've been
> working on the last few months called cucumber-nagios[0]. It takes the
> result of a Cucumber run and outputs it in the Nagios plugin format.
> Essentially you use it to express your intentions in plain language,
> and verify your intentions periodically through your monitoring
> system. Just like what you've posted about. :-)
> Anyhow, I spoke about cucumber-nagios at the excellent Devopsdays in
> Belgium last weekend, and I got talking with people about expanding
> the library of steps to cover things like logins over SSH, file
> manipulation, and mail delivery. It would be cool if we could
> centralise our efforts and focus on building an awesome library of
> reusable steps to test our infrastructure.
> Your point about doing behaviour driven development when writing
> software is right on the mark. From an infrastructure perspective, I
> like to think of Cucumber as the testing tool, and Puppet as the
> programming language.
> Anyhow, i'd be interested to hear what other people think about this idea!
I'd be interested in hearing more about this. I've been working on a project in which we do a lot of automated server builds and configuration using various OSS tools, including puppet. However, one thing that I've been looking for is a way to write a set of test suites to validate the server configuration. The combination of TDD and BDD with server rollouts is very attractive.
I'd also be curious to know what CI server people are using to help coordinate these intermediate builds and tests.
> for a while I was experimenting with cucumber to test scripts running
> on linux machines. And yes , IMHO it is the way to go .
> It worked well with the behavior testing just a you guys described.
> still just as in regular development, bdd can be complemented with
> tdd.
> The way i did the tdd, was by using the exit codes of the shell
> commands. I created an abstraction in ruby for execute (local, remote)
> and upload, download of files.
> If the command did not have exit 0 (the default) then i imagined the
> test being failed.
> Using snapshots of the virtual machines, I ran the scripts against a
> virtual machine (used virtualbox), with the ability to do a rollback.
> I know this is not the way puppet normally operates, but I used the
> (if i recall correctly) the puppetrun command to execute it and see
> what happened
> If all of these exit codes where ok (similar to unit tests), i ran the
> bdd tests to see if everything was correct.
> By integrating the tdd, and the bdd within a CI-system, I was able to
> have it run continuously and rebuild the system over and over again.
> Another thing I want to mention is that you can easily abstract
> commands in a custom dsl, but I really like to see what actual
> commands got executed.
> This is slightly off topic maybe, but by logging/running the actual
> commands and have the whole environment build like that (including
> making a floppy, booting the vms)
> it allowed me to create a kind of install document with the actual
> commands instead of pointing to a newly invented DSL.
> So I say yes to an abstraction layer, but no to hiding the actual
> commands.
> just my 2 c
> On Nov 6, 3:56 pm, Lindsay Holmwood<lind...@holmwood.id.au> wrote:
>> 2009/11/6 Martin Englund<martin.engl...@sun.com>:
>>> Folks,
>>> I've been struggling a bit with how we're using puppet (at my job):
>>> how do you validate that puppet has done what it is supposed to, and
>>> even troublesome, how you validate that it has done what you intended
>>> it to do?
>>> Since I'm the only one who is writing the puppet profiles and working
>>> with it on a daily basis, I'm the only one who can decipher the puppet
>>> logs. I often get the question: how do we know when the system is
>>> ready for production?
>>> I've been playing around[1] with cucumber& webrat, and have pieced
>>> together a way to do behavior driven infrastructure testing. Puppet
>>> takes care of getting the system configured correctly, but there are
>>> often other pieces involved, like opening firewall ports, adding DNS
>>> entries, sendmail routing, etc. Which must be done outside of puppet
>>> to get the system ready for release.
>>> When you write code, you always use unit testing& integration testing
>>> to verify that the application is working as expected, but why don't
>>> we use that when we install a system?
>>> What are you using to verify that your system is correctly configured
>>> and behaves the way you want?
>> [although i've commented on the excellent blog entry, i'm posting here]
>> Hi Martin,
>> It looks like there's a bit of crossover here with a project i've been
>> working on the last few months called cucumber-nagios[0]. It takes the
>> result of a Cucumber run and outputs it in the Nagios plugin format.
>> Essentially you use it to express your intentions in plain language,
>> and verify your intentions periodically through your monitoring
>> system. Just like what you've posted about. :-)
>> Anyhow, I spoke about cucumber-nagios at the excellent Devopsdays in
>> Belgium last weekend, and I got talking with people about expanding
>> the library of steps to cover things like logins over SSH, file
>> manipulation, and mail delivery. It would be cool if we could
>> centralise our efforts and focus on building an awesome library of
>> reusable steps to test our infrastructure.
>> Your point about doing behaviour driven development when writing
>> software is right on the mark. From an infrastructure perspective, I
>> like to think of Cucumber as the testing tool, and Puppet as the
>> programming language.
>> Anyhow, i'd be interested to hear what other people think about this idea!
I've used Test::Unit to wrap validations of generated config files in
the past. I want to know as early as possible if something isn't
going to work.
Functionally testing the config from the outside in might be easy but
slow. You can deploy to a VM and actually test it using the goodness
of cucumber. I'm still looking for the equivalent of unit tests: how
do you quickly test function from the inside to give enough confidence
and a fast feedback loop? Maybe we need to build out Patrick's
approach with a library of command-line tools to verify that
everything works.
J.
2009/11/7 Isaac Christoffersen <isaac.christoffer...@gmail.com>:
> I'd be interested in hearing more about this. I've been working on a
> project in which we do a lot of automated server builds and
> configuration using various OSS tools, including puppet. However, one
> thing that I've been looking for is a way to write a set of test suites
> to validate the server configuration. The combination of TDD and BDD
> with server rollouts is very attractive.
> I'd also be curious to know what CI server people are using to help
> coordinate these intermediate builds and tests.
> On 11/7/09 11:00 AM, Patrick Debois wrote:
>> for what it's worth:
>> for a while I was experimenting with cucumber to test scripts running
>> on linux machines. And yes , IMHO it is the way to go .
>> It worked well with the behavior testing just a you guys described.
>> still just as in regular development, bdd can be complemented with
>> tdd.
>> The way i did the tdd, was by using the exit codes of the shell
>> commands. I created an abstraction in ruby for execute (local, remote)
>> and upload, download of files.
>> If the command did not have exit 0 (the default) then i imagined the
>> test being failed.
>> Using snapshots of the virtual machines, I ran the scripts against a
>> virtual machine (used virtualbox), with the ability to do a rollback.
>> I know this is not the way puppet normally operates, but I used the
>> (if i recall correctly) the puppetrun command to execute it and see
>> what happened
>> If all of these exit codes where ok (similar to unit tests), i ran the
>> bdd tests to see if everything was correct.
>> By integrating the tdd, and the bdd within a CI-system, I was able to
>> have it run continuously and rebuild the system over and over again.
>> Another thing I want to mention is that you can easily abstract
>> commands in a custom dsl, but I really like to see what actual
>> commands got executed.
>> This is slightly off topic maybe, but by logging/running the actual
>> commands and have the whole environment build like that (including
>> making a floppy, booting the vms)
>> it allowed me to create a kind of install document with the actual
>> commands instead of pointing to a newly invented DSL.
>> So I say yes to an abstraction layer, but no to hiding the actual
>> commands.
>> just my 2 c
>> On Nov 6, 3:56 pm, Lindsay Holmwood<lind...@holmwood.id.au> wrote:
>>> 2009/11/6 Martin Englund<martin.engl...@sun.com>:
>>>> Folks,
>>>> I've been struggling a bit with how we're using puppet (at my job):
>>>> how do you validate that puppet has done what it is supposed to, and
>>>> even troublesome, how you validate that it has done what you intended
>>>> it to do?
>>>> Since I'm the only one who is writing the puppet profiles and working
>>>> with it on a daily basis, I'm the only one who can decipher the puppet
>>>> logs. I often get the question: how do we know when the system is
>>>> ready for production?
>>>> I've been playing around[1] with cucumber& webrat, and have pieced
>>>> together a way to do behavior driven infrastructure testing. Puppet
>>>> takes care of getting the system configured correctly, but there are
>>>> often other pieces involved, like opening firewall ports, adding DNS
>>>> entries, sendmail routing, etc. Which must be done outside of puppet
>>>> to get the system ready for release.
>>>> When you write code, you always use unit testing& integration testing
>>>> to verify that the application is working as expected, but why don't
>>>> we use that when we install a system?
>>>> What are you using to verify that your system is correctly configured
>>>> and behaves the way you want?
>>> [although i've commented on the excellent blog entry, i'm posting here]
>>> Hi Martin,
>>> It looks like there's a bit of crossover here with a project i've been
>>> working on the last few months called cucumber-nagios[0]. It takes the
>>> result of a Cucumber run and outputs it in the Nagios plugin format.
>>> Essentially you use it to express your intentions in plain language,
>>> and verify your intentions periodically through your monitoring
>>> system. Just like what you've posted about. :-)
>>> Anyhow, I spoke about cucumber-nagios at the excellent Devopsdays in
>>> Belgium last weekend, and I got talking with people about expanding
>>> the library of steps to cover things like logins over SSH, file
>>> manipulation, and mail delivery. It would be cool if we could
>>> centralise our efforts and focus on building an awesome library of
>>> reusable steps to test our infrastructure.
>>> Your point about doing behaviour driven development when writing
>>> software is right on the mark. From an infrastructure perspective, I
>>> like to think of Cucumber as the testing tool, and Puppet as the
>>> programming language.
>>> Anyhow, i'd be interested to hear what other people think about this idea!
I haven't had time to do much more than scan the web site, but they seem to be worth watching.
I've also started including test.pp scripts as part of each module. These scripts check directories or do some execs (i.e. wgets, pings, lookups). These scripts then run with every puppet update push. However, this isn't ideal for production and there needs to be a better way to trigger the running of unit tests. But it has been helpful in validating iptables configuration and ensuring services and their clients are on the same network.
> I've used Test::Unit to wrap validations of generated config files in > the past. I want to know as early as possible if something isn't > going to work.
> Functionally testing the config from the outside in might be easy but > slow. You can deploy to a VM and actually test it using the goodness > of cucumber. I'm still looking for the equivalent of unit tests: how > do you quickly test function from the inside to give enough confidence > and a fast feedback loop? Maybe we need to build out Patrick's > approach with a library of command-line tools to verify that > everything works.
> J.
> 2009/11/7 Isaac Christoffersen<isaac.christoffer...@gmail.com>:
>> I'd be interested in hearing more about this. I've been working on a >> project in which we do a lot of automated server builds and >> configuration using various OSS tools, including puppet. However, one >> thing that I've been looking for is a way to write a set of test suites >> to validate the server configuration. The combination of TDD and BDD >> with server rollouts is very attractive.
>> I'd also be curious to know what CI server people are using to help >> coordinate these intermediate builds and tests.
>> On 11/7/09 11:00 AM, Patrick Debois wrote:
>>> for what it's worth:
>>> for a while I was experimenting with cucumber to test scripts running >>> on linux machines. And yes , IMHO it is the way to go .
>>> It worked well with the behavior testing just a you guys described. >>> still just as in regular development, bdd can be complemented with >>> tdd. >>> The way i did the tdd, was by using the exit codes of the shell >>> commands. I created an abstraction in ruby for execute (local, remote) >>> and upload, download of files.
>>> If the command did not have exit 0 (the default) then i imagined the >>> test being failed. >>> Using snapshots of the virtual machines, I ran the scripts against a >>> virtual machine (used virtualbox), with the ability to do a rollback. >>> I know this is not the way puppet normally operates, but I used the >>> (if i recall correctly) the puppetrun command to execute it and see >>> what happened >>> If all of these exit codes where ok (similar to unit tests), i ran the >>> bdd tests to see if everything was correct.
>>> By integrating the tdd, and the bdd within a CI-system, I was able to >>> have it run continuously and rebuild the system over and over again.
>>> Another thing I want to mention is that you can easily abstract >>> commands in a custom dsl, but I really like to see what actual >>> commands got executed. >>> This is slightly off topic maybe, but by logging/running the actual >>> commands and have the whole environment build like that (including >>> making a floppy, booting the vms) >>> it allowed me to create a kind of install document with the actual >>> commands instead of pointing to a newly invented DSL. >>> So I say yes to an abstraction layer, but no to hiding the actual >>> commands.
>>> just my 2 c
>>> On Nov 6, 3:56 pm, Lindsay Holmwood<lind...@holmwood.id.au> wrote:
>>>> 2009/11/6 Martin Englund<martin.engl...@sun.com>:
>>>>> Folks,
>>>>> I've been struggling a bit with how we're using puppet (at my job): >>>>> how do you validate that puppet has done what it is supposed to, and >>>>> even troublesome, how you validate that it has done what you intended >>>>> it to do?
>>>>> Since I'm the only one who is writing the puppet profiles and working >>>>> with it on a daily basis, I'm the only one who can decipher the puppet >>>>> logs. I often get the question: how do we know when the system is >>>>> ready for production?
>>>>> I've been playing around[1] with cucumber& webrat, and have pieced >>>>> together a way to do behavior driven infrastructure testing. Puppet >>>>> takes care of getting the system configured correctly, but there are >>>>> often other pieces involved, like opening firewall ports, adding DNS >>>>> entries, sendmail routing, etc. Which must be done outside of puppet >>>>> to get the system ready for release.
>>>>> When you write code, you always use unit testing& integration testing >>>>> to verify that the application is working as expected, but why don't >>>>> we use that when we install a system?
>>>>> What are you using to verify that your system is correctly configured >>>>> and behaves the way you want?
>>>> [although i've commented on the excellent blog entry, i'm posting here]
>>>> Hi Martin, >>>> It looks like there's a bit of crossover here with a project i've been >>>> working on the last few months called cucumber-nagios[0]. It takes the >>>> result of a Cucumber run and outputs it in the Nagios plugin format. >>>> Essentially you use it to express your intentions in plain language, >>>> and verify your intentions periodically through your monitoring >>>> system. Just like what you've posted about. :-)
>>>> Anyhow, I spoke about cucumber-nagios at the excellent Devopsdays in >>>> Belgium last weekend, and I got talking with people about expanding >>>> the library of steps to cover things like logins over SSH, file >>>> manipulation, and mail delivery. It would be cool if we could >>>> centralise our efforts and focus on building an awesome library of >>>> reusable steps to test our infrastructure.
>>>> Your point about doing behaviour driven development when writing >>>> software is right on the mark. From an infrastructure perspective, I >>>> like to think of Cucumber as the testing tool, and Puppet as the >>>> programming language.
>>>> Anyhow, i'd be interested to hear what other people think about this idea!
My understanding is that Test Driven Development hit the mainstream
(did smalltalk have a unit test framework?) with JUnit.[1] Junit
allowed developers to automate their unit tests. All the unit tests
for your project should run in seconds, if your codebase and tests are
loosely coupled.
e.g. testThatTheFooObjectReturnsABarWhenPassedABaz
Those TDD style tests can be pretty ugly. In theory you can use them
as a spec to describe what code does, but in practice, no. So a guy
called Dan North coined the term Behaviour Driven Development, and
focussed on the language used to help Business Analysts or other guys
in suits to specify. This is where the Given/When/Then syntax comes
from.
He worked on JBehave as an alternative to JUnit so you can specify
behavior. But Rspec and Cucumber are where the notion has taken off.
I'd write more but I have to make my kids' dinner :)
2009/11/6 Martin Englund <martin.engl...@sun.com>:
> It is indeed better to work on this jointly. You are much further > along the way, so I'll clone your cucumber-nagios repository and work > in integrating the stuff I've done into your code. You'll be hearing > from me shortly...
So i've added some SSH steps to cucumber-nagios, along the lines of what Martin blogged about a few days ago.
I've published a blog post about the new feature, and how this could potentially be used withina Behaviour Driven Infrastructure techniques:
> Those TDD style tests can be pretty ugly. In theory you can use them > as a spec to describe what code does, but in practice, no. So a guy > called Dan North coined the term Behaviour Driven Development, and > focussed on the language used to help Business Analysts or other guys > in suits to specify. This is where the Given/When/Then syntax comes > from.
We use Puppet at our site quite extensively and try to start implementing a feature by writing a custom Nagios check[1]. This starts as a red one. After that we modify the manifests so that the check passes. I think this is pretty much a TDD style of development. What part of the procedure would I need to change in a BDD style of development? Is this just another representation of the checks? Are there differences in the process?
Regards
Christian
[1] With the Nagios::Plugin Perl library, this is just a breeze. -- Dipl.-Inf. Christian Kauhaus <>< · k...@gocept.com · systems administration gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 11 · fax +49 345 1229889 1 Zope and Plone consulting and development
