Slides TDD with puppet - puppetcamp paris yesterday

[Johan De Wit]

Hi,

http://www.slideshare.net/johandw/20140408-tdd-puppetcampparis

Comments are welcome

Grts


Johan


--
Johan De Wit

Open Source Consultant

Red Hat Certified Engineer (805008667232363)
Puppet Certified Professional 2013/2014 (PCP0000006)
_________________________________________________________

Open-Future Phone +32 (0)2/255 70 70
Zavelstraat 72 Fax +32 (0)2/255 70 71
3071 KORTENBERG Mobile +32 (0)474/42 40 73
BELGIUM http://www.open-future.be
_________________________________________________________


Next Events:
Puppet Introduction Course | http://www.open-future.be/puppet-introduction-course-14th-april
Puppet Advanced Training | http://www.open-future.be/puppet-advanced-training-15-till-17th-april
Linux Training | https://www.open-future.be/linux-training-5-till-9th-may
Puppet Introduction Course | https://www.open-future.be/puppet-introduction-course-12th-may
Subscribe to our newsletter | http://eepurl.com/BUG8H

[Julien Deloubes]

Hi Johan,
congrats for for presentation yesterday, very interesting, i miss the end because i had to pass the cert at 3PM, so i will finish it on slideshare :)
Bye

[Gareth Rushgrove]

Ha.

Snap.

A talk I gave at Puppetcamp London:
https://speakerdeck.com/garethr/test-driven-development-for-puppet

And another talk from Loaddays by Garrett:
http://www.slideshare.net/gh/20140406-loa-daystddwithpuppettutorial

That's three talks with the same title in less than a week. I believe
we're on to something.

Gareth

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/53451EBC.3090608%40open-future.be.
> For more options, visit https://groups.google.com/d/optout.



--
Gareth Rushgrove
@garethr

devopsweekly.com
morethanseven.net
garethrushgrove.com

[Johan De Wit]

And add another one to the list. Well, i did the same talk for the
Berlin Puppet User Group, as the one I did in Paris.

I still am so surprised when asking who is doing some kind of 'testing',
almost nobody raises is hand .....

Most people just don't' see the sense of doing rspec unit tests - why
writing the same code twice ?

Well, I think there is still a lot of talking to do ....

Next thing I will be organising is a TDD workshop with the Belgian
Puppet User Group very soon.

Grts

Johan

[Alexander Fortin]

On Sunday, April 13, 2014 11:57:19 AM UTC+2, Johan De Wit wrote:

I still am so surprised when asking who is doing some kind of 'testing',
almost nobody raises is hand .....

Most people just don't' see the sense of doing rspec unit tests - why
writing the same code twice ?

Well, I think there is still a lot of talking to do ....

Hi Johan,

sorry for the missed presentation at Puppet User Group Berlin, I'll make it one day or another and I'll share the slides at least ;)

I'll be very happy to discuss further the testing topic, because testing is also very important to us. Actually we do lots of testing on our manifests (but not with rspec) and catalogs, and precisely I think that puppet catalog-diff [1] is the key piece in the pipeline for us, because it shows us for real what's actually changing with every commit we push to our manifests, plus the noop runs before merging to prod give us the final safe net to be sure we don't get any unexpected change in production. For the last 8 months or so it's been working surprisingly well for us and I'm really willing to share our experience with the community and also getting and suggestions about how to improve it even further.

By the way, I'm another one that's not so convinced about the rspec tests value, to me seems that the unit tests themselves are much less relevant for a declarative-like language like Puppet. I mean, there's no 'design' that has to emerge by the unit tests getting green, in our team we already share a defined design for module structure, and, say, if the coder write the spec for a file to be there, I don't see why I should trust that more than a definition of that same thing in the manifest itself, hence the feeling of code duplication with no real value added.

What I can see very clearly for normal software development, those values coming from TDD, I can't see easily for Puppet manifests writing. Something like Beaker [2] is what we'd like to add to our testing pipeline, i.e. running tests for the full stack in a VM, but again, stil more then willing to change my mind about rspec-puppet ;)

[1] https://github.com/ripienaar/puppet-catalog-diff

[2] https://github.com/puppetlabs/beaker/wiki/Overview

-- 
http://about.me/alexanderfortin

[Sven Sporer]

Hey Alexander,

you could also check out serverspec[1] for acceptance tests. It provides simple RSpec tests for your server with a lot of puppet-like matchers, but it's not tied to Puppet (heresy! :). We use it to validate our modules and are very happy with it.

Sven

[1] http://serverspec.org/

[Nikola Petrov]

I see the value of using rspec in TDD just because I can then go back
and refactor the manifests properly. I haven't used catalog-diff but I
am suspecting that it might be a substitute for the rspec tests. Tell me
does it provide the following:

* Automatic runs through guard/rake/whatever on every commit to verify
that I am not breaking things like permissions on files
* Does it work on the catalog semantic level or syntactic level - what
I mean by that is that if I have defined 4 files in a class each with
the same user/group/mode and I move those into default parameters for
the File resource in the scope - will it detect that I am not changing
anything?
* Will I be able to test that things haven't changed once I move to
another puppet version - let's say the default mode for directories
change in the next major puppet version?

>
> What I can see very clearly for normal software development, those values
> coming from TDD, I can't see easily for Puppet manifests writing. Something
> like Beaker [2] is what we'd like to add to our testing pipeline, i.e.
> running tests for the full stack in a VM, but again, stil more then willing
> to change my mind about rspec-puppet ;)

Beaker with http://serverspec.org/ is quite nice - note that I am still
not using it long enough to complain(waiting it to happen :))

[Felix Frank]

On 04/15/2014 09:25 AM, Nikola Petrov wrote:
> I haven't used catalog-diff but I
> am suspecting that it might be a substitute for the rspec tests. Tell me

No. rspec allows you to implement unit tests to catch certain regression
bugs. catalog-diff is rather concerned with integration testing
(although you can use rspec for that as well).

> does it provide the following:
>
> * Automatic runs through guard/rake/whatever on every commit to verify
> that I am not breaking things like permissions on files

You could do that, but compiling each of your nodes' catalog on every
commit would be excessive.

> * Does it work on the catalog semantic level or syntactic level - what
> I mean by that is that if I have defined 4 files in a class each with
> the same user/group/mode and I move those into default parameters for
> the File resource in the scope - will it detect that I am not changing
> anything?

Absolutely. The mode of each file that the agent is supposed to manage
is part of the catalog. A change in agent behavior is always prompted by
a change in its catalog.

> * Will I be able to test that things haven't changed once I move to
> another puppet version - let's say the default mode for directories
> change in the next major puppet version?

Generally yes, though at PuppetCamp I heard that catalog-diff had issues
with a certain release. I believe that a given catalog-diff version may
not work with both your old and new puppet release. But apparently it's
possible to coerce the catalog output into a diffable format if push
comes to shove.

Cheers,
Felix

[Alexander Fortin]

On Tuesday, April 15, 2014 1:10:05 PM UTC+2, Felix.Frank wrote:

On 04/15/2014 09:25 AM, Nikola Petrov wrote:
> I haven't used catalog-diff but I
> am suspecting that it might be a substitute for the rspec tests. Tell me

No. rspec allows you to implement unit tests to catch certain regression
bugs. catalog-diff is rather concerned with integration testing
(although you can use rspec for that as well).

I think with catalog-diff we're catching regression bugs too, meaning that if the module interface is broken in the proposed commit, it will show up as an error (unless no node is actually including the refactored module).

 

> does it provide the following:
>
> * Automatic runs through guard/rake/whatever on every commit to verify
> that I am not breaking things like permissions on files

You could do that, but compiling each of your nodes' catalog on every
commit would be excessive.

I agree it can be overkill for many environments, I guess depends on what you actually want to test, or better, what kind of confidence you want to have that your committed code is not actually breaking anything.

To me seems that the catalog-diff approach is catching a broad variety of problems, also some that (I think) can't be easily catch by expressions in rspec.

For example, somewhere in a random manifest I'm just changing declaration style for a class, from 'include myclass' to "class {'myclass': }", leading to possible declaration dependency order / duplicated declarations problems: Node A (which I am currently working on) is including only one declaration, but node B was evaluating that class declaration two times, hence the duplicate declaration only on node B.

Can rspec-puppet catch also this kind of problems? More in general, I'd like to know if rspec-puppet can give me the confidence that, given a node resource, the new version of the catalog will compile for that node.

 

 --

Alex

[Johan De Wit]

Great having a discussion on testing. 

I have to reread all your comments, having a busy week catching up with work after a week of having fun at all the puppetcamps.

I keep on tracking this thread, and will add my thoughts soon.

Thanks for all your comments'

Johan

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/826f278e-1b69-4721-9e35-7028627c44fa%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

-- 
Johan De Wit

Open Source Consultant

Red Hat Certified Engineer              (805008667232363)
Puppet Certified Professional 2013/2014 (PCP0000006)
_________________________________________________________
 
Open-Future                 Phone     +32 (0)2/255 70 70
Zavelstraat 72              Fax       +32 (0)2/255 70 71
3071 KORTENBERG             Mobile    +32 (0)474/42 40 73
BELGIUM                     http://www.open-future.be
_________________________________________________________
 

Upcoming Events:

[Nikola Petrov]

On Tue, Apr 15, 2014 at 06:46:21AM -0700, Alexander Fortin wrote:
> On Tuesday, April 15, 2014 1:10:05 PM UTC+2, Felix.Frank wrote:
> >
> > On 04/15/2014 09:25 AM, Nikola Petrov wrote:
> > > I haven't used catalog-diff but I
> > > am suspecting that it might be a substitute for the rspec tests. Tell me
> >
> > No. rspec allows you to implement unit tests to catch certain regression
> > bugs. catalog-diff is rather concerned with integration testing
> > (although you can use rspec for that as well).
> >
>
> I think with catalog-diff we're catching regression bugs too, meaning that
> if the module interface is broken in the proposed commit, it will show up
> as an error (unless no node is actually including the refactored module).

Hmm I am suspecting that sadly we are talking about sort of different
things here. In my company we have reusable puppet modules between
different teams - those are build by experts and require major testing
because we don't want to break things in incremental versions. The thing
is that those modules are not applied as is but are mostly reused in
different project specific modules(i.e. site modules)

I can see catalog-diff valuable for our site modules because it is
actually testing the end result - what will be applied to a given host.
In most cases the reusable modules are used in different ways with
different parameters - let's say a tomcat module that is responsible for
deploying war files and what not but cannot be used standalone(you have
to wrap it in some way)

>
>
>
> > > does it provide the following:
> > >
> > > * Automatic runs through guard/rake/whatever on every commit to verify
> > > that I am not breaking things like permissions on files
> >
> > You could do that, but compiling each of your nodes' catalog on every
> > commit would be excessive.
> >

I don't see it as such a big problem. You can always put more resources
on your CI server to make that possible which leads me to my next
question - have you guys automated this in some way. I would be glad to
catch regressions from catalog-diff on all my hosts but automatically -
let's say through rspec or some other srcipt.

Shameless plug: I have been doing a more mature approval port for java
lately https://github.com/nikolavp/approval and although it is not
ready, there is a good looking thing for ruby here
https://github.com/kytrinyx/approvals. The idea is that you approve the
output of master compile the first time the test is run and then(on
further runs) if there are differences(detected through catalog-diff)
you are prompted(let's say in your editor). This allows you to
"approve" the differences if you know that they won't do any harm but in
a more automatic way.

> >
> I agree it can be overkill for many environments, I guess depends on what
> you actually want to test, or better, what kind of confidence you want to
> have that your committed code is not actually breaking anything.
>
> To me seems that the catalog-diff approach is catching a broad variety of
> problems, also some that (I think) can't be easily catch by expressions in
> rspec.
>
> For example, somewhere in a random manifest I'm just changing declaration
> style for a class, from 'include myclass' to "class {'myclass': }", leading
> to possible declaration dependency order / duplicated declarations
> problems: Node A (which I am currently working on) is including only one
> declaration, but node B was evaluating that class declaration two times,
> hence the duplicate declaration only on node B.
>
> Can rspec-puppet catch also this kind of problems? More in general, I'd
> like to know if rspec-puppet can give me the confidence that, given a node
> resource, the new version of the catalog will compile for that node.
>

Actually no - that's what we use staging for. You push your modules up
there and see if they run properly on real machines. We also use the
modules through vagrant on every developer machines - so they are
constantly tested.

Of course we want to start using beaker for those kind of tests so we
don't push crap on the bear metal(although it is not metal most of the
time)
> --
> Alex

>
> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/826f278e-1b69-4721-9e35-7028627c44fa%40googlegroups.com.

[Felix Frank]

On 04/17/2014 09:06 AM, Nikola Petrov wrote:
> Of course we want to start using beaker for those kind of tests so we
> don't push crap on the bear metal(although it is not metal most of the
> time)

...but usually just bears. ;-D

(Yeah I know. Sorry. It is just such a cute typo that I couldn't resist.)

[Nikola Petrov]

:D. Too fast for da bears.

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/534F86CA.2030408%40alumni.tu-berlin.de.