Puppet module testing & code coverage...

[Gavin Williams]

Afternoon all

I've read a fair few blogs, slideshares and other sources recently around Module testing in Puppet, and it has spurned me on to start writing a test framework around our internal modules... Yes, we aren't currently testing our internal modules... Yes, we should be, and yes, I could give you a list of excuses why we aren't, but the long and short is we (I) want to start to :)

Anyhow, I've been playing with puppetlabs_spec_helper, puppet-lint, hiera-puppet-helper, librarian-puppet and Jenkins over the past couple of days, and have got a working test framework finally... After lots of to-ing and fro-ing with Hiera helpers, Rspec helpers, etc...

So I've currently got Jenkins running parser tests, lint tests, and am guna start running spec tests shortly, of which I currently have the grand sum of 1 (yes, one).
However one thing I'm struggling to find is how to generate Puppet test code coverage stats...

So Puppetlabs_spec_helper includes 'rake coverage', with a description of:
"rake coverage         # Generate code coverage information".

However this doesn't appear to currently do anything useful :(

$ ruby -S rcov -Ispec:lib --exclude spec ./spec/classes/server/linux_spec.rb
-bash-4.1$

I found a snippet on the Puppetlabs blog [1] which says:

'coverage' runs rcov on your spec tests; it is only useful if your module includes Ruby extensions to Puppet.

So my reading of that is that rcov doesn't check Puppet manifest coverage, only Ruby extensions?
If that's the case, is there any alternative that will generate Puppet manifest code coverage?

Or am I completely barking up the wrong tree???

As always, thanks in advance for any responses/comments/etc...

Cheers
Gavin

[1] http://puppetlabs.com/blog/the-next-generation-of-puppet-module-testing

[Stuart Cracraft]

Gavin,

Congratulations on your momentous decision. And it is an important one.

You may want to go to www.amazon.com and pick up How Google Tests Software

by Whittaker, Arbon, and Carollo.

It is also a serious effort.

Stuart

[Felix Frank]

Hi,

On 12/03/2013 06:23 PM, Stuart Cracraft wrote:
> You may want to go to www.amazon.com <http://www.amazon.com> and pick up

> How Google Tests Software by Whittaker, Arbon, and Carollo.

I encourage the plugging of pertinent literature, but I feel
uncomfortable about linking to specific retailers. That just crosses the
line to blunt advertising.

> If that's the case, is there any alternative that /will /generate Puppet
> manifest code coverage?

I'm not sure what you expect from 'manifest code coverage'.

If you write a spec to find out how a piece of manifest code (e.g. a
class) behaves, the blog you linked indicates that 'rake spec' is the
task you're looking for.

HTH,
Felix