How to get Simplecov to report coverage correctly for puppet manifests

[Darragh Bailey]

Hi,

I'm to understand how to get a full coverage for puppet code with spec tests to be generated so I can get output like the example from simplecov at https://github.com/colszowka/simplecov#example-output.

Ideally I would subsequently try to hook into the coverage repo and do some post processing looking at where most of the code churn has been in our puppet control code repo and target those areas as important for test coverage (and likely refactoring).

However I've stuck on not being able to get output from simplecov that shows the manifests that were tested by the spec tests, although the summary triggered from running 'RSpec::Puppet::Coverage.report!' suggests that something was capable of tracking coverage correctly.

Although we have a different layout than normal, my testing so far suggests that the reporting simply doesn't work between puppet and simplecov.

Usually when something like this happens, I assume I'm doing something blindingly stupid, so I've put together some example code that shows my problem should anyone like to run through the spec tests from it.

Doing some simple experiments I've pushed up a repository showing the issue https://github.com/electrofelix/puppet-simplecov_bug containing 3 commits

- (HEAD commit): convert to match more closely skeleton layout from https://github.com/garethr/puppet-module-skeleton/blob/master/skeleton/Gemfile

- (HEAD~1 commit): try to follow a more standard layout

- (root commit): Use simple define taken from our codebase and matching the layout we have as closely as possible

Note we're still stuck on Puppet 3.8 and using ruby 2.1.10 via rvm while I was doing the spec tests, plan to upgrade to 5 in the near future, so if this problem has been solved via something in the puppet engine (some suggestions that the multiple loading by puppet was wiping SimpleCov's tracking over coverage) and we just need to hurry up that's great.

But no matter what layout/format I seem to try, the simplecov report prints nothing as having been covered, e.g.:

***************************************************************************************************

Total resources:   9
Touched resources: 8
Resource coverage: 88.89%
Untouched resources:

  Ssh::Config_entry[example custom-host]

Finished in 2.01 seconds (files took 1.05 seconds to load)
12 examples, 0 failures
Coverage report generated for RSpec to <path>/puppet-simplecov_bug/html. 0 / 67 LOC (0.0%) covered.

COVERAGE:   0.00% -- 0/67 lines in 2 files

+----------+-------------------------------+-------+--------+-----------------------------------------+
| coverage | file                          | lines | missed | missing                                 |
+----------+-------------------------------+-------+--------+-----------------------------------------+
|   0.00%  | manifests/server/localuser.pp | 67    | 67     | 5-13, 15-19, 21-34, 36-44, 46-55, 57-76 |
+----------+-------------------------------+-------+--------+-----------------------------------------+
1 file(s) with 100% coverage not shown

***************************************************************************************************

RSpec::Puppet::Coverage.report causes the first part to be printed and certainly it shows it's tracking coverage on the files touched, however the simplecov report at the end does not appear to be able to correlate the tests in 'spec/defines/server_localuser_spec.rb' as having tested 'manifests/server/localuser.pp'.

Additionally what is missing from both is all files that are not touched at all so far by tests, though I do have some code that appears to be able to handle that by passing 'track_files("**/*.pp")' to the SimpleCov object configuration.

Thinking maybe I started from the wrong place, I took https://github.com/garethr/puppet-module-skeleton and created a fresh module from that and then dropped in both the spec file (https://github.com/electrofelix/puppet-simplecov_bug/blob/master/spec/defines/server_localuser_spec.rb) and define (https://github.com/electrofelix/puppet-simplecov_bug/blob/master/manifests/server/localuser.pp) that were being tested and the result remained the same.

Rspec::Puppet::Coverage.report! gave:

Total resources:   15
Touched resources: 14
Resource coverage: 93.33%
Untouched resources:

  Ssh::Config_entry[example custom-host]

But SimpleCov gave:

Coverage report generated for RSpec to <path>/test-coverage/coverage. 0.0 / 0.0 LOC (100.0%) covered.

So even worse than before...

Resources I've looked at:

https://github.com/garethr/puppet-module-skeleton

https://github.com/rodjek/rspec-puppet

https://github.com/rodjek/rspec-puppet#producing-coverage-reports

https://github.com/rodjek/rspec-puppet/issues/316

https://github.com/puppetlabs/puppetlabs-rook (seems to have the same issue)

https://github.com/colszowka/simplecov#example-output

https://github.com/colszowka/simplecov/issues/682

https://github.com/colszowka/simplecov/issues/389

https://github.com/colszowka/simplecov/issues/340

https://github.com/colszowka/simplecov/issues/77

my example of the issue:

https://github.com/electrofelix/puppet-simplecov_bug

Hoping that there is something relatively trivial here that I'm missing that allows the reports generated by simplecov to correctly reference spec tests with puppet code.

Regards,

Darragh

[David Schmitt]

Hi Darragh,

SimpleCov is a ruby-only solution and the puppet code (*.pp) coverage from rspec-puppet does not integrate into it. As far as I can see there is currently no way to get at the raw data without hacking rspec-puppet itself.

Cheers, David

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/3afe6401-44d9-4bcb-8628-6bc76e4b3491%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Cheers, David

https://twitter.com/dev_el_ops

[Darragh Bailey]

On Mon, 18 Jun 2018, 09:19 David Schmitt, <david....@puppet.com> wrote:

Hi Darragh,

SimpleCov is a ruby-only solution and the puppet code (*.pp) coverage from rspec-puppet does not integrate into it. As far as I can see there is currently no way to get at the raw data without hacking rspec-puppet itself.

Cheers, David

Thanks that's good to know. Is there any coverage solution out there for puppet code that would produce something like simplecov, rather than just a simple summary of resources covered?

--

Darragh Bailey
"Nothing is foolproof to a sufficiently talented fool" - unknown

[David Schmitt]

On Mon, Jun 18, 2018 at 11:03 AM Darragh Bailey <daragh...@gmail.com> wrote:

On Mon, 18 Jun 2018, 09:19 David Schmitt, <david....@puppet.com> wrote:

Hi Darragh,

SimpleCov is a ruby-only solution and the puppet code (*.pp) coverage from rspec-puppet does not integrate into it. As far as I can see there is currently no way to get at the raw data without hacking rspec-puppet itself.

Cheers, David

Thanks that's good to know. Is there any coverage solution out there for puppet code that would produce something like simplecov, rather than just a simple summary of resources covered?

Not to my knowledge. From the code linked above it should be straight-forward for an engaged coder to create the required report file in simplecov's format to upload to a service like http://codecov.io for display/evaluation purposes. I'm sure the PDK Team would welcome such an addition.

Cheers, David

 

--

Darragh Bailey
"Nothing is foolproof to a sufficiently talented fool" - unknown

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CACR765_qGFCWsi7vKSPBGLN8V%2BmkMZuUa14u2i8iQxS%2B_Yzrng%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.