Question about collate

[Mike Place]

Hi all!

I'm seeing some unexpected behavior with SimpleCov and I would like to understand whether I'm facing a bug, a bad configuration, or if I'm simply misunderstanding the intended behavior. I am using version 0.18.5 of SimpleCov.

I have a series of tests which are executed in Jenkins using a matrix strategy. Each test ends up outputting SimpleCov output into a unique directory. After all tests have completed, we try to use SimpleCov to collate the results, through a rake task as follows:

namespace :coverage do

desc "Collates all result sets generated by the different test runners"

task :report do

require 'simplecov'

require 'simplecov-cobertura'

SimpleCov.formatter = SimpleCov::Formatter::CoberturaFormatter

SimpleCov.collate Dir["coverage/matrix_results/**/.resultset.json"]

end

end

This appears to work and a "coverage.xml" file is produced from the many ".resultset.json" files stored in a series of directories.

In inspecting the various ".resultset.json" files directly, I can see that coverage date is being accurately reported. Additionally, if I limit myself to just collating a single ".resultset.json" file the data in "coverage.xml" which is produced is accurate.

However, if I collate all of my ".resultset.json" files, I end up with 0% coverage across the board in my "coverage.xml" file.

My questions are:

1) Is the fact that I would see 0% coverage in this case expected behavior? I am assuming that by collating these files, that coverage data will be additive -- that is to say that it will never decrease as the result of collating an additional result set, but only potentially increase. Is this assumption correct?

2) Is there a way to debug this further? Can anybody recommend a technique that might help me understand what is happening here?

3) Is there a potentially better approach than the one I'm using?

Thanks in advance.

Best,

-Mike

[Tobias Pfeiffer]

Hi there!

It's fun to answer questions about features you didn't implement
yourself as I feel I'd need to know the answer straight away, but I don't :D

However, from re-reading the email it seems to me you have to put the
formatter inside the collate block
https://github.com/colszowka/simplecov#merging-test-runs-under-different-execution-environments

So to use the example from the README:

# lib/tasks/coverage_report.rake
namespace :coverage do

task :report do
require 'simplecov'

SimpleCov.collate Dir["simplecov-resultset-*/.resultset.json"],
'rails' do
formatter SimpleCov::Formatter::MultiFormatter.new([
SimpleCov::Formatter::SimpleFormatter,
SimpleCov::Formatter::HTMLFormatter
])
end
end
end

Hope that fixes your problem! If not please try with the HTML formatter
aka not Cobertura. If that still doesn't work please respond here or
open an issue :)

Thanks, cheers + stay safe,
Tobi

> However, if I collate /all /of my ".resultset.json" files, I end up with

> 0% coverage across the board in my "coverage.xml" file.
>
> My questions are:
>
> 1) Is the fact that I would see 0% coverage in this case expected
> behavior? I am assuming that by collating these files, that coverage
> data will be additive -- that is to say that it will never decrease as
> the result of collating an additional result set, but only potentially
> increase. Is this assumption correct?
>
> 2) Is there a way to debug this further? Can anybody recommend a
> technique that might help me understand what is happening here?
>
> 3) Is there a potentially better approach than the one I'm using?
>
> Thanks in advance.
>
> Best,
>
> -Mike
>

> --
> You received this message because you are subscribed to the Google
> Groups "SimpleCov" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to simplecov+...@googlegroups.com
> <mailto:simplecov+...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/simplecov/064292df-a03e-4af3-9e3b-82df8ba97329%40googlegroups.com
> <https://groups.google.com/d/msgid/simplecov/064292df-a03e-4af3-9e3b-82df8ba97329%40googlegroups.com?utm_medium=email&utm_source=footer>.

--
http://www.pragtob.info/

[Mike Place]

Hi Tobias and list!

Thanks so much for these suggestions. They were helpful!

It turned out that this wasn't a problem with collate and it was working as expected. Instead, the problem was that I wasn't actually collating all of my result sets and the few that were being collated did actually have 0% coverage. Once I added in the remaining result sets, I get correct coverage results.

Cheers and stay safe,

-Mike

> an email to simp...@googlegroups.com
> <mailto:simp...@googlegroups.com>.