Get coverage per test case

[Martin Pokorny]

Hi guys,

Is there some way to see which test case or test script covers which line?

Reason.  We make in our company cpp coverage. We produce for each test suite extra cov file. From this we extract a big json file with mapping which test suite cover which file. This helps us to start test in feature branches depending on changes.  It decrease testing time rapidly with same code quality. Otherwise must developer start all of our test which takes more hours. And I want implement similar solutions for jenkins (as extra plugin). In jenkins is used mvn and jacoco.

So any tips? 

Thx 

Br Martin 

[Marc Hoffmann]

Hi Martin,

this is not supported out of the box by JaCoCo. But we have the building blocks to create something like this.

* You need to hook into your test frameworks to tell test cases apart

* Between each test case you need to trigger a exec file dump (or receive data via TCP) and reset probes

No the hard parts start which I have no idea how this scales for large projects: For every test case (exec file) you create a coverage report. This can be in-memory only, no need to write HTML or XML to disk for this. From this you would need to create a gigantic matrix between your code base and the test cases. Maybe you can aggregate both a bit to keep the matrix in a reasonable size (e.g. only in Java module or package level).

Regards,

-marc

Beside your scenario may I add my personal view on this: Code coverage should only be applied in white box testing (i.e. Unit Testing). This means there is a simple 1:1 mapping between test cases and code. Maybe not on method or class level, but at least on package or module level. That’s why I recommend to only count coverage within the module. Implicit coverage by using other modules must not be counted for white box test coverage.

--
You received this message because you are subscribed to the Google Groups "JaCoCo and EclEmma Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jacoco+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/CAPUx_sXByvbS3F_0-ucjpQk93v3068COpeLtrxmU9sbv73Qy7Q%40mail.gmail.com.

[Martin Pokorny]

Thx Marc

I think this is what I need. An hook in test teardown should bu no problem. So I have cov info per each test. When I use only test file vs covered files it shall be not so much information. About how to process and store the data I need some research. But there are many open-source tools with similar data binding. I will try to make it as next GSoC project via jenkins community. 

And about indirect covered files.  That is the biggest challenge.  When developer changes some behaviour he now in most cases which test shall be started. But in most cases are started some unit tests in best case component tests. But who cares about possible side effects?  And how shall the test executor executable know which test is the best one? This leads to binding more complex (probably AI) integration.  I know about the challenge.  We use it in our company - partly.  Now I want make some more standardised tool that might helps many communities around the world. 

Yes I know it is challenge, but I will try it

To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/A6012157-9598-4D97-A65E-F1E69C79E391%40mountainminds.com.

[Ala Batarseh]

 Hello all,

I thought I should comment on this issue. I am much interested in coverage per test, and I started a project on github some time ago to support "Scenario Test Coverage"  with Junit5 here https://github.com/nbs403/remote-jacoco-reporter 

I tried it at work for a proof of concept and hopefully will get back to it soon

[Hunk Huang]

Intersting in this topic too, Generate mapping between test case and source code may help reduce UT time significantly.