Code Coverage Figures: Percentage vs. Untested Code

[Marc Hoffmann]

Hi all,

as the Jenkins plug-in will also show code coverage figures, I would
like to suggest a different presentation: Basically we shouldn't care
about percentage, it's a fruiteless metric that can not be compared. For
example we take to projects with the following code coverage:

A: 80%
B: 30%

Question: Which one has better testing? Let's assume project A is 100
KLOC, project B is 1 KLOC. So in Project A we have 20 KLOC of untested
code, in Project B it's 700 lines only. So the 80% project has a way
higher technical depth.

Therefore native JaCoCo reports show "missed" items (lines, methods,
classes, etc.) and also sort by this criterion. I would recommend to
follow the same principle for the Jenkins plug-in: For example for the
trend graph I would prefer two stacked areas: missed items (red) and
covered items (green), the total high represents the total amount of
items. So if your code base grows over time you get good trend chart
about the amount of total and untested code. From my point of view way
better than any percentage figures.

Just my 2ct,
-marc

[Mickael Istria]

On 07/02/2012 05:40 PM, Marc Hoffmann wrote:

Therefore native JaCoCo reports show "missed" items (lines, methods, classes, etc.) and also sort by this criterion. I would recommend to follow the same principle for the Jenkins plug-in: For example for the trend graph I would prefer two stacked areas: missed items (red) and covered items (green), the total high represents the total amount of items. So if your code base grows over time you get good trend chart about the amount of total and untested code. From my point of view way better than any percentage figures.

+1 !
https://issues.jenkins-ci.org/browse/JENKINS-14279

--
Mickael Istria
Eclipse developer at JBoss, by Red Hat
My blog - My Tweets

[Ognjen Bubalo]

Hi Marc,

Sounds reasonable. I was just wondering that I heard that the managers like percentage figures, but it would be more usable on the way you described.

I think that change won't take too much effort. 

Thank you,

Ogi

[Marc R. Hoffmann]

Forget about managers -- usually they don't write unit tests to improve
the test coverage. Code coverage tools should be used by the people who
actually create tests. Focussing on the areas with the biggest amount of
untested code really helps to improve your test coverage (even the
percentage ;-).

Cheers,
-marc

[Ognjen Bubalo]

Hello!

I have a change. What do you think? 

I also copied the column order, so now it is like in the JaCoCo's report.

The next step would be the chart.

Br, Ogi

[Marc Hoffmann]

Hallo Ogi,

the bars still represent percentage. In JaCoCo reports
(http://www.eclemma.org/jacoco/trunk/coverage/) the total length of the
bar represent the amount of code. So you can visually compare the amout
of e.g. untested code.

Cheers,
-marc


http://www.eclemma.org/jacoco/trunk/coverage/

On Wed, 4 Jul 2012 07:33:46 -0700 (PDT), Ognjen Bubalo wrote:
> Hello!
>
> I have a change. What do you think?
> I also copied the column order, so now it is like in the JaCoCo's
> report.
>

> [1]

> The next step would be the chart.
>
> Br, Ogi
>
>

> Links:
> ------
> [1]
>
> https://lh5.googleusercontent.com/-zft2dePXFZs/T_RUJajTYJI/AAAAAAAAABU/1hDL_BgWmuA/s1600/sample.jpg

[Ognjen Bubalo]

Hi Marc,

By total length of the bar you mean the green-red bar's length or the maximum length?

So if we look here:

http://www.eclemma.org/jacoco/trunk/coverage/org.jacoco.agent/org.jacoco.agent/AgentJar.html

The getResource() method with this counter: <counter type="INSTRUCTION" missed="5" covered="8"/>

show us that the whole getResource() method has 13 instructions. The whole AgentJar has 94 instructions, so how come the green bar is this long?

Or if we look at extractTo(File destination) it is all green.

So what is the maximum of a bar?

I do not understand something :).

Thank you,

Ogi

[Marc R. Hoffmann]

Hi Ogi,

of course bar lengths can only be compared relatively with one group. In
JaCoCo reports the maximum bar length is fixed. Bars within the same
group are scaled in a way that the element with the max. number of items
has a bar of exactly this maximum length.

Cheers,
-marc

[Ognjen Bubalo]

Hi Marc,

Ok thank you! I think I understand it now. It will be done soon.

Br, Ogi