Average coverage taken as 60% whilst actually 72%

906 views
Skip to first unread message

Joris van der Geer

unread,
Dec 9, 2021, 5:06:00 PM12/9/21
to JaCoCo and EclEmma Users
The limit for mnimum coverage is incorrectly computed by averaging rounded numbers, instead of rouding the average.

A discrepancy of well over 10% can result between the HTML summary report and the pass/fail criterion.

For example, if the limit is set at 0.7 - note, one decimal - , and the summary table shows 72% in the HTML report, then the limit is taken as 0.6, equivalent to 60%

That is 12% off. The problem is that the average is calculated over rounded  / truncated numbers, instead of rounding the result. This aggravates rounding errors due to accumulation. The actual difference between rounded and not rounded gets inflated by each element.

HTML report :
  • a               70%
  • b               72%
  • c               70%
  • d               65%
  • e               86%
  • f                93%
  • g               100%
  • ----------------------
  • average   73%

JaCoCo version 0.8.5.201910111838

Maven build : "Rule violated for bundle mybundle: complexity covered ratio is 0.6, but expected minimum is 0.7"
 

Marc Hoffmann

unread,
Dec 9, 2021, 5:14:58 PM12/9/21
to jac...@googlegroups.com
Hi Joris,

The limit for mnimum coverage is incorrectly computed by averaging rounded numbers, instead of rouding the average.

This is not how JaCoCo reports work. The coverage percentage is always covered items divided by total items on the respective granularity. Nowhere we do calculate an “average” value.

Regards,
-marc
 

--
You received this message because you are subscribed to the Google Groups "JaCoCo and EclEmma Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jacoco+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/5d4469de-4955-4ce3-9395-7c91615978aen%40googlegroups.com.

Joris van der Geer

unread,
Dec 9, 2021, 5:21:43 PM12/9/21
to JaCoCo and EclEmma Users
The value as used in Maven's 'JaCoCo report check' limit is rounded the wrong way. As you see, the report shows 72%, yet the error messasge from the check says it is 0.6. That is the discrepancy. Likely the check does not use the 'total' field but does its own averaging the wrong way.

Marc Hoffmann

unread,
Dec 9, 2021, 5:50:48 PM12/9/21
to JaCoCo and EclEmma Users
Can you please show the configuration of your check goal the the exact output you see?



Message has been deleted

Joris van der Geer

unread,
Dec 9, 2021, 7:00:42 PM12/9/21
to JaCoCo and EclEmma Users
Sure, it is the "0.6" versus "73%" discrepancy, caused by the checker.
 
Abbreviated config in pom.xml:

                <id>jacoco-check</id>
                <phase>prepare-package</phase>
                <goals>
                  <goal>check</goal>
                </goals>
                <configuration>
                  <rules>
                    <rule implementation="org.jacoco.maven.RuleConfiguration">
                      <element>BUNDLE</element>
                      <limits>
                        <limit implementation="org.jacoco.report.check.Limit">
                          <counter>COMPLEXITY</counter>
                          <value>COVEREDRATIO</value>
                          <minimum>0.7</minimum>
                        </limit>
                      </limits>
                    </rule>
                  </rules>
                </configuration>

Maven output error :

Rule violated for bundle mybundle: complexity covered ratio is 0.6, but expected minimum is 0.7

Output report table :

            Missed Instructions  Cov.  Missed Branches Cov. Missed Cxty Missed Lines Missed Methods Missed Classe
a           3352,171                    86%  72172                    70%  78        218   88         581     8            88             0           12 
b           120731                       85%   3078                      72%  39        95     34         244    9            41             0             4
c            64480                         88%   614                        70%   9         48    13         110     3            38             0             7
d           59434                          88%   917                        65%   9         23     7           98       0            10            0              2
e           39512                          92%   319                        86%   6         43    12         113     3             32            0             7
f           271,801                       98%   341                         93%   4         47     7          132     1             25            0             5
g          44                               100%  4                             100%  0          3      0            10     0                1            0             1
h                                              100%                                  n/a    0           3     0              3     0                 3            0             1
Total    644 of 6,833               90%  123 of 468             73% 145      480  161      1,291 24          238            0             39
     
Created with JaCoCo 0.8.5.201910111838

Marc Hoffmann

unread,
Dec 10, 2021, 1:21:40 AM12/10/21
to JaCoCo and EclEmma Users
Hi Joris,

the output formatting of the check goal is determined by the precision given in the limit configuration. Can you please change the config to

   <minimum>0.700</minimum>


Also note that the report goal and the check goal are independent from each other. If you have any excludes configured in the report goal make sure you have the same configuration in the check goal.

Regards,
-marc


Joris van der Geer

unread,
Dec 10, 2021, 3:53:33 AM12/10/21
to JaCoCo and EclEmma Users
There is one set of exclusions for both check and report. I cannot rerun locally so that takes a while. But my point remains that it behaves as if the check uses the single-digit to compute the total in the first place.
As you see, the report shows 73%, whilst the check reports 0.6. With more digits it would still be 0.6xx. Or, with different rounding, still 0.7 < 73%. The precision - or the lack thereof - appears to be applied to internal calcs as well. Adding precision is a workable workaround, but I do see a bug here. There is a discrepancy more than explainable by rounding at output formatting only.

Marc Hoffmann

unread,
Dec 10, 2021, 4:00:14 AM12/10/21
to jac...@googlegroups.com
But my point remains that it behaves as if the check uses the single-digit to compute the total in the first place.

This is not how JaCoCo works. Have a look at the code base. As I explained before internally JaCoCo always calculates covered items / total items in full precision. Only the error message is rounded to the same precision as given in the configuration.

Let’s see what the new error message with extended digits says and then look what causes the diff between the report and the check goal.


Joris van der Geer

unread,
Dec 10, 2021, 4:45:27 AM12/10/21
to JaCoCo and EclEmma Users
The goal is 0.7, the coverage is 73%. No output formatting can make 73 be below 0.7. As the 0.7 is specified literally, it cannot be interpreted as 0.75. The 3 numbers '0.73', '0.7' and '0.6' tell the story. If the issue is where I think it is, extending digits to 0.7000 will indeed work around the issue, but, as a workaround typically does, mask it. It is catchy the way it works now. I can understand that no-one has time for it and it is a minor issue, but the combination of the 3 numbers is clear.

Evgeny Mandrikov

unread,
Dec 15, 2021, 4:21:04 PM12/15/21
to JaCoCo and EclEmma Users
According to

            Missed Instructions  Cov.  Missed Branches Cov. Missed Cxty Missed Lines Missed Methods Missed Classe
a           3352,171                    86%  72172                    70%  78        218   88         581     8            88             0           12 
b           120731                       85%   3078                      72%  39        95     34         244    9            41             0             4
c            64480                         88%   614                        70%   9         48    13         110     3            38             0             7
d           59434                          88%   917                        65%   9         23     7           98       0            10            0              2
e           39512                          92%   319                        86%   6         43    12         113     3             32            0             7
f           271,801                       98%   341                         93%   4         47     7          132     1             25            0             5
g          44                               100%  4                             100%  0          3      0            10     0                1            0             1
h                                              100%                                  n/a    0           3     0              3     0                 3            0             1
Total    644 of 6,833               90%  123 of 468             73% 145      480  161      1,291 24          238            0             39

number of covered branches is 123,
total number of branches is 468,
and
73% is ratio of uncovered branches to total,
i.e. (468-123)/468

whereas your check configuration is not about branches, but about 

<counter>COMPLEXITY</counter>

missed complexity is 145,
total complexity is 480,
so ratio of covered complexity to total is (480-145)/480,
which is less than 0.7
which is 0.697 with 3 digits precision.
 
Reply all
Reply to author
Forward
0 new messages