Why does JaCoCo report uncovered lines which are in fact covered?
4,487 views
Skip to first unread message
maila...@gmail.com
Apr 27, 2015, 3:21:45 PM4/27/15
to jac...@googlegroups.com
Hello everyone,
I have an issue with JaCoCo that I don't understand. It might be a bug or just me being stupid so I thought I ask before opening an issue.
I created a project on GitHub [1] and integrated it with coveralls.io using the JaCoCo Maven plugin to generate the coverage report. The issue surfaces in the run method of the Main class on line 48 [2]: all branches of the if statement are covered in tests and yet JaCoCo reports the body of the if statement as uncovered. It is not an issue with coveralls.io because the exact same coverage is shown in the JaCoCo HTML report if I build the project on my machine. The HTML report even states on line 48 that "all 4 branches [are] covered".
Am I doing something wrong?
[1] https://github.com/hzpz/access-export
[2] https://coveralls.io/builds/2422271/source?filename=src%2Fmain%2Fjava%2Fnet%2Fkockert%2Faccess%2Fexport%2FMain.java#L48
I have an issue with JaCoCo that I don't understand. It might be a bug or just me being stupid so I thought I ask before opening an issue.
I created a project on GitHub [1] and integrated it with coveralls.io using the JaCoCo Maven plugin to generate the coverage report. The issue surfaces in the run method of the Main class on line 48 [2]: all branches of the if statement are covered in tests and yet JaCoCo reports the body of the if statement as uncovered. It is not an issue with coveralls.io because the exact same coverage is shown in the JaCoCo HTML report if I build the project on my machine. The HTML report even states on line 48 that "all 4 branches [are] covered".
Am I doing something wrong?
[1] https://github.com/hzpz/access-export
[2] https://coveralls.io/builds/2422271/source?filename=src%2Fmain%2Fjava%2Fnet%2Fkockert%2Faccess%2Fexport%2FMain.java#L48
Marc R. Hoffmann
Apr 29, 2015, 2:24:18 PM4/29/15
to jac...@googlegroups.com
Hi,
I don't know, what the SystemExitHandler in line 42 actually does, but if it does a System.exit() and the method never returns the whole block is not marked as covered. This is a documented limitation, see our FAQ: http://www.eclemma.org/jacoco/trunk/doc/faq.html
Regards,
-marc
I don't know, what the SystemExitHandler in line 42 actually does, but if it does a System.exit() and the method never returns the whole block is not marked as covered. This is a documented limitation, see our FAQ: http://www.eclemma.org/jacoco/trunk/doc/faq.html
Code with exceptions shows no coverage. Why?
JaCoCo determines code execution with so called probes. Probes are inserted into the control flow at certain positions. Code is considered as executed when a subsequent probe has been executed. In case of exceptions such a sequence of instructions is aborted somewhere in the middle and not marked as executed.
Regards,
-marc
maila...@gmail.com
Apr 29, 2015, 3:42:34 PM4/29/15
to jac...@googlegroups.com
thanks for your reply!
I know about this limitation and that is one reason the SystemExitHandler interface exists ;-) The implementation I use in tests does not call System.exit() but throws an exception instead. This way I am able to use JUnit's @Test annotation with an 'expected' in my tests.
By the way, both Cobertura as well as the coverage runner integrated in IntelliJ IDEA 14.1 report the coverage that I am expecting. I am more and more convinced that this is a bug in JaCoCo. Or is it just another limitation?
Any thoughts?
Timo
Marc R. Hoffmann
Apr 29, 2015, 3:50:13 PM4/29/15
to jac...@googlegroups.com
It is a limitation by design: JaCoCo adds probes very sparingly to limit
the runtime overhead. That's why it is the fastest coverage tool which
scales also for *very* large code bases.
In case you're interested here are more detailed information about the
probe insertion strategies of JaCoCo:
http://www.eclemma.org/jacoco/trunk/doc/flow.html
Regards,
-marc
the runtime overhead. That's why it is the fastest coverage tool which
scales also for *very* large code bases.
In case you're interested here are more detailed information about the
probe insertion strategies of JaCoCo:
http://www.eclemma.org/jacoco/trunk/doc/flow.html
Regards,
-marc
Timo
Apr 29, 2015, 4:29:42 PM4/29/15
to jac...@googlegroups.com
Thanks again for the explanation and the link!
I see the problem now, but I still don't understand how JaCoCo can tell that all four branches of the if statement are covered but assumes that the body was not covered. I guess that is because I have almost no understanding of how Java source code is compiled to bytecode and know even less how exactly JaCoCo inserts probes. Something to figure out on a cold and rainy evening ;-)
Any ideas how I could restructure my code to see less "false positives" with JaCoCo?
To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/554135F6.1050200%40mountainminds.com.--
You received this message because you are subscribed to a topic in the Google Groups "JaCoCo and EclEmma Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jacoco/gIubNSFGNlU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jacoco+un...@googlegroups.com.
Marc R. Hoffmann
Apr 29, 2015, 5:08:36 PM4/29/15
to jac...@googlegroups.com
Hi,
indeed, interesting observation with the 4 covered branches. I can reproduce with the small example below. Actually if you look at the byte code the probes inserted by JaCoCo will create enough information that both paths for the conditions a and b have been taken. But for the body itself the implicit exception will prevent it from getting marked as covered.
Regards,
-marc
import org.junit.Test;
public class Branches4 {
void target(boolean a, boolean b) {
if (a || b) {
exception();
}
}
private void exception() {
throw new IllegalStateException();
}
@Test
public void test1() {
target(false, false);
}
@Test(expected=IllegalStateException.class)
public void test2() {
target(true, false);
}
@Test(expected=IllegalStateException.class)
public void test3() {
target(false, true);
indeed, interesting observation with the 4 covered branches. I can reproduce with the small example below. Actually if you look at the byte code the probes inserted by JaCoCo will create enough information that both paths for the conditions a and b have been taken. But for the body itself the implicit exception will prevent it from getting marked as covered.
Regards,
-marc
import org.junit.Test;
public class Branches4 {
void target(boolean a, boolean b) {
if (a || b) {
exception();
}
}
private void exception() {
throw new IllegalStateException();
}
@Test
public void test1() {
target(false, false);
}
@Test(expected=IllegalStateException.class)
public void test2() {
target(true, false);
}
@Test(expected=IllegalStateException.class)
public void test3() {
target(false, true);
You received this message because you are subscribed to the Google Groups "JaCoCo and EclEmma Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jacoco+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/CAONNSUYDwcxgVW73pDMhzF4YmnEbea0%2BAGuE6j4mZo%2BRu1fudg%40mail.gmail.com.
Timo
Apr 30, 2015, 3:16:11 AM4/30/15
to jac...@googlegroups.com
I figured as much. I was thrown off by the example (http://www.eclemma.org/jacoco/trunk/doc/flow.html). If you look at the control flow diagram, the probes get inserted _after_ the instructions in the body (invoke b() and invoke c()). That cannot be what is actually happening, otherwise JaCoCo couldn't know about the branch coverage (if b() or c() would throw an exception).
Thanks for taking the time to discuss this with me!To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/55414854.7010703%40mountainminds.com.
Marc Hoffmann
Apr 30, 2015, 3:31:04 AM4/30/15
to jac...@googlegroups.com
Your right. In the example given in the paper the control flow merges
after the invokation of b() and c().
If you decompile your example you will see that the control flow for a
|| b merges *before* the method invokations, that's why there are probes
before the exception happens.
On 2015-04-30 09:16, Timo wrote:
> I figured as much. I was thrown off by the example
> (http://www.eclemma.org/jacoco/trunk/doc/flow.html [1]). If you look
>> http://www.eclemma.org/jacoco/trunk/doc/flow.html [1]
>> [2]
>>
> https://coveralls.io/builds/2422271/source?filename=src%2Fmain%2Fjava%2Fnet%2Fkockert%2Faccess%2Fexport%2FMain.java#L48
>> [5]
>
> For more options, visit https://groups.google.com/d/optout [8].
> For more options, visit https://groups.google.com/d/optout [8].
> [10].
>
> For more options, visit https://groups.google.com/d/optout [8].
> [11].
> For more options, visit https://groups.google.com/d/optout [8].
>
>
> Links:
> ------
> [1] http://www.eclemma.org/jacoco/trunk/doc/flow.html
> [2] http://www.eclemma.org/jacoco/trunk/doc/faq.html
> [3] http://coveralls.io
> [4] https://github.com/hzpz/access-export
> [5]
> https://coveralls.io/builds/2422271/source?filename=src%2Fmain%2Fjava%2Fnet%2Fkockert%2Faccess%2Fexport%2FMain.java#L48
> [6] https://groups.google.com/d/topic/jacoco/gIubNSFGNlU/unsubscribe
> [7]
> https://groups.google.com/d/msgid/jacoco/554135F6.1050200%40mountainminds.com
> [8] https://groups.google.com/d/optout
> [9]
> https://groups.google.com/d/msgid/jacoco/CAONNSUYDwcxgVW73pDMhzF4YmnEbea0%2BAGuE6j4mZo%2BRu1fudg%40mail.gmail.com?utm_medium=email&utm_source=footer
> [10]
> https://groups.google.com/d/msgid/jacoco/55414854.7010703%40mountainminds.com?utm_medium=email&utm_source=footer
> [11]
> https://groups.google.com/d/msgid/jacoco/CAONNSUbJf0fqyhm65psLLABs1Ftn%3Di9du4UYg1181mEFyXeSVg%40mail.gmail.com?utm_medium=email&utm_source=footer
after the invokation of b() and c().
If you decompile your example you will see that the control flow for a
|| b merges *before* the method invokations, that's why there are probes
before the exception happens.
On 2015-04-30 09:16, Timo wrote:
> I figured as much. I was thrown off by the example
>>
>> Regards,
>> -marc
>>
>> On 29.04.15 21:42, maila...@gmail.com wrote:
>> Am Mittwoch, 29. April 2015 20:24:18 UTC+2 schrieb Marc R.
>> Hoffmann:
>> Hi,
>>
>> I don't know, what the SystemExitHandler in line 42 actually
>> does,
>> but if it does a System.exit() and the method never returns
>> the
>> whole block is not marked as covered. This is a documented
>> limitation, see our FAQ:
>> http://www.eclemma.org/jacoco/trunk/doc/faq.html [2]
>> Regards,
>> -marc
>>
>> On 29.04.15 21:42, maila...@gmail.com wrote:
>> Am Mittwoch, 29. April 2015 20:24:18 UTC+2 schrieb Marc R.
>> Hoffmann:
>> Hi,
>>
>> I don't know, what the SystemExitHandler in line 42 actually
>> does,
>> but if it does a System.exit() and the method never returns
>> the
>> whole block is not marked as covered. This is a documented
>> limitation, see our FAQ:
>>
>> Code with exceptions shows no coverage. Why?
>> JaCoCo determines code execution with so called probes.
>> Probes are inserted into the control flow at certain
>> positions. Code is considered as executed when a
>> subsequent
>> probe has been executed. In case of exceptions such a
>> sequence
>> of instructions is aborted somewhere in the middle and
>> not
>> marked as executed.
>>
>> Regards,
>>
>> -marc
>>
>> On 27.04.15 21:21, maila...@gmail.com wrote:
>>
>> Hello everyone,
>>
>> I have an issue with JaCoCo that I don't understand. It might be a
>> bug or just me being stupid so I thought I ask before opening an
>> issue.
>>
>> I created a project on GitHub [1] and integrated it with
>> coveralls.io [3] using the JaCoCo Maven plugin to generate the
>> Code with exceptions shows no coverage. Why?
>> JaCoCo determines code execution with so called probes.
>> Probes are inserted into the control flow at certain
>> positions. Code is considered as executed when a
>> subsequent
>> probe has been executed. In case of exceptions such a
>> sequence
>> of instructions is aborted somewhere in the middle and
>> not
>> marked as executed.
>>
>> Regards,
>>
>> -marc
>>
>> On 27.04.15 21:21, maila...@gmail.com wrote:
>>
>> Hello everyone,
>>
>> I have an issue with JaCoCo that I don't understand. It might be a
>> bug or just me being stupid so I thought I ask before opening an
>> issue.
>>
>> I created a project on GitHub [1] and integrated it with
>> coverage report. The issue surfaces in the run method of the Main
>> class on line 48 [2]: all branches of the if statement are covered
>> in tests and yet JaCoCo reports the body of the if statement as
>> uncovered. It is not an issue with coveralls.io [3] because the
>> class on line 48 [2]: all branches of the if statement are covered
>> in tests and yet JaCoCo reports the body of the if statement as
>> exact same coverage is shown in the JaCoCo HTML report if I build
>> the project on my machine. The HTML report even states on line 48
>> that "all 4 branches [are] covered".
>>
>> Am I doing something wrong?
>>
>> [1] https://github.com/hzpz/access-export [4]
>> the project on my machine. The HTML report even states on line 48
>> that "all 4 branches [are] covered".
>>
>> Am I doing something wrong?
>>
>> [2]
>>
> https://coveralls.io/builds/2422271/source?filename=src%2Fmain%2Fjava%2Fnet%2Fkockert%2Faccess%2Fexport%2FMain.java#L48
>> [5]
>> Hi Marc,
>>
>> thanks for your reply!
>>
>> I know about this limitation and that is one reason the
>> SystemExitHandler interface exists ;-) The implementation I use in
>> tests does not call System.exit() but throws an exception instead.
>> This way I am able to use JUnit's @Test annotation with an
>> 'expected' in my tests.
>>
>> By the way, both Cobertura as well as the coverage runner
>> integrated in IntelliJ IDEA 14.1 report the coverage that I am
>> expecting. I am more and more convinced that this is a bug in
>> JaCoCo. Or is it just another limitation?
>>
>> Any thoughts?
>>
>> Timo
>
> --
> You received this message because you are subscribed to a topic in
> the Google Groups "JaCoCo and EclEmma Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/jacoco/gIubNSFGNlU/unsubscribe [6].
>>
>> thanks for your reply!
>>
>> I know about this limitation and that is one reason the
>> SystemExitHandler interface exists ;-) The implementation I use in
>> tests does not call System.exit() but throws an exception instead.
>> This way I am able to use JUnit's @Test annotation with an
>> 'expected' in my tests.
>>
>> By the way, both Cobertura as well as the coverage runner
>> integrated in IntelliJ IDEA 14.1 report the coverage that I am
>> expecting. I am more and more convinced that this is a bug in
>> JaCoCo. Or is it just another limitation?
>>
>> Any thoughts?
>>
>> Timo
>
> --
> You received this message because you are subscribed to a topic in
> the Google Groups "JaCoCo and EclEmma Users" group.
> To unsubscribe from this topic, visit
> To unsubscribe from this group and all its topics, send an email to
> jacoco+un...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jacoco/554135F6.1050200%40mountainminds.com
> [7].
> jacoco+un...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jacoco/554135F6.1050200%40mountainminds.com
>
> For more options, visit https://groups.google.com/d/optout [8].
>
> --
> You received this message because you are subscribed to the Google
> Groups "JaCoCo and EclEmma Users" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to jacoco+un...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jacoco/CAONNSUYDwcxgVW73pDMhzF4YmnEbea0%2BAGuE6j4mZo%2BRu1fudg%40mail.gmail.com
> [9].
> --
> You received this message because you are subscribed to the Google
> Groups "JaCoCo and EclEmma Users" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to jacoco+un...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jacoco/CAONNSUYDwcxgVW73pDMhzF4YmnEbea0%2BAGuE6j4mZo%2BRu1fudg%40mail.gmail.com
> For more options, visit https://groups.google.com/d/optout [8].
>
> --
> You received this message because you are subscribed to a topic in
> the Google Groups "JaCoCo and EclEmma Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/jacoco/gIubNSFGNlU/unsubscribe [6].
> --
> You received this message because you are subscribed to a topic in
> the Google Groups "JaCoCo and EclEmma Users" group.
> To unsubscribe from this topic, visit
> To unsubscribe from this group and all its topics, send an email to
> jacoco+un...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jacoco/55414854.7010703%40mountainminds.com
> jacoco+un...@googlegroups.com.
> To view this discussion on the web visit
> [10].
>
> For more options, visit https://groups.google.com/d/optout [8].
>
> --
> You received this message because you are subscribed to the Google
> Groups "JaCoCo and EclEmma Users" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to jacoco+un...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jacoco/CAONNSUbJf0fqyhm65psLLABs1Ftn%3Di9du4UYg1181mEFyXeSVg%40mail.gmail.com
> --
> You received this message because you are subscribed to the Google
> Groups "JaCoCo and EclEmma Users" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to jacoco+un...@googlegroups.com.
> To view this discussion on the web visit
> [11].
> For more options, visit https://groups.google.com/d/optout [8].
>
>
> Links:
> ------
> [1] http://www.eclemma.org/jacoco/trunk/doc/flow.html
> [2] http://www.eclemma.org/jacoco/trunk/doc/faq.html
> [3] http://coveralls.io
> [4] https://github.com/hzpz/access-export
> [5]
> https://coveralls.io/builds/2422271/source?filename=src%2Fmain%2Fjava%2Fnet%2Fkockert%2Faccess%2Fexport%2FMain.java#L48
> [6] https://groups.google.com/d/topic/jacoco/gIubNSFGNlU/unsubscribe
> [7]
> https://groups.google.com/d/msgid/jacoco/554135F6.1050200%40mountainminds.com
> [8] https://groups.google.com/d/optout
> [9]
> https://groups.google.com/d/msgid/jacoco/CAONNSUYDwcxgVW73pDMhzF4YmnEbea0%2BAGuE6j4mZo%2BRu1fudg%40mail.gmail.com?utm_medium=email&utm_source=footer
> [10]
> https://groups.google.com/d/msgid/jacoco/55414854.7010703%40mountainminds.com?utm_medium=email&utm_source=footer
> [11]
> https://groups.google.com/d/msgid/jacoco/CAONNSUbJf0fqyhm65psLLABs1Ftn%3Di9du4UYg1181mEFyXeSVg%40mail.gmail.com?utm_medium=email&utm_source=footer
Reply all
Reply to author
Forward
This conversation is locked
You cannot reply and perform actions on locked conversations.
0 new messages