evosuite reports other coverage data than jacoco

[Henrik Schole]

Dear all,

in my current project, evosuite seems to report other coverage data during test suite generation time than the tool that I use at runtime.

To reproduce the situation, you can find the code
https://bitbucket.org/h_schole/xcorpus-sample

Usage (installed java, ant and mercurial needed):
Clone, then

cd into xcorpus-sample/antlr-3.4/.xcorpus

To execute and produce the runtime coverage report call
ant -f exercise.xml generated-tests

If you wish to re-generate the relevant test call
ant -f exercise.xml generate-single-test


The evosuite report is located in antlr-3.4/.xcorpus/evosuite-report,
the jacoco report in antlr-3.4/.xcorpus/output/TIMESTAMP/ in different formats.

The relevant class is org.antlr.misc.Barrier.


For this class,

Evosuite reports ~98% coverage (seems to be branch coverage?)

Jacoco reports 91% instruction coverage and 75% branch coverage.


Do you see any reason for this?

Thank you for your time,
Henrik

[José Miguel Rojas]

Dear Henrik,

Two factors might have to do with the disagreement you are seeing.

First, by default, EvoSuite targets a combination of several coverage criteria, which are indicated in the 'criterion' column of the report:

> cat evosuite-report/statistics.csv 

TARGET_CLASS,criterion,Coverage,Total_Goals,Covered_Goals,,,,,,,

org.antlr.misc.Barrier,LINE,BRANCH,EXCEPTION,WEAKMUTATION,OUTPUT,METHOD,METHODNOEXCEPTION,CBRANCH,0.975,49,43

You can decide which criteria to target using the "-criterion" option, e.g., "-criterion BRANCH" will only target branch coverage.

Second, branch coverage is simply interpreted by most coverage tools like Jacoco as the number of branches of conditional statements covered by the test suite. EvoSuite, in contrast, looks at branches at the control flow graph level. Moreover, methods without any branching also contribute to the coverage computation for EvoSuite.

Hope that helps. Cheers,

--
You received this message because you are subscribed to the Google Groups "EvoSuite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to evosuite+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/evosuite/4afc3a19-a2ce-40d1-84c4-31f53f72def9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

José Miguel

[Henrik Schole]

Dear Jose,

alright, thanks for the fast reply.

Are there other ways to modify the report format? For the research project I am working on it would be of interest to keep the individual coverage criterions (one column each for branch, method etc.)

Cheers,
Henrik

[Jose Miguel Rojas Siles]

Sure, there is the "output_variables" argument, e.g.:

-Doutput_variables="TARGET_CLASS,Size,Length,criterion,Coverage,BranchCoverage,MethodCoverage,OutputCoverage"

Cheers,

[Henrik Schole]

Great, that's exactly what I was looking for. Thanks!

[Devendra Reddy]

Can you tell me how u used evosuite with ANT for generating the test cases