Coverage reports for tests

[Lucas Werkmeister]

Hi all!

I just managed to get coverage reports for ceylon.ast tests working, and I thought that this would be useful to others as well, as coverage reports are pretty useful information when writing or improving tests.

(If you don’t know what I’m talking about, the short version is that coverage reports tell you what parts of your program are actually ran when you run your tests.
Bad coverage usually means that you should write more tests to ensure that you’ll catch bugs early.
This is what coverage can look like when integrated into the IDE;
the screenshot indicates that perhaps you should consider testing the addAll() method with an empty collection argument.)

If you’re using Ant, you can mostly copy the bottom part of my build.xml. You just need to change the names on lines 316-318 and 353.
See this commit for usage.

If you want to do it manually, here’s what you have to do:

1. Download and extract JaCoCo from here. (You only need the jacocoant.jar and jacocoagent.jar files.)
2. Run your tests with JaCoCo attached. If you use ceylon run or ceylon test, simply set the environment variable JAVA_OPTS to -javaagent:[yourpath/]jacocoagent.jar. For example, on Linux:
   
   JAVA_OPTS="-javaagent:lib/jacocoagent.jar" ceylon test test.my.module
   
   If you call java directly, add -javaagent:lib/jacocoagent.jar to the command line.
   You can also customize JaCoCo; see here for more information.
3. Now you have a jacoco.exec file in your project directory, which contains the raw coverage data.
   To generate a report, JaCoCo will need the class files, and it can’t get them from the CAR file, so we need to extract them.
   For example, on Linux:
   
   find modules -name '*.car' -exec unzip {} -x '*META-INF*' -d classes/ \;
4. Now we can generate the report.
   Unfortunately, that’s a bit of a pain from the command line; apparently it’s simply not possible without either Ant or Maven.
   You can use this sample build.xml file; adapt the file and directory paths in it as necessary, then run ant.
5. You’re done, look at the file coverage-report/index.html in your browser! If you did everything correctly (and I didn’t mess up the instructions ;-) ), then you should get detailed coverage information, and if you click all the way through to a function and click on that, you even get to see the source code, with green/yellow/red background indicating full/partial/no coverage.

Keep in mind that Ceylon instructions don’t directly correspond to Java (or JVM) instructions. For certain code, the Ceylon compiler even adds code that will never be executed. For example:

• A switch/case statement without an else case is usually compiled to an if/else if chain, with a final else clause that looks like this:
  
  else throw new .com.redhat.ceylon.compiler.java.language.EnumeratedTypeError("Supposedly exhaustive switch was not exhaustive");
• The initializer of a toplevel object contains code to detect and handle cyclic initialization; unless your objects are actually cyclic, it will never run.
• Defaulted function parameters generate an overloaded function (with less parameters) and a function that holds the default value computation; depending on how you call these functions (with or without explicit arguments), some of these default values may never be evaluated.
• The catch block of a try-with-resources can get pretty complicated when multiple resources are used (shuffling around suppressed exceptions), resulting in a disproportionate amount of instructions. (This is true for Java as well.)
  

So don’t despair because you don’t get 100% coverage!

I hope this helps some people :)

– Lucas

[Stephane Epardaud]

Great! Would it make sense to integrate this in the IDE and tweak it for Ceylon's specifics?

--
You received this message because you are subscribed to the Google Groups "ceylon-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ceylon-users...@googlegroups.com.
To post to this group, send email to ceylon...@googlegroups.com.
Visit this group at http://groups.google.com/group/ceylon-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceylon-users/41cd201f-42f0-4d18-8f45-12a39c71c192%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Stéphane Épardaud

[Philippe Lhoste]

On Thursday, 11 September 2014 22:42:14 UTC+2, Lucas Werkmeister wrote:

I just managed to get coverage reports for ceylon.ast tests working, and I thought that this would be useful to others as well, as coverage reports are pretty useful information when writing or improving tests.

Great! I was precisely wondering if we could have such tool for Ceylon, as it is quite useful (if you are not coding in TDD..)
At wok, for Java, we use Emma (and EclEmma Eclipse plugin).

 

So don’t despair because you don’t get 100% coverage!

If you have 100 % coverage, you might do wrong tests, particularly in Java were it is useless to test beans (accessors), or toString() (mostly used as debug information).
So we usually set a threshold, like 85 % of logic must be tested. "Logic", because in GWT, we have Views which are dependent on JavaScript, so cannot be tested with simple (and fast) unit tests. But using MVP, we try and keep views as dumb as possible (driven by presenters), so not testing them is not a problem.

[Lucas Werkmeister]

About IDE integration, for the record, I created jacoco/eclemma#80 a while ago.

I think it actually shouldn’t be too difficult… running with coverage really is as simple as adding the javaagent to the JVM options. The hard part was getting the report, and the IDE already has the class and source files, and EclEmma also obviously has the report generation built-in (no need to use Ant or Maven).