SCALAC.MIXIN and SCALAC.CASE filters for JaCoCo Gradle plugin

[Sergey Zhemzhitsky]

Hello there,

I'd like to return to the discussion of the following threads:

- SCALAC.MIXIN and SCALAC.CASE filters for JaCoCo Gradle plugin

- using SCALAC.CASE (or other filter)

It is told there that even maven plugin does not support these filters, and I'd like to understand what maven plugin is meant there?

For example, here is the one that seems to support SCALAC.MIXIN and SCALAC.CASE filters and these maven plugin is mentioned in the documentation too.

SBT javoco plugin seems to support the necessary filters and even more.

Are there any plans to add support of the following filters to the Gradle too? It would be really-really great.

[Henrik Horneber]

Hi,,

As far as I know, both the scala plugin and the gradle plugin are provided by third parties. 

--
You received this message because you are subscribed to the Google Groups "JaCoCo and EclEmma Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jacoco+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/f09b37f3-7495-4fb9-9541-57a521601231%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

"I find many of the machines of violence very attractive. Tanks, airplanes, warships, especially aircraft carriers."
- 14th Dalai Lama

[Evgeny Mandrikov]

Hi,

On Wednesday, February 13, 2019 at 3:23:09 PM UTC+1, Sergey Zhemzhitsky wrote:

Hello there,

I'd like to return to the discussion of the following threads:

- SCALAC.MIXIN and SCALAC.CASE filters for JaCoCo Gradle plugin

- using SCALAC.CASE (or other filter)

It is told there that even maven plugin does not support these filters, and I'd like to understand what maven plugin is meant there?

jacoco-maven-plugin which is developed as part of JaCoCo project ( https://www.jacoco.org/jacoco/ , https://github.com/jacoco/ ) - see https://www.jacoco.org/jacoco/trunk/doc/integrations.html

 

For example, here is the one that seems to support SCALAC.MIXIN and SCALAC.CASE filters 

This one is third-party.

 

and these maven plugin is mentioned in the documentation too.

This page is not official JaCoCo documentation, but wiki with all the consequences such as

this mention was not done by JaCoCo developers ( currently Marc Hoffmann and me ),

but by https://github.com/timezra when wiki was editable by everyone.

 

SBT javoco plugin seems to support the necessary filters and even more.

This one is also third-party.

 

Are there any plans to add support of the following filters to the Gradle too? It would be really-really great.

 

Gradle Plugin is also third-party.

We don't develop filters specifically in jacoco-maven-plugin,

but in JaCoCo Core Library ( https://github.com/jacoco/jacoco/tree/v0.8.3/org.jacoco.core/src/org/jacoco/core/internal/analysis/filter ),

which is used by jacoco-maven-plugin and other integrations.

Gradle Plugin for example simply executes JaCoCo Ant Tasks developed by us,

and this is exactly how it gets all filters that were recently implemented ( https://www.jacoco.org/jacoco/trunk/doc/changes.html ) - see also https://groups.google.com/d/msg/jacoco/aMi1SkcfLZc/_UWgrC9xAQAJ

We don't have short-term plans to work on filters for Scala, but we noted your interest.

Hope this clarifies.

Regards,

Evgeny

[Sergey Zhemzhitsky]

Hi Evgeny,

Thanks a lot for clarification.

Would you mind if I add the corresponding pull request to support scala filters then?

Kind Regards,

Sergey

[Evgeny Mandrikov]

On Thursday, February 14, 2019 at 5:48:41 PM UTC+1, Sergey Zhemzhitsky wrote:

Thanks a lot for clarification.

BTW even if nice to see interest in JaCoCo for Scala, nevertheless IMO important to note/ask about alternatives - e.g. why not https://github.com/scoverage which seems to provide Gradle plugin?

 

Would you mind if I add the corresponding pull request to support scala filters then?

In any case of course you can open pull request!

However please be aware that

• filters require clear detailed description and careful testing (unit tests and so called "validation tests" that use real compiler, etc) - please have a look at tickets linked from changelog entries about filtering
• our knowledge of Scala is very limited ;)
  
• as we don't have short-term plans to work on filters for Scala and taking into account previous point, we can't guarantee that PR will be immediately reviewed/merged
  

So maybe you can start from description here? And IMO the most interesting question - is there reliable unambiguous way to distinguish bytecode which can be produced by Scala compiler from bytecode which can be produced by Java/Kotlin/Groovy? e.g. Kotlin annotates its classes by "kotlin/Metadata", recent Groovy versions annotate methods by "Generated" - see similar discussions about Kotlin at https://github.com/jacoco/jacoco/issues/552#issuecomment-392611472 and about Groovy at https://github.com/jacoco/jacoco/pull/733#issuecomment-416176217

[Sergey Zhemzhitsky]

Hi Evgeny,

> why not https://github.com/scoverage which seems to provide Gradle plugin?

there are my few points why I'd prefer jacoco to other code coverage tools

- bytecode instrumentation at runtime (no need to double compile as well as no risk of deploying instrumented code automatically)
- the same tool for each language built on top of jvm (my team currently uses java, scala and groovy)
- the same analysis and reporting tools (i.e. no need to configure build pipelines somehow differently across different languages)

> is there reliable unambiguous way to distinguish bytecode which can be produced by Scala compiler from bytecode which can be produced by Java/Kotlin/Groovy?

- modern scala annotates top-level classes with either "scala/reflect/ScalaSignature" or "scala/reflect/ScalaLongSignature" and adds either "ScalaSig" or "Scala" class attributes to the nested classes.
- here is the doc (https://www.scala-lang.org/old/sid/10) that can be used as a reference and the corresponding sample that can be extended by all the scala-related filters to prevent affecting filters for other languages.

There are three options of adding scala support (and extension points as well) which come to my mind

- backporting filters from sbt-scala plugin (I've already done some basic steps)
- adding support of java spi to allow adding new filters as easy as just dropping a jar into the build classpath (example is available here, the drawback - it requires java 6)

- in case it's necessary to support java 5 as well, java spi sample can be easily replaced with ClassLoader.getResources(...)

What do you think?

Kind Regards,
Sergey

[Evgeny Mandrikov]

On Friday, February 15, 2019 at 1:11:56 PM UTC+1, Sergey Zhemzhitsky wrote:

> why not https://github.com/scoverage which seems to provide Gradle plugin?

there are my few points why I'd prefer jacoco to other code coverage tools

- bytecode instrumentation at runtime (no need to double compile as well as no risk of deploying instrumented code automatically)
- the same tool for each language built on top of jvm (my team currently uses java, scala and groovy)
- the same analysis and reporting tools (i.e. no need to configure build pipelines somehow differently across different languages)

Makes sense. Thanks for sharing.

Should be noted that one of the reasons to ask question about tools specialized for Scala - is that I heard multiple times that Scala code requires "better granularity at a level of expressions". One of the times was quite recently at FOSDEM after my presentation of filtering in JaCoCo (https://fosdem.org/2019/schedule/event/kotlin_code_coverage_bytecode/) from a guy working on Scala in EPFL. Can't promise anything, but could try to reach them back since located quite close. Or maybe you can provide us information about this?

 

> is there reliable unambiguous way to distinguish bytecode which can be produced by Scala compiler from bytecode which can be produced by Java/Kotlin/Groovy?

- modern scala annotates top-level classes with either "scala/reflect/ScalaSignature" or "scala/reflect/ScalaLongSignature" and adds either "ScalaSig" or "Scala" class attributes to the nested classes.
- here is the doc (https://www.scala-lang.org/old/sid/10) that can be used as a reference

This is very valuable information! Thanks.

 

There are three options of adding scala support (and extension points as well) which come to my mind

- adding support of java spi to allow adding new filters as easy as just dropping a jar into the build classpath (example is available here, the drawback - it requires java 6)

- in case it's necessary to support java 5 as well, java spi sample can be easily replaced with ClassLoader.getResources(...)

IMO this is complete change of the subject of discussion - before it was about addition of filters directly into JaCoCo.

Going back to initial subject

and the corresponding sample that can be extended by all the scala-related filters to prevent affecting filters for other languages.

- backporting filters from sbt-scala plugin (I've already done some basic steps)

Have you ever tried them on real projects? And connected question - do you know open-source projects written in Scala (preferably not small) that already use or for which we can try to use JaCoCo?

IMO filters in sbt-jacoco are extremely weak, e.g.

https://github.com/sbt/sbt-jacoco/blob/v3.1.0/src/main/scala/com/github/sbt/jacoco/filter/ScalaSyntheticMethod.scala

excludes not only methods generated by compiler in "case classes" but also hand-written one-liners in all classes - for example:

class Example(a: Int, b: Int) {

  override def hashCode(): Int = 31 * a + b

}

To me seems that one-liners are pretty common case in Scala.

Furthermore please note that Java compiler has option to switch-off generation of line numbers and so all filters so far implemented for Java compiler do not rely on line numbers in order to have consistent result in measurement of instructions coverage between these two cases. On the other hand Kotlin compiler has no such option, and that's why filters for Kotlin can rely on line numbers. And AFAIK Scala compiler has option to switch-off generation of line numbers.

Is there a way to extract information about generated methods from annotation?

Regards,

Evgeny

[Sergey Zhemzhitsky]

> I heard multiple times that Scala code requires "better granularity at a level of expressions"

Basically I believe that all these expressions will occur in the bytecode one way or another and the only issue is to distinguish them somehow.

> Have you ever tried them on real projects?

I'd like to use all the power of jacoco for scala analysis on my current project (~27K SLOC of scala) and would like to improve code coverage reports for instructions and branches (with lines coverage thankfully everything is ok)

> IMO filters in sbt-jacoco are extremely weak ... To me seems that one-liners are pretty common case in Scala.

Well, it seems so. Then those filters can be treated as just a starting point (having in mind these corner cases and testing them carefully).

> AFAIK Scala compiler has option to switch-off generation of line numbers.

That's true. But in that case only coverage of scala classes will be affected (imo even then coverage will be better than now and to my mind not so may people switches lines generation off on the daily basis)

> Is there a way to extract information about generated methods from annotation?

I believe not at the moment. I've already asked the corresponding question - https://github.com/scala/bug/issues/11407

[Evgeny Mandrikov]

On Tuesday, February 19, 2019 at 5:14:43 PM UTC+1, Sergey Zhemzhitsky wrote:

> Is there a way to extract information about generated methods from annotation?

I believe not at the moment. I've already asked the corresponding question - https://github.com/scala/bug/issues/11407

Excuse me for not explicit enough question and let me clarify - actually it was not about introduction of new annotation, but about already existing "ScalaSignature" annotation that you mentioned earlier and that contains some metadata in some format.

However if Scala compiler for methods will be generating annotation such as "scala.Generated" (in fact any annotation whose simple name contains "Generated"), then without any changes in JaCoCo, these methods already will be automatically excluded - see https://github.com/jacoco/jacoco/pull/822 

[Evgeny Mandrikov]

On Tuesday, February 19, 2019 at 5:14:43 PM UTC+1, Sergey Zhemzhitsky wrote:

> I heard multiple times that Scala code requires "better granularity at a level of expressions"

Basically I believe that all these expressions will occur in the bytecode one way or another and the only issue is to distinguish them somehow.

While all the expressions appear in the bytecode (not counting cases when for example ECJ performs dead code elimination) and so appear as part of instructions and branches counters in JaCoCo, mapping them to source (i.e./e.g. which exact part of one-line-expression was executed) without additional information from compiler (e.g. recall absence of column numbers in bytecode) is extremely hard task and I would say unsolvable in general case - as a rough analogy you can treat compiler as "hash function in black box" for source code. Consider following very basic case:

return !x

for which AFAIR javac generates exactly the same bytecode as for 

return !x ? 1 : 0

which happens to be exactly the same bytecode as generated by ECJ for

return x ? 0 : 1

This also demonstrates complexity of implementation of filters in JaCoCo.

Coverage tools that perform instrumentation at source level, able to provide better precision by addition of more probes (and increased overhead for performance) to identify parts of expressions - this is their way of "getting help from compiler".

And I guess that statement about "better granularity" comes from the fact that in Scala long one-liner expressions are more frequent language construction than in Java. But maybe this is completely unrelated story if you satisfied with lines counter in JaCoCo.

 

> Have you ever tried them on real projects?

I'd like to use all the power of jacoco for scala analysis on my current project (~27K SLOC of scala) and would like to improve code coverage reports for instructions and branches (with lines coverage thankfully everything is ok)

> IMO filters in sbt-jacoco are extremely weak ... To me seems that one-liners are pretty common case in Scala.

Well, it seems so. Then those filters can be treated as just a starting point (having in mind these corner cases and testing them carefully).

> AFAIK Scala compiler has option to switch-off generation of line numbers.

That's true. But in that case only coverage of scala classes will be affected (imo even then coverage will be better than now and to my mind not so may people switches lines generation off on the daily basis)

Feel free to use these starting point(s) to experiment/test/validate implementation of filters for Scala using your custom build of JaCoCo for your codebase. And we'd be glad to hear about results/observations.

Good luck.

Evgeny