scalac compilation requires transitive dependencies on classpath at compile-time whereas javac does not

[Chris Marshall]

If I have a Java library logging.jar with a class in it like this

class Log {

  public void log(String s) { System.out.println(s); }

}

If I then write a library businessy.jar which depends on logging.jar and which contains this class:

abstract class Businessy {

  protected Log log = new Log();

  public abstract void makeSomeMoney();

}

Then, if I now write some Java code like so:

class PetStore extends Businessy {

  public void makeSomeMoney() {  /* sell pets */ }

}

Then javac will let me compile this code with *only* businessy.jar on my classpath [1]. However, if I write this scala code:

class PetStore extends Businessy {

  def makeSomeMoney() {  /* sell pets */ }

}

Then scalac will fail to compile it unless I *also* include logging.jar. That is, I have to know about the businessy library's transitive dependencies at compile time. I can't find any information about this requirement in the language spec, or any questions about it on the mailing lists (other than those assuming it as a fait accompli). So here are a few questions:

1. Why does scalac require these transitive deps to be present at compile time? 
   

Actually that's just one question.

Chris

[1] - it will only fail to compile if I try and use Log in some way

[Grzegorz Kossakowski]

Hi Chris,

 

Not all transitive are necessarily required. However, in this case you are inheriting from a class that has a protected field. It's visible to your subclass so it's very likely that some piece of the code in scalac will try to load it's type (e.g. override checks) even if it's not strictly necessary. This behavior is not speced because it's an implementation detail of the compiler.

--
Grzegorz Kossakowski

Scalac hacker at Typesafe

twitter: @gkossakowski

github: @gkossakowski

[Nils Kilden-Pedersen]

If I recall correctly, it's because Scala's access modifiers are different than Java's, and the Scala modifiers are embedded in the byte code. And the compiler can't tell if the class is from javac or scalac, so it pessimistically needs the class byte code.

 

--
Grzegorz Kossakowski

Scalac hacker at Typesafe

twitter: @gkossakowski

github: @gkossakowski

--
You received this message because you are subscribed to the Google Groups "scala-language" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-languag...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Chris Marshall]

Hi Grzegorz -

If this is true then it's a bad decision because my project must explicitly declare logging.jar as a dependency. For example, suppose the businessy JAR changes its logging to use Log4J. But I still tell my clients (incorrectly) that they depend on logging.jar (because why would I remove the dependency?). This sort of thing leads to JAR hell!

Chris

--
You received this message because you are subscribed to a topic in the Google Groups "scala-language" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/scala-language/0B678ZjZQAw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to scala-languag...@googlegroups.com.

[Jason Zaugg]

Could you make the logger private, rather than protected? That will avoid this problem.

-jason

--

You received this message because you are subscribed to the Google Groups "scala-language" group.

To unsubscribe from this group and stop receiving emails from it, send an email to scala-languag...@googlegroups.com.

[Chris Marshall]

Jason - the point is a more general one about how scalac and javac handle transitive deps that are not used. I mean - if my PetStore code never tries to reference the log field, why does it matter what visibility it has?

In javac, even adding this to PetStore means that the logging JAR becomes necessary

class PetStore extends Businessy {

  public void makeSomeMoney() {  System.out.println(log); }

}

Chris

[Jason Zaugg]

On Fri, Dec 6, 2013 at 2:49 PM, Chris Marshall <oxbow...@gmail.com> wrote:

Jason - the point is a more general one about how scalac and javac handle transitive deps that are not used. I mean - if my PetStore code never tries to reference the log field, why does it matter what visibility it has?

In javac, even adding this to PetStore means that the logging JAR becomes necessary

class PetStore extends Businessy {

  public void makeSomeMoney() {  System.out.println(log); }

}

I test a cut down version of with your scenario (details below.)

It works in 2.11.0-M4, specifically since I fixed SI-7439.

-jason

% tail sandbox/{C.scala,J.java,S.scala}

==> sandbox/C.scala <==

class C

==> sandbox/J.java <==

public class J {

protected C c = new C();

}

==> sandbox/S.scala <==

class S extends J {

  toString

}

% sandbox/C.scala && javac -d . -classpath .:build/pack/lib/scala-library.jar sandbox/J.java && qbin/scalac sandbox/S.scala && rm C.class && scalac-hash v2.11.0-M4~35 sandbox/S.scala

warning: Class C not found - continuing with a stub.

one warning found

% sandbox/C.scala && javac -d . -classpath .:build/pack/lib/scala-library.jar sandbox/J.java && qbin/scalac sandbox/S.scala && rm C.class && scalac-hash v2.11.0-M4~36 sandbox/S.scala

warning: Class C not found - continuing with a stub.

error: error while loading J, class file './J.class' is broken

(class java.lang.NullPointerException/null)

sandbox/S.scala:1: error: J does not have a constructor

class S extends J {

                ^

one warning found

two errors found

[Jason Zaugg]

On Fri, Dec 6, 2013 at 3:28 PM, Jason Zaugg <jza...@gmail.com> wrote:

It works in 2.11.0-M4, specifically since I fixed SI-7439.

I'm backporting it to 2.10.4, too.

https://github.com/scala/scala/pull/3230

-jason

[Chris Marshall]

That's great - I should have specified that I was using 2.10.3 and not the latest milestone

Chris

--

[Som Snytt]

The latest Scala fashion tee reads:

"Retronym has a PR for that."

--

You received this message because you are subscribed to the Google Groups "scala-language" group.

To unsubscribe from this group and stop receiving emails from it, send an email to scala-languag...@googlegroups.com.

[P. Oscar Boykin]

I am working with bazel and scala:

https://github.com/bazelbuild/rules_scala

one thing bazel likes to do is separate compile and runtime dependencies. This is so we can better see if a target needs to be recompiled: if none of the compile time deps change their API, you don't need to recompile.

Unfortunately, when trying to keep the deps tight we get a lot of warnings like:

warning: Class org.apache.commons.logging.Log not found - continuing with a stub.

My questions:

1) under what circumstances is this warning of an issue that will manifest as a failure at runtime *assuming at runtime that jar IS on the classpath*?

2) Is there a way to silence this warning safely without just addind more (unneeded, it seems) items to the classpath?

3) What could be done to better support build tools that try to keep compile dependencies as a subset of runtime dependencies that are actually needed by the source code in question?

Thanks!

[Jason Zaugg]

What Scala version(s) are you testing with? Robustness to missing dependencies fluctuated somewhat through 2.10 and 2.11 releases.

Could you boil this down to a test case like this one? I’ll then be better able to answer your questions.

% cat sandbox/test.scala
class A

class B {
  def foo(a: A) = a
  def bar = 42
}

% (export V=2.11.7; ~/scala/$V/bin/scalac sandbox/test.scala; rm A.class; ~/scala/$V/bin/scala -nc -e 'new B().bar')

% (export V=2.11.7; ~/scala/$V/bin/scalac sandbox/test.scala; rm A.class; ~/scala/$V/bin/scala -nc -e 'new B().foo(null)')
error: missing or invalid dependency detected while loading class file 'B.class'.
Could not access type A in package <empty>,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'B.class' was compiled against an incompatible version of <empty>.
/code/scala on topic/nuke-impl-classes-2*

As you can see from my test, the intention is to create a stub symbol for the type of the parameter a when we read the B.class, and defer any error until typechecking can’t proceed without knowing more about that type.

Regards,

-jason

​

[P. Oscar Boykin]

I am using 2.11.7

It's a bit hard to boil down since I am seeing it when depending on Hadoop code which has a very deep dependency graph.

I'm instantiating GenericOptionsParser:

https://hadoop.apache.org/docs/r1.0.4/api/org/apache/hadoop/util/GenericOptionsParser.html

and Configuration:

https://hadoop.apache.org/docs/r2.6.4/api/org/apache/hadoop/conf/Configuration.html

and it is complaining about:

07:09:29 warning: Class org.apache.commons.logging.Log not found - continuing with a stub.
07:09:29 one warning found

which will be added to the runtime classpath, but is indeed not on the compile time classpath.

Is that enough to clarify the question some?

So, I think we are dealing with cases where the classes in question never appear explicitly in the code, or even in a superclass of any new classes, but are used by some of the classes instantiated by the code.

This code is not OSS, and hadoop deps are pretty deep, so it is a bit of a pain to replicate it (which I'm hoping to avoid if the answer is that there is little that can be done).

[Jason Zaugg]

​

Are you compiling with -optimize at the time? I notice that under this mode, our ClassFileParser (that we use to read the signatures of Java originated classes) reads the signatures of private fields which would include private static Logger LOG = ....

Looks like the same problem (right down to Hadoop) was reported in the comments of a kindred bug.

We should definitely demote that warning if it is just on a a private field. Please raise a ticket and I’ll improve this.

Here's the test case: https://gist.github.com/64fe18c74eac9d49092e

In the meantime, you can safely ignore the warning. It will turn into an error if it really prevents the compiler from making a decision later on. We don't currently have a means to disable the warning, sort of customizing the reporter in the compiler, which is a bit of a pain.

-jason

--

You received this message because you are subscribed to the Google Groups "scala-language" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-languag...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[P. Oscar Boykin]

Thanks so much, Jason.

Added issue:

https://issues.scala-lang.org/browse/SI-9673

I am not using -optimize, or indeed any flag to compile.  You can see the invocation here:

https://github.com/bazelbuild/rules_scala/blob/master/scala/scala.bzl#L78

I am using that with no `scala_opts` set.