Compiler/optimiser question

[Gavin Baumanis]

Hi Everyone,

Firstly, I don't pretend to know anything about the internals of the Scala compiler, but recently came across an issue that I thought I was weird.

I added a library dependency to my Play Application in preparation for using it.

I got side-tracked, worked on a different issue, and then attempted to STAGE and DIST my application.

Stage stated that it finished successfully - but did not start the server.

and when runnign the sheel script created by DIST, I got an error message about no such method for main;

Exception in thread "main" java.lang.NoSuchMethodError: play.api.PlayConfig.getOptional(Ljava/lang/String;Lplay/api/ConfigLoader;)Lscala/Option;

via the Melbourne Scala Users group I was informed that the error message is seen, most likely as a result of incompatible libraries being used.

I removed the new dependency, and of course STAGE and DIST work correctly.

I have two queries, please;

1. Since I hadn't used the library at all, it was just in BUILD.SBT only. - why was it actually part of the application?

I assume the answer is pretty easy in that - it was set as a dependency and so it was loaded...

But if no method os ever called from the library - could it (should it) not be excluded from the build?

2, With the offending line of code included, my application still worked, without error in DEV mode, why did it not fail here, too?

or conversely - if it can somehow work in DEV mode - why didn;t it just work in PROD mode, too?

As always, thanks in advance.

-Gavin.

[Adriaan Moors]

Hi Gavin,

That's pretty puzzling indeed! It sounds like more of a build system issue, though (in play's case: sbt). There's little the compiler can do about this.

It's very hard to diagnose issues like this without a small reproduction, so I'm afraid I can't comment more on what exactly caused this.

cheers

adriaan

--
You received this message because you are subscribed to the Google Groups "scala-internals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Naftoli Gugenheim]

On Sat, Apr 30, 2016, 8:01 AM Gavin Baumanis <gavinb...@gmail.com> wrote:

Hi Everyone,

Firstly, I don't pretend to know anything about the internals of the Scala compiler, but recently came across an issue that I thought I was weird.

Why do you think this has anything to do with the compiler or optimizer?

I added a library dependency to my Play Application in preparation for using it.

Would you like to share what dependency it was?

I got side-tracked, worked on a different issue, and then attempted to STAGE and DIST my application.

Stage stated that it finished successfully - but did not start the server.

and when runnign the sheel script created by DIST, I got an error message about no such method for main;

Exception in thread "main" java.lang.NoSuchMethodError: play.api.PlayConfig.getOptional(Ljava/lang/String;Lplay/api/ConfigLoader;)Lscala/Option;

Sounds like it's trying to call a method in PlayConfig, but the JVM is saying that PlayConfig does not actually have such a method. So how did it compile? This can happen if code was compiled against one version of a class, but is run with a different version on the classpath.

via the Melbourne Scala Users group I was informed that the error message is seen, most likely as a result of incompatible libraries being used.

I removed the new dependency, and of course STAGE and DIST work correctly.

I have two queries, please;

1. Since I hadn't used the library at all, it was just in BUILD.SBT only. - why was it actually part of the application?

I assume the answer is pretty easy in that - it was set as a dependency and so it was loaded...

But if no method os ever called from the library - could it (should it) not be excluded from the build?

That would take a lot of work to compute. If you really want to eliminate anything not used, you could use something like proguard. (FTR scalajs does this automatically.)

2, With the offending line of code included, my application still worked, without error in DEV mode, why did it not fail here, too?

or conversely - if it can somehow work in DEV mode - why didn;t it just work in PROD mode, too?

Could it be that somehow both versions of PlayConfig are on the classpath, so it's selecting the first one, and the  classpath is in a different order?

As always, thanks in advance.

-Gavin.

--

[Gavin Baumanis]

​My Answers are below - Thanks.​

On 01 May 2016, at 11:55, Naftoli Gugenheim <nafto...@gmail.com> wrote:

On Sat, Apr 30, 2016, 8:01 AM Gavin Baumanis <gavinb...@gmail.com> wrote:

Hi Everyone,

Firstly, I don't pretend to know anything about the internals of the Scala compiler, but recently came across an issue that I thought I was weird.

Why do you think this has anything to do with the compiler or optimiser?

Well… really - whatever part of the compilation process from source code, to a finished zip/tar.gz it is responsible for determining what gets included and what doesn’t.

I added a library dependency to my Play Application in preparation for using it.

Would you like to share what dependency it was?

"com.mohiva" %% "play-silhouette" % "3.0.4"

I got side-tracked, worked on a different issue, and then attempted to STAGE and DIST my application.

Stage stated that it finished successfully - but did not start the server.

and when runnign the sheel script created by DIST, I got an error message about no such method for main;

Exception in thread "main" java.lang.NoSuchMethodError: play.api.PlayConfig.getOptional(Ljava/lang/String;Lplay/api/ConfigLoader;)Lscala/Option;

Sounds like it's trying to call a method in PlayConfig, but the JVM is saying that PlayConfig does not actually have such a method. So how did it compile? This can happen if code was compiled against one version of a class, but is run with a different version on the class path.

via the Melbourne Scala Users group I was informed that the error message is seen, most likely as a result of incompatible libraries being used.

I removed the new dependency, and of course STAGE and DIST work correctly.

I have two queries, please;

1. Since I hadn't used the library at all, it was just in BUILD.SBT only. - why was it actually part of the application?

I assume the answer is pretty easy in that - it was set as a dependency and so it was loaded...

But if no method os ever called from the library - could it (should it) not be excluded from the build?

That would take a lot of work to compute. If you really want to eliminate anything not used, you could use something like proguard. (FTR scalajs does this automatically.)

Just for the record - I am pretty ignorant to the compiler / optimiser / linker process in general - and completely ignorant to whatever it is that Scala uses and how.

I am completely fine with;

“You" included it in your build file as a dependency - so it got included…

And because it had a dependency on a previous version of netty - well - you get what you get what deserve….

(Well maybe nothing so harsh…. but since I am talking about myself… :) )

I suppose a better set of questions might have been;

* Here is what I am seeing….

* I would have thought that if a method of a library was never called upon, it would have been excluded from the build.

* Can someone share an overview of the Scala build process, please.

Additionally,

* Does it not make sense to exclude libraries that are not used - even if specified?

* is it technically possible to check for?

* is the work required for what I expected, simply too difficult or wasteful from a time management perspective?

Finally, 

Just for a recipe to test;

I used the Scala / Play seed template.

I altered the versions of SBT / Scala and Play to use the latest “public released” versions  0.13.11 / 2.11.8 / 2.5.3

I added in the Silhouette Authentication - again using the public release version, 3.0.4

completely omitted all other items from “my” build.

Made no alterations to code, just to the configured versions, as above.

The application still runs fine and dandy in Dev mode.

And still fails to run when using DIST or STAGE

You can view the code, here;

https://gist.github.com/gavinbaumanis

[Naftoli Gugenheim]

On Sun, May 1, 2016 at 12:19 AM Gavin Baumanis <gavinb...@gmail.com> wrote:

​My Answers are below - Thanks.​

On 01 May 2016, at 11:55, Naftoli Gugenheim <nafto...@gmail.com> wrote:

On Sat, Apr 30, 2016, 8:01 AM Gavin Baumanis <gavinb...@gmail.com> wrote:

Hi Everyone,

Firstly, I don't pretend to know anything about the internals of the Scala compiler, but recently came across an issue that I thought I was weird.

Why do you think this has anything to do with the compiler or optimiser?

Well… really - whatever part of the compilation process from source code, to a finished zip/tar.gz it is responsible for determining what gets included and what doesn’t.

I added a library dependency to my Play Application in preparation for using it.

Would you like to share what dependency it was?

"com.mohiva" %% "play-silhouette" % "3.0.4"

http://mvnrepository.com/artifact/com.mohiva/play-silhouette_2.11/3.0.4 shows that it depends on Play 2.4.2

 

I got side-tracked, worked on a different issue, and then attempted to STAGE and DIST my application.

Stage stated that it finished successfully - but did not start the server.

and when runnign the sheel script created by DIST, I got an error message about no such method for main;

Exception in thread "main" java.lang.NoSuchMethodError: play.api.PlayConfig.getOptional(Ljava/lang/String;Lplay/api/ConfigLoader;)Lscala/Option;

Sounds like it's trying to call a method in PlayConfig, but the JVM is saying that PlayConfig does not actually have such a method. So how did it compile? This can happen if code was compiled against one version of a class, but is run with a different version on the class path.

via the Melbourne Scala Users group I was informed that the error message is seen, most likely as a result of incompatible libraries being used.

I removed the new dependency, and of course STAGE and DIST work correctly.

I have two queries, please;

1. Since I hadn't used the library at all, it was just in BUILD.SBT only. - why was it actually part of the application?

I assume the answer is pretty easy in that - it was set as a dependency and so it was loaded...

But if no method os ever called from the library - could it (should it) not be excluded from the build?

That would take a lot of work to compute. If you really want to eliminate anything not used, you could use something like proguard. (FTR scalajs does this automatically.)

Just for the record - I am pretty ignorant to the compiler / optimiser / linker process in general - and completely ignorant to whatever it is that Scala uses and how.

I am completely fine with;

“You" included it in your build file as a dependency - so it got included…

And because it had a dependency on a previous version of netty - well - you get what you get what deserve….

(Well maybe nothing so harsh…. but since I am talking about myself… :) )

I suppose a better set of questions might have been;

* Here is what I am seeing….

* I would have thought that if a method of a library was never called upon, it would have been excluded from the build.

As I said, that is very computationally expensive and complicated to implement, so you need to explicitly use a tool to do that

 

* Can someone share an overview of the Scala build process, please.

I guess you mean, an overview of SBT?

I would start with https://jazzy.id.au/2015/03/03/sbt-task-engine.html and http://www.scala-sbt.org/0.13/docs/Getting-Started.html

 

Additionally,

* Does it not make sense to exclude libraries that are not used - even if specified?

* is it technically possible to check for?

* is the work required for what I expected, simply too difficult or wasteful from a time management perspective?

Finally, 

Just for a recipe to test;

I used the Scala / Play seed template.

I altered the versions of SBT / Scala and Play to use the latest “public released” versions  0.13.11 / 2.11.8 / 2.5.3

I added in the Silhouette Authentication - again using the public release version, 3.0.4

Indeed, as above, you seem to be pulling in conflicting Play versions. I don't know why one isn't evicted from the classpath though. You should discuss that on the play mailing list.

 

completely omitted all other items from “my” build.

Made no alterations to code, just to the configured versions, as above.

The application still runs fine and dandy in Dev mode.

And still fails to run when using DIST or STAGE

You can view the code, here;

https://gist.github.com/gavinbaumanis