Classpath validator, maven and cans of worms

[Matthew Farwell]

Hi,

I'd like to start a debate about the new classpath validator. I saw it go past in the group, but didn't realise quite what the implications were. I saw that there was a discussion about this on scala-ide-user, but I wanted to specifically talk about what the behaviour of scala ide should be, so I'm starting another thread here.

So, some context: I upgraded to the latest version yesterday (good work all btw), and I started to run into problems.  I have a maven project (with m2eclipse) which uses 1) scalariform (0.1.1) (2.9.1) and 2) scalatest (1.6.1) (2.9.1). Both of these have transitive dependencies on scala-library 2.9.1. So now I get the 'You have two versions of the library on your classpath' error. I install m2eclipse-scala, which removes the direct dependency, but not the transitive ones (bug, probably), so I'm still getting the error from scala-ide. So I remove the (2.9.2) library from the build path, and I get a warning (version of library is different etc). Everything compiles when I do project clean. I change a class, I get an error when I save: 'Cannot find Scala Library on the classpath: Verify your build path' (again a bug). I have to do a project clean every time I change a file now.

So, In upgrading my scala-ide, I've had to install another plugin (small impact) and change my dependencies in my pom (potentially large impact, always a pain to do).

There are a number of points to make:

1) I cannot upgrade the versions of scalariform & scalatest yet, because 2.9.2 versions don't exist for those libraries. This is a general problem, not specific to those two libraries.

2) It is notoriously hard to cleanly remove a dependency from maven. In my case, I can add exclusions to all of the libraries which have transitive dependencies on the scala-library. But this also means that if I'm creating a main which is runnable with just a jvm then the scala-library won't get included in the dependencies.

3) mucking with poms is notoriously dark magic, not everyone (including me) understands exactly what scope means what. And not everything works in all cases (for instance deploying to Tomcat from within Eclipse deploys any jars with provided scope, even though it shouldn't do).

4) This problem exists with java, but isn't as 'in your face'. Mainly because the java rt.jar isn't a maven dependency, but comes with the jdk/jre. With java, you select the jdk you want to use as an execution environment: You can still get problems with usage of methods which don't exist on the target jvm because you've compiled it with a newer jvm with --source. I have been bitten by this before with java, hence the reason for the existence of http://maven.apache.org/plugins/maven-enforcer-plugin/usage.html which checks exactly that.

5) I could fix this by having a dependency in maven on the specific version which is used in the plugin in eclipse, but I actually can't find it, and it may break existing code

So there are a number of options:

1) Persuade all of the library writers to not include transitive dependencies on the scala-library. I don't believe this is possible or realistic, even though it makes life 'easier' for everyone.

2) The classpath validator could assume that the libraries are backwards compatible. So if I have a scala-ide which expects a scala library of 2.9.2 and I add a 2.9.1 afterwards in the classpath, this is acceptable. The other way around won't be. This would mean I have to rearrange my build path in eclipse to have the built-in scala library before the one in the maven dependency. Impact on the project? Not sure.

3) Downgrade the classpath validator to a warning.

4) Have an option which switches off the validator, with a popup when you do which says 'Are you sure you know what you're doing, etc?'. This could be done on a per-project basis.

Personally, I would prefer at least option 3 and in addition option 4, because I don't like warnings I can't configure away. I like my projects to have 0 warnings. I can do the required work if necessary.

Please don't take this email as a criticism, it isn't meant to be one. I think the classpath validator gives useful information, but as always, the real world (i.e. maven and dependency management) intervenes.

[iulian dragos]

On Fri, Nov 4, 2011 at 10:11 AM, Matthew Farwell <mat...@farwell.co.uk> wrote:

Hi,

Hi Matthew,

 

I'd like to start a debate about the new classpath validator. I saw it go past in the group, but didn't realise quite what the implications were. I saw that there was a discussion about this on scala-ide-user, but I wanted to specifically talk about what the behaviour of scala ide should be, so I'm starting another thread here.

Thanks a lot for taking the time to write this email. This is great feedback.

 

So, some context: I upgraded to the latest version yesterday (good work all btw), and I started to run into problems.  I have a maven project (with m2eclipse) which uses 1) scalariform (0.1.1) (2.9.1) and 2) scalatest (1.6.1) (2.9.1). Both of these have transitive dependencies on scala-library 2.9.1. So now I get the 'You have two versions of the library on your classpath' error. I install m2eclipse-scala, which removes the direct dependency, but not the transitive ones (bug, probably), so I'm still getting the error from scala-ide. So I remove the (2.9.2) library from the build path, and I get a warning (version of library is different etc). Everything compiles when I do project clean. I change a class, I get an error when I save: 'Cannot find Scala Library on the classpath: Verify your build path' (again a bug). I have to do a project clean every time I change a file now. 

I can confirm this behavior. It took me some time to set up everything (not a true maven user). I don't understand everything about maven and transitive dependencies, so we'd need some help. m2e adds a classpath container called 'Maven Dependencies', where it drops all dependent jars, right? Some of them are duplicated, because of transitive dependencies on scala-library? We should ignore duplicates if they have the same version (first step). Second is to allow forward (2.9.2-snapshot) compatibility. 

On the last point, I don't understand why m2e-scala doesn't fix the problem. If it 'hides' the scala-library from Maven, you should be left with just one (2.9.2). Is it because of transitive dependencies?

There's definitely a bug right now (not finding any scala-library on the classpath after removing the one from the plugin).

 

So, In upgrading my scala-ide, I've had to install another plugin (small impact) and change my dependencies in my pom (potentially large impact, always a pain to do).

There are a number of points to make:

1) I cannot upgrade the versions of scalariform & scalatest yet, because 2.9.2 versions don't exist for those libraries. This is a general problem, not specific to those two libraries. 

2) It is notoriously hard to cleanly remove a dependency from maven. In my case, I can add exclusions to all of the libraries which have transitive dependencies on the scala-library. But this also means that if I'm creating a main which is runnable with just a jvm then the scala-library won't get included in the dependencies.

Agreed on both points. The validator should be more lenient when the versions agree.

 

3) mucking with poms is notoriously dark magic, not everyone (including me) understands exactly what scope means what. And not everything works in all cases (for instance deploying to Tomcat from within Eclipse deploys any jars with provided scope, even though it shouldn't do).

4) This problem exists with java, but isn't as 'in your face'. Mainly because the java rt.jar isn't a maven dependency, but comes with the jdk/jre. With java, you select the jdk you want to use as an execution environment: You can still get problems with usage of methods which don't exist on the target jvm because you've compiled it with a newer jvm with --source. I have been bitten by this before with java, hence the reason for the existence of http://maven.apache.org/plugins/maven-enforcer-plugin/usage.html which checks exactly that.

5) I could fix this by having a dependency in maven on the specific version which is used in the plugin in eclipse, but I actually can't find it, and it may break existing code

So there are a number of options:

1) Persuade all of the library writers to not include transitive dependencies on the scala-library. I don't believe this is possible or realistic, even though it makes life 'easier' for everyone.

2) The classpath validator could assume that the libraries are backwards compatible. So if I have a scala-ide which expects a scala library of 2.9.2 and I add a 2.9.1 afterwards in the classpath, this is acceptable. The other way around won't be. This would mean I have to rearrange my build path in eclipse to have the built-in scala library before the one in the maven dependency. Impact on the project? Not sure.

I think this is safe.

 

3) Downgrade the classpath validator to a warning.

4) Have an option which switches off the validator, with a popup when you do which says 'Are you sure you know what you're doing, etc?'. This could be done on a per-project basis.

This is the number one problem for newcomers, and I would hate to give them an easy way to turn it off. Since they are the most likely to not understand what it does, they will just press the button, and hey! the error is gone (nevermind nothing else works). We just need to make it 'do the right thing' and not be annoying.

 

Personally, I would prefer at least option 3 and in addition option 4, because I don't like warnings I can't configure away. I like my projects to have 0 warnings. I can do the required work if necessary.

Please don't take this email as a criticism, it isn't meant to be one. I think the classpath validator gives useful information, but as always, the real world (i.e. maven and dependency management) intervenes.

It's most appreciated, thank you again for taking the time to describe this problem!

iulian

 

 

--
« Je déteste la montagne, ça cache le paysage »
Alphonse Allais

[Mirco Dotta]

Hi Matthew,

Thanks a lot for taking the time of writing this down, your feedback is *truly* appreciated.

Another user in the scala-ide-user mailing list just reported a similar issue, and he fixed that by
installing the latest version of m2eclipse plugin. Here is the link to the discussion

https://groups.google.com/group/scala-ide-user/browse_thread/thread/4ef0597a0190dae4?hl=en

I wonder if you are already on the latest version of m2eclipse and, if not, if you could check if the
problem goes really away when using the latest m2eclipse. It would be definitely useful to know.

Cheers,
Mirco

[iulian dragos]

Hi all,

I propose to continue here the discussion related to the classpath validator. I see the following hard requirements:

- the validator should work for maven projects, both using m2e and without (generating eclipse project files on the command line)

- it should prevent one from getting into a broken state caused by wrong/missing scala library (the presentation compiler being extremely sensitive to that)

I propose the following changes:

- it should allow several scala-library.jar on the classpath as long as they are exactly the same version

- should issue a warning if the libraries differ in the micro part (2.9.1 vs. 2.9.0), and the same between the compiler and the library

- should issue an error when they differ in the minor part (2.8 vs. 2.9)

Please let us know if this is reasonable, or if there are use-cases that wouldn't work in this way.

I want to emphasize that we need to get this right, but there really shouldn't be a way to disable it. The IDE (presentation compiler) is extremely sensitive to the scala-library, and if they don't match, nothing will work: no hyperlinking, hundreds of spurious errors (red screen of death), etc. Of course, the validator should not be in the way, but rather a helpful assistant when things go wrong. Maven users are particularly prone to troubles, since maven manages the scala-library dependency and there is no way to ensure the version in your IDE matches your maven project. The IDE needs to protect itself from misconfiguration, and give meaningful errors. I see this as the number one issue for newcomers.

iulian

[Luc Bourlier]

Hi all,

I finally had time to have a look at this issue.

The creation of the classpath validator is uncovering a limitation in the sbt builder.

It is looking for a scala library jar named 'scala-library.jar', only for that and complains when it doesn't see it.

When using maven, the scala library jar has a name containing the version number, so the sbt builder doesn't find it.

Before the classpath validator, sbt was getting multiple scala library jar, and picking the one it likes. The classpath validator 'forces' only one scala library jar, and the sbt builder can't find any it likes.

See https://scala-ide-portfolio.assembla.com/spaces/scala-ide/tickets/1000728 

As it was stated, it is also possible to get in the state where maven provides to different but compatible version of the scala library jar.

I created a change in the classpath validator to allow multiple scala library jars with compatible version: https://github.com/scala-ide/scala-ide/pull/36

++,

Luc