Re: Macros in trunk

[iulian dragos]

[moved to scala-internals]

On Apr 13, 2012, at 10:16 AM, Eugene Burmako wrote:

[INFO] uncaught exception during compilation: java.lang.UnsupportedOperationException
[ERROR] error: java.lang.UnsupportedOperationException: Scala reflection not available on this platform
[INFO] at scala.reflect.package$.mkMirror(package.scala:23)
[INFO] at scala.reflect.package$.mirror(package.scala:13)
[INFO] at scala.tools.nsc.Phases$TimingModel.<init>(Phases.scala:29)
[INFO] at scala.tools.nsc.Global.scala$tools$nsc$Global$$phaseTimings(Global.scala:726)
[INFO] at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1313)
[INFO] at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1278)
[INFO] at scala.tools.nsc.Global$Run.compileSources(Global.scala:1272)
[INFO] at scala.tools.nsc.Global$Run.compile(Global.scala:1405)
[INFO] at scala.tools.nsc.Driver.doCompile(Driver.scala:31)
[INFO] at scala.tools.nsc.Main$.doCompile(Main.scala:81)
[INFO] at scala.tools.nsc.Driver.process(Driver.scala:52)
[INFO] at scala.tools.nsc.Driver.main(Driver.scala:65)
[INFO] at scala.tools.nsc.Main.main(Main.scala)
What we see here indicates that scala-compiler needs to be on classpath.

 

Tags (aka manifests & classmanifests) now use reflection mirror (aka scala.reflect.mirror). Hence, the very startup of the compiler, which instantiates a class manifest, triggers initialization of the mirror. This initialization tries to load scala.reflect.runtime.Mirror class from the classpath and instantiate it. If it fails, you see the error message above.

 

In 2.10.0 final release, we will have a fallback mirror in case you don't have scala-compiler.jar on the classpath. But for now this is a mandatory requirement.

 

Let's rerun the build, and see what happens.

Why doing the same thing twice would yield different results?

The above build is using plain maven. So any build using maven with 2.10 is broken, if I understand correctly. I think that should be fixed before the final release.

At any rate, the (reconstructed) class path is printed a few lines above the exception, and it *does* contain the scala-compiler.jar (it's a huge class path spanning many lines):

https://jenkins.scala-ide.org:8496/jenkins/view/Scala%20IDE%20Helium-2.10(trunk)/job/scala-ide-nightly-master-trunk/274/console

cheers,
iulian

--
« Je déteste la montagne, ça cache le paysage »
Alphonse Allais

[Eugene Burmako]

Yeah, but it uses scala.compiler-2.10.0.v20120412-164128-6680522ccb.jar/lib/scala-compiler.jar, whereas the first build that featured macros appeared today (13.04) at night.

I meant "[let's fix the classpath and] rerun the build".

[iulian dragos]

On Fri, Apr 13, 2012 at 4:00 PM, Eugene Burmako <eugene....@epfl.ch> wrote:

Yeah, but it uses scala.compiler-2.10.0.v20120412-164128-6680522ccb.jar/lib/scala-compiler.jar, whereas the first build that featured macros appeared today (13.04) at night.

I meant "[let's fix the classpath and] rerun the build".

Thanks, I'm not that good reading between the lines.

iulian

[Mirco Dotta]

That one is the OSGi version of the latest 2.10-SNAPSHOT

If you open the compiler.properties:

#Fri, 13 Apr 2012 02:13:00 +0200

version.number=2.10.0-20120412-164128-6680522ccb

maven.version.number=2.10.0-SNAPSHOT

osgi.version.number=2.10.0.v20120412-164128-6680522ccb

copyright.string=Copyright 2002-2011, LAMP/EPFL

[iulian dragos]

On Fri, Apr 13, 2012 at 4:10 PM, Mirco Dotta <mirco...@typesafe.com> wrote:

That one is the OSGi version of the latest 2.10-SNAPSHOT

If you open the compiler.properties:

#Fri, 13 Apr 2012 02:13:00 +0200

version.number=2.10.0-20120412-164128-6680522ccb

Presumably because the timestamp comes from the git commit date, not the build date.

iulian

[Josh Suereth]

exactly.

[Eugene Burmako]

All right, I get it.

Since it's OSGi, maybe it has a different notion of reflection? No idea how it works though. Could you, please, explain?

What Scala reflection needs is a successful call to Class.forName("scala.reflect.runtime.package$", true, getClass.getClassLoader) invoked from ReflectionUtils. What could mess with that?

[Eugene Burmako]

http://billhiggins.us/blog/2011/02/24/using-reflection-in-osgi/

This link suggests that we need to update manifests in both scala library and scala compiler to allow classes from library to reflectively load classes from scala compiler. Could you please check this out?

[iulian dragos]

On Fri, Apr 13, 2012 at 4:32 PM, Eugene Burmako <eugene....@epfl.ch> wrote:

All right, I get it.

Since it's OSGi, maybe it has a different notion of reflection? No idea how it works though. Could you, please, explain?

What Scala reflection needs is a successful call to Class.forName("scala.reflect.runtime.package$", true, getClass.getClassLoader) invoked from ReflectionUtils. What could mess with that?

Note that these exceptions happen during the *build* of the IDE. There's no OSGi there, it's a plain maven build. Josh, do you know if maven could interfere with this?

iulian

[iulian dragos]

On Fri, Apr 13, 2012 at 4:42 PM, Eugene Burmako <eugene....@epfl.ch> wrote:

http://billhiggins.us/blog/2011/02/24/using-reflection-in-osgi/

This link suggests that we need to update manifests in both scala library and scala compiler to allow classes from library to reflectively load classes from scala compiler. Could you please check this out?

We're not that far yet (that would happen when the IDE *runs*), but these manifests are built automatically (that's why we have to repackage the scala library and compiler when distributing the IDE).

iulian

[Luc Bourlier]

I tried with a simple maven project, the maven scala compile task just doesn't start for 2.10.0-SNAPSHOT.

I attached the pom. The error is the same as for Scala IDE. 

Luc

--
Luc Bourlier - Scala IDE contributor

TypeSafe - The software stack for applications that scale

[Josh Suereth]

Not sure, but I can dig into it later this weekend.   I'm a bit swamped right now.

[Mirco Dotta]

As a side note, does it make sense to release a 2.10-M3 if maven-scala-plugin is broken? Does it work with sbt?

I know M3 should go live before ScalaDays, but I wonder if anyone will be able to use it at all (unless Josh has an incantation to fix this right away)

[iulian dragos]

Yes, I thought we intended to have milestones that keep a certain number of base projects working: Sbt, Scala IDE. Not sure about Sbt, but maven is broken (I don't remember any point in time when we managed to break maven, by the way), and the IDE can't even build on M3. It's a show-stopper, IMO.

iulian

[Josh Suereth]

+1.  I won't be able to look until monday-ish, assuming I have wifi on the flight to report back what I'm doing !

[Eugene Burmako]

I will look into that after I finish with urgent macro stuff that still needs to be implemented.

Could you please prepare a non-scary test case for me? Also, who can I bother if I have any questions today in the evening / tomorrow?

[iulian dragos]

On Sat, Apr 14, 2012 at 2:03 PM, Eugene Burmako <eugene....@epfl.ch> wrote:

I will look into that after I finish with urgent macro stuff that still needs to be implemented.

Could you please prepare a non-scary test case for me?

Maven is by definition scary :) I think Luc's pom-file is a good test case, though it has some non-required tags in there. I don't think you'd need to change anything though, so I zipped up a directory containing a similar pom file, and a 'hello world' source file. Just unzip and run `mvn compile' in that directory. You should see it fail.

If you are completely new to maven, I recommend this excellent tutorial by Josh:

http://www.scala-lang.org/node/345

iulian

[Eugene Burmako]

reproduced, thanks. will look into it.

[Eugene Burmako]

Here's what I see:

[INFO] primordial classloader with boot classpath [c:\PROGRA~1\Java
\JDK16~1.0_2\jre\lib\resources.jar;c:\PROGRA~1\Java\JDK16~1.0_2\jre\lib
\rt.jar;c:\PROGRA~1\Java\JDK16~1.0_2\jre\lib\sunrsasign.jar;c:
\PROGRA~1\Java\JDK16~1.0_2\jre\lib\js
se.jar;c:\PROGRA~1\Java\JDK16~1.0_2\jre\lib\jce.jar;c:\PROGRA~1\Java
\JDK16~1.0_2\jre\lib\charsets.jar;c:\PROGRA~1\Java\JDK16~1.0_2\jre\lib
\modules\jdk.boot.jar;c:\PROGRA~1\Java\JDK16~1.0_2\jre\classes;C:\Users
\xeno.by\.m2\repository\org\
skife\com\typesafe\config\typesafe-config\0.3.0\typesafe-
config-0.3.0.jar;C:\Users\xeno.by\.m2\repository\org\scala-lang\scala-
library\2.10.0-SNAPSHOT\scala-library-2.10.0-SNAPSHOT.jar]
[INFO] loading class named: scala.reflect.runtime.package$

Compiler is not on the classpath, and, I believe, this is a
responsibility of maven-scala plugin. Right?

On Apr 14, 4:58 pm, iulian dragos <jagua...@gmail.com> wrote:

> On Sat, Apr 14, 2012 at 2:03 PM, Eugene Burmako <eugene.burm...@epfl.ch>wrote:
>
> > I will look into that after I finish with urgent macro stuff that still
> > needs to be implemented.
>
> > Could you please prepare a non-scary test case for me?
>
> Maven is by definition scary :) I think Luc's pom-file is a good test case,
> though it has some non-required tags in there. I don't think you'd need to
> change anything though, so I zipped up a directory containing a similar pom
> file, and a 'hello world' source file. Just unzip and run `mvn compile' in
> that directory. You should see it fail.
>
> If you are completely new to maven, I recommend this excellent tutorial by
> Josh:
>
> http://www.scala-lang.org/node/345
>
> iulian
>
>
>
>
>
> > Also, who can I bother if I have any questions today in the evening /
> > tomorrow?
>

> > On 14 April 2012 13:58, Josh Suereth <joshua.suer...@gmail.com> wrote:
>
> >> +1.  I won't be able to look until monday-ish, assuming I have wifi on
> >> the flight to report back what I'm doing !
>

> >> On Sat, Apr 14, 2012 at 7:37 AM, iulian dragos <jagua...@gmail.com>wrote:
>
> >>> Yes, I thought we intended to have milestones that keep a certain number
> >>> of base projects working: Sbt, Scala IDE. Not sure about Sbt, but maven is
> >>> broken (I don't remember any point in time when we managed to break maven,
> >>> by the way), and the IDE can't even build on M3. It's a show-stopper, IMO.
>
> >>> iulian
>

> >>> On Fri, Apr 13, 2012 at 5:33 PM, Mirco Dotta <mirco.do...@typesafe.com>wrote:
>
> >>>> As a side note, does it make sense to release a 2.10-M3 if
> >>>> maven-scala-plugin is broken? Does it work with sbt?
>
> >>>> I know M3 should go live before ScalaDays, but I wonder if anyone will
> >>>> be able to use it at all (unless Josh has an incantation to fix this right
> >>>> away)
>
> >>>> On Apr 13, 2012, at 5:03 PM, Josh Suereth wrote:
>
> >>>> Not sure, but I can dig into it later this weekend.   I'm a bit swamped
> >>>> right now.
>

> >>>> On Fri, Apr 13, 2012 at 10:43 AM, iulian dragos <jagua...@gmail.com>wrote:
>
> >>>>> On Fri, Apr 13, 2012 at 4:32 PM, Eugene Burmako <
> >>>>> eugene.burm...@epfl.ch> wrote:
>
> >>>>>> All right, I get it.
>
> >>>>>> Since it's OSGi, maybe it has a different notion of reflection? No
> >>>>>> idea how it works though. Could you, please, explain?
>
> >>>>>> What Scala reflection needs is a successful call to
> >>>>>> Class.forName("scala.reflect.runtime.package$", true,
> >>>>>> getClass.getClassLoader) invoked from ReflectionUtils. What could mess with
> >>>>>> that?
>
> >>>>> Note that these exceptions happen during the *build* of the IDE.
> >>>>> There's no OSGi there, it's a plain maven build. Josh, do you know if maven
> >>>>> could interfere with this?
>
> >>>>> iulian
>

> >>>>>> On 13 April 2012 16:13, iulian dragos <jagua...@gmail.com> wrote:
>
> >>>>>>> On Fri, Apr 13, 2012 at 4:10 PM, Mirco Dotta <
> >>>>>>> mirco.do...@typesafe.com> wrote:
>
> >>>>>>>> That one is the OSGi version of the latest 2.10-SNAPSHOT
>
> >>>>>>>> If you open the compiler.properties:
>
> >>>>>>>> #Fri, 13 Apr 2012 02:13:00 +0200
>
> >>>>>>>> version.number=2.10.0-20120412-164128-6680522ccb
>
> >>>>>>> Presumably because the timestamp comes from the git commit date, not
> >>>>>>> the build date.
>
> >>>>>>> iulian
>
> >>>>>>>> maven.version.number=2.10.0-SNAPSHOT
> >>>>>>>> osgi.version.number=2.10.0.v20120412-164128-6680522ccb
> >>>>>>>> copyright.string=Copyright 2002-2011, LAMP/EPFL
>
> >>>>>>>> On Apr 13, 2012, at 4:00 PM, Eugene Burmako wrote:
>
> >>>>>>>> Yeah, but it uses

> >>>>>>>> scala.compiler-2.10.0.v20120412-164128-6680522ccb.jar/lib/scala-compiler.ja­r,

> >>>>>>>> whereas the first build that featured macros appeared today (13.04) at
> >>>>>>>> night.
>
> >>>>>>>> I meant "[let's fix the classpath and] rerun the build".
>

> >>>>>>>> On 13 April 2012 15:46, iulian dragos <jagua...@gmail.com> wrote:
>
> >>>>>>>>> [moved to scala-internals]
>
> >>>>>>>>> On Apr 13, 2012, at 10:16 AM, Eugene Burmako wrote:
>
> >>>>>>>>>> [INFO] uncaught exception during compilation:
> >>>>>>>>>> java.lang.UnsupportedOperationException
> >>>>>>>>>> [ERROR] error: java.lang.UnsupportedOperationException: Scala
> >>>>>>>>>> reflection not available on this platform
> >>>>>>>>>> [INFO] at scala.reflect.package$.mkMirror(package.scala:23)
> >>>>>>>>>> [INFO] at scala.reflect.package$.mirror(package.scala:13)
> >>>>>>>>>> [INFO] at
> >>>>>>>>>> scala.tools.nsc.Phases$TimingModel.<init>(Phases.scala:29)
> >>>>>>>>>> [INFO] at

> >>>>>>>>>> scala.tools.nsc.Global.scala$tools$nsc$Global$$phaseTimings(Global.scala:72­6)

> >>>>>>>>>https://jenkins.scala-ide.org:8496/jenkins/view/Scala%20IDE%20Helium-...

>
> >>>>>>>>> cheers,
> >>>>>>>>> iulian
>
> >>>>>>>>> --
> >>>>>>>>> « Je déteste la montagne, ça cache le paysage »
> >>>>>>>>> Alphonse Allais
>
> >>>>>>> --
> >>>>>>> « Je déteste la montagne, ça cache le paysage »
> >>>>>>> Alphonse Allais
>
> >>>>> --
> >>>>> « Je déteste la montagne, ça cache le paysage »
> >>>>> Alphonse Allais
>
> >>> --
> >>> « Je déteste la montagne, ça cache le paysage »
> >>> Alphonse Allais
>
> --
> « Je déteste la montagne, ça cache le paysage »
> Alphonse Allais
>

>  test-one.zip
> 8KViewDownload

[Eugene Burmako]

Josh, is there a way to configure maven-scala to include scala-
compiler.jar into library classpath? If no, is there an easy way to
patch and rebuild it?

[Eugene Burmako]

Okay, updated pom.xml/project/dependencies, and scala-compiler gets added, but it's still not in the primordial classloader. Hmmm

[Grzegorz Kossakowski]

On 14 April 2012 23:45, Eugene Burmako <eugene....@epfl.ch> wrote:

Here's what I see:

[INFO] primordial classloader with boot classpath [c:\PROGRA~1\Java
\JDK16~1.0_2\jre\lib\resources.jar;c:\PROGRA~1\Java\JDK16~1.0_2\jre\lib
\rt.jar;c:\PROGRA~1\Java\JDK16~1.0_2\jre\lib\sunrsasign.jar;c:
\PROGRA~1\Java\JDK16~1.0_2\jre\lib\js
se.jar;c:\PROGRA~1\Java\JDK16~1.0_2\jre\lib\jce.jar;c:\PROGRA~1\Java
\JDK16~1.0_2\jre\lib\charsets.jar;c:\PROGRA~1\Java\JDK16~1.0_2\jre\lib
\modules\jdk.boot.jar;c:\PROGRA~1\Java\JDK16~1.0_2\jre\classes;C:\Users
\xeno.by\.m2\repository\org\
skife\com\typesafe\config\typesafe-config\0.3.0\typesafe-
config-0.3.0.jar;C:\Users\xeno.by\.m2\repository\org\scala-lang\scala-
library\2.10.0-SNAPSHOT\scala-library-2.10.0-SNAPSHOT.jar]
[INFO] loading class named: scala.reflect.runtime.package$

Compiler is not on the classpath, and, I believe, this is a
responsibility of maven-scala plugin. Right?

It depends on what do you mean by "responsibility". Until now there was no need to have compiler on a classpath and I believe most tools didn't put it.

AFAIK, sbt doesn't put compiler on a classpath by default. It looks like M3 will break almost every non-toy Scala project on this planet :-/

I think you cannot avoid doing this so at least provide help to people. Is there any way to print big, fat message explaining why scala-compiler.jar is now needed on a classpath? Also, including clear instructions how to fix Maven and sbt-based projects is a must.

--
Grzegorz Kossakowski

[Grzegorz Kossakowski]

On 14 April 2012 23:54, Eugene Burmako <eugene....@epfl.ch> wrote:

Okay, updated pom.xml/project/dependencies, and scala-compiler gets added, but it's still not in the primordial classloader. Hmmm

Try plugin dependencies: http://maven.apache.org/pom.html#Plugins

--
Grzegorz Kossakowski

[Eugene Burmako]

scala-tools is down. is this normal and it will be up eventually, or I
should report this to someone?

On Apr 14, 11:56 pm, Grzegorz Kossakowski
<grzegorz.kossakow...@gmail.com> wrote:

[Eugene Burmako]

Yeah, I will surely print a message.

Speaking of SBT. Is there any up-to-date guide about how to set up SBT for M2? A few weeks ago I downloaded fresh sbt-launcher from Typesafe, but it failed to launch whining about missing scala/ScalaObject. I didn't have much time to look into that and put my SBT adventures on hold. Could you, please, help me out?

[Grzegorz Kossakowski]

On 15 April 2012 00:12, Eugene Burmako <eugene....@epfl.ch> wrote:

scala-tools is down. is this normal and it will be up eventually, or I
should report this to someone?

It's been moved. For releases use Maven central: http://repo1.maven.org/

For snapshots use: https://oss.sonatype.org/content/repositories/snapshots/

Scala nightly builds are deployed there, see https://scala-webapps.epfl.ch/jenkins/job/scala-nightly-publish/42/console

--
Grzegorz Kossakowski

[Grzegorz Kossakowski]

On 15 April 2012 00:17, Eugene Burmako <eugene....@epfl.ch> wrote:

Yeah, I will surely print a message.

Speaking of SBT. Is there any up-to-date guide about how to set up SBT for M2? A few weeks ago I downloaded fresh sbt-launcher from Typesafe, but it failed to launch whining about missing scala/ScalaObject. I didn't have much time to look into that and put my SBT adventures on hold. Could you, please, help me out?

Unfortunately I cannot help. :-(

I'm swamped with university project until tomorrow's night. On Monday I have a few meetings to attend..

--
Grzegorz Kossakowski

[Josh Suereth]

I can help with both Maven and SBT, when I'm not otherwise engaged.  So tomorrow + monday.

I need to know what classloaders you're looking at, and why.

As for SBT, you have to use the 0.12.0-M2 "release" for scala 2.10.

- Josh

[Josh Suereth]

Ok, I figured it out.   scala-library is loaded in the boot classloader by the maven scala plugin for speed when forking.  If you set <fork>false</fork> in the config, it suddenly works.   This is because scala.reflect.mirror is initialized using "getClass.getClassLoader" on *mirror* which is the bootclasspath.

The compiler should *fix* this for its usage by using its own classloader (most likely an outer loader from the library).  The current behavior is scary for OSGi usage.  Also, all the manifests used by the compiler will bomb (if the compiler is in a lower classloader) since the default mirror is used.  While SBT keeps pretty good classloader isolation, it's not quite OSGi crazy, so I don't think you've broken it.

This at the very *least* deserves a big warning when using Manifest that you need to bundle the compiler.  Not many do this now.

Now for some action items:

(1) Is there *any way* we can have manifests work without requiring the compiler on the classpath?   I mean, a good thing about this is we can go refactor a ton of code to use ArrayTag or ClassTag instead of Manifest (which should help us all out a lot).   However, this seems pretty bad to me, especially given the classloader fragility.   Any dependence of the library on the compiler classloader is fraught with peril unless we *explicitly* pass in classloaders and such in general usage.  Manifests are not the place to do this, IMHO.

(2) The default scala mirror terrifies me.   I'm glad we can construct new mirrors, but we need lots of warnings around the default.  I'm also nervous about anything which relies on it for accuracy.    This class is perfect for scripting and simple builds, but falls apart in any big complicated system that uses partitioned classloaders.  This is the usual case for build tools and OSGi.

(3) Should the maven-scala-plugin change?  If you force us to, we can, but I think it's a poor decision.  We're but one tool that will break by the classloader magikz.   If you can fix things to run in this simple scenario, hopefully we can hit OSGi without issues.

Hope that helps!

- Josh

[Paul Phillips]

There's a ridiculous amount of stuff going on which I have no chance
to adequately address in the time available to me today; however:

- the compiler should start using its own classloader regardless of
anything else
- using manifests really cannot require the compiler, that's an
unacceptable regression and it shouldn't even be considered (for
release anyway, we will have to live with however things are for M3)
- I have a lot of classloader improvement done and waiting for me to
integrate it, but it will probably explode my machine under merge
conflicts