emulation strategy

70 views
Skip to first unread message

Stephen Haberman

unread,
Jul 16, 2011, 8:11:45 PM7/16/11
to scal...@googlegroups.com
Hi,

So, I figured out more how GWT emulation works, in terms of solving the
TypeOracle "unresolved class" errors we currently get for all of the
pruned parallel collection types.

The trick is the bytecode. The TypeOracle assumes, while building
itself, that any class/method signatures in the bytecode will only refer
to types that are within the types that came from the resource oracle.

So, if our List.jribble doesn't refer to Parallel (because we manually
deleted it from the AST), but the List.class bytecode has a class/method
signature that does refer to Parallel, the bytecode is treated as
authoritative, so an error is logged.

But the bytecode--it turns out it is *not* the normal bytecode that
the user has on their regular JVM classpath. (Although that is the way
I naively implemented it for jribble units).

The bytecode is, for regular GWT code, what comes out of the embedded
JDT run--which means the bytecode for, say, java.util.HashMap, is the
result of compiling their emulated/re-rooted j.u.HashMap implementation,
not the JDK j.u.HashMap.

So, that's how they can support something like HashMap, which in the JDK
has signatures that point to unsupported types (like java.io), and would
fail going through the TypeOracle. The emulated source (and its
bytecode), from the embedded JDT, and cached in the unit cache
and .gwtar files, don't have those types in their signatures.

Which is exactly what we need.

So, besides just transforming the jribble ASTs to not refer to various
unsupported types, we also need to compile the post-transformation
result (which only exists now as jribble ASTs) to get new GWT-specific
bytecode.

This bytecode won't conflict with/replace the normal scala-library.jar
that is on the user's classpath, but instead is only used to build the
TypeOracle and then passed into the CompilingClassLoader for dev mode.

I think this makes a strong case for maintaining a scala library fork as
source code (as Aaron said, just like GWT maintains their java.util
"fork"). That way we can get both ASTs and bytecode, and our TypeOracle
(and DevMode) will be all the happier for it.

I haven't completely figured out how this could work, but one main thing
is that when the jribble backend runs, say on this source fork of the
library, besides just .jribble files, I'd need .class files that are
specific to the fork (and so loadable via something that isn't
getResourceAsStream('.../Foo.class'), as that'd return the regular
scala-library bytecode).

It almost sounds like the scala compiler should be embedded into the
GWT side, because when compiling in the IDE, it has the server-side
classpath, and produces bytecode/compile errors accordingly. What needs
to happen is the compiler is invoked with the client-side classpath,
which only the GWT side knows about (because it's made up everything
in the resource oracle, via imports in gwt.xml files). This way, if the
user's code refers to a pruned type, they get a compiler error that
tells them as much.

This is a bit more involved that what we're doing now, but, given
my current understanding, I think it's probably the best way to match
what GWT is doing and how it expects things to work.

...we might be able to continue on without embedding scalac on the
GWT side. I think, long-term, that's the best approach/user
experience (especially for compile errors), but we may be able to
limp along without it, as long as there's another way for the GWT
side to get the jribble-specific bytecode, e.g. .jribble-class
files or something odd like that.

(I haven't thought through supporting super source within the
user's own project--that is almost definitely where we'd need
the embedded compiler, because the Scala IDE would never even
try to compile .scala files that lived in the super source
folder. Super source is really what we're trying to do for our
forked scala library, albeit we could have a manual/hacked process
for just the library but then say super source isn't supported
for anything else.)

Thoughts?

- Stephen

Stephen Haberman

unread,
Jul 18, 2011, 1:11:45 AM7/18/11
to scal...@googlegroups.com

> It almost sounds like the scala compiler should be embedded into the
> GWT side

I've been mulling this over. Going this route would basically mean
getting rid of .jribble files and working almost exactly like the
embedded JDT compiler. So:

* Resource oracle scans for .scala files
* These can potentially be super-sourced, which the resource
oracle will transparently take care of based on gwt.xml files
* The compiler interface (JDT or scalac) is then invoked for dev
mode/web mode compile, but it doesn't use the same server-side
classpath it is executing with at runtime, but instead hand-feeds
the embedded compiler only those compilation units that the resource
oracle hands to it.
* The compiler's ASTs + bytecode are saved off as mini ASTs +
GWT-specific bytecode (as it's potentially for super sourced elements,
so is different bytecode than .class files in regular jars) into the
gwt-unitcache or .gwtar files.

So, since this is all happening within the same JVM instance, we could
go directly from scala ASTs (with some amount of transforms) -> GWT
ASTs, no .jribble serialization is needed.

I know this kills the idea of the jribble file format being an easy way
for other languages to target GWT--just make a .jribble file, and you're
done.

That goal sounds cool, but it means your .jribble file was very likely
compiled with the (wrong) server-side classpath, which means no support
for super sourcing, and potential blow ups (method/type not found
assertion errors) at UnifyAst time if you wander outside of your gwt.xml
bounds.

Instead, I think the more robust (but admittedly more work) solution is
for other languages to reuse whatever extension point we poke into
CompilationStateBuilder to also hook in their own embedded compiler
interfaces.

Briefly, the API for this would look a lot like GWT's JdtCompiler, which
wraps the JDT. You're basically given a List<CompilationUnitBuilder>
(which will be mixed .java and .yourlanguage source files, picked up
from the appropriate/super-sourced-if-needed places), you need to turn
your files into CompilationUnits with the mini ASTs, CompiledClasses,
and GWT-specific bytecode.

Once you do that, GWT will do the rest. Which, really, at least
compared with what used to be required, isn't that bad, IMO.

Also, this approach is basically what the GWT side is doing now, except
that it's cheating and reusing the server-side .class files as the
client-side bytecode. Which actually gets us a long way--until the
GWT/client-side bytecode needs to be different than the server-side
bytecode due to super sourcing.

(...hm, instead of removing jribble ASTs entirely, we could potentially
still go scala AST -> jribble AST, and then still use JribbleAstBuilder
to go from jribble AST -> GWT AST. This would mean that other languages
that implemented similar compiler interfaces could transform their ASTs
into the simpler jribble ASTs and then reuse the JribbleAstBuilder
and JribbleReferenceMapper implementations to make the GWT ASTs. Which
would be nice.)

Anyway, I'm rambling, but I was stuck in the car for a long drive today
so had plenty of time to think about it.

- Stephen


Grzegorz Kossakowski

unread,
Jul 18, 2011, 4:10:11 AM7/18/11
to scal...@googlegroups.com
I think I'm getting a bit lost in your prose. From GWT's perspective Jribble is just source code in the same way as Java is. The only difference is that we build GWT mini ASTs from Jribble using simple translation mechanism instead of firing up a proper compiler.

Also, we ship class files along jribble files so it should be transparent to GWT if ASTs it receives are coming from JDT (thus from Java source code) or from Jribble, no?

Moreover, we control building of compilation units for jribble files so we can use whatever we like mechanism for loading class files. Thus, we can load class files corresponding to a forked Scala library, right?

I can assure that this approach of integrating scala compiler into gwt has been considered throughly and got rejected for many reasons including:
  • heavyweight dependencies (we'd depend on whole scala-compiler.jar and scala-library.jar, that's a lot)
  • tight coupling of GWT and Scala projects. Effectively both project would need to synchronize their releases and released version of GWT would be tied to certain version of Scala compiler. This is really bad given how fast Scala evolves. Also, internals of compiler have no guarantees on stability. Actually, I'd go as far as saying that we have a guaranteed breakage for every major release of Scala (and probably often for minor ones too).
Those reasons mean we would have absolutely no hope to get anything merged back upstream to GWT. Same applies to customisations to Scala compiler. We would stay in with two forked projects forever which is not a desirable goal.

I'm very sure that stable layer in form of Jribble is absolutely critical for long-term success of this project. We'll have to overcome any hurdles coming as a consequence of this decision.

--
Grzegorz Kossakowski

Stephen Haberman

unread,
Jul 18, 2011, 9:50:52 AM7/18/11
to scal...@googlegroups.com

> From GWT's perspective Jribble is just source code in the same way as Java is.

Not quite; GWT takes .java files and does it's own resolution/type checking
against the client-side classpath.

When we take .jribble files, we assume all of the resolution and type checking
as done by the scala compiler is correct, and so jump straight to AST making.

But if the code is using any types that are super-sourced (say our
scala library fork), the resolution is potentially wrong.

> Also, we ship class files along jribble files so it should be
> transparent to GWT if ASTs it receives are coming from JDT (thus from
> Java source code) or from Jribble, no?

It will be transparent, yes, but that doesn't mean things will work all
of the time. If the .class files are compiled against the normal
server-side classes, then the user will get undefined behavior in both
dev mode and compiled mode when working against super-sourced types
(well, those that have different type signatures than what they're
replacing).

> Thus, we can load class files corresponding to a forked Scala
> library, right?

True, we'd have to come up with something other than
getResourceAsStream.

That is what I had originally been thinking when I realized there was
another type of bytecode floating around. We would basically be hacking
our own super source feature, rather than building it in as a supported
featre.

> - heavyweight dependencies (we'd depend on whole


> scala-compiler.jar and scala-library.jar, that's a lot)

The approach I proposed would work just fine by defining an interface
in GWT proper like ExtraCompilationUnitCompiler and then having another
module/jar that contained the actual implementation and scala
dependencies.

Much like the scala folk are fine with jribble in their source tree, but
not their main distribution, this would keep scala+gwt code in the GWT
source tree, but in a different artifact like gwt-dev-scala.jar.

(Or, potentially, only ExtraCompilationUnitCompiler is all we patch
back to GWT, and we ship/maintain gwt-dev-scala.jar ourselves. This
makes our patch to GWT very tiny, likely to be accepted, and gives us
the freedom to release new versions of gwt-dev-scala as needed.)

> Effectively both project would need to synchronize their releases and
> released version of GWT would be tied to certain version of Scala
> compiler.

Not necessarily. If ExtraCompilationUnitCompiler was the primary change
to GWT, it would be introduced once and then rarely change. So main GWT
releases wouldn't be required. However, yes, the gwt-dev-scala.jar would
have to change for each scala release, but I don't see how this is much
different than, say, sbt or Scala IDE having to be kept up to date with
each scala compiler release. We'd be doing the same thing they are,
embedding the compiler.

> I'm very sure that stable layer in form of Jribble is absolutely
> critical for long-term success of this project.

If you can propose how to get .jribble files compiled against
super-sourced bytecode (not what's on the Scala IDE classpath), and
then store the resulting bytecode somewhere that isn't the
regular .class files, I'd be very fine avoiding the extra work
involved in what I proposed.

- Stephen


Lex Spoon

unread,
Jul 19, 2011, 3:12:08 PM7/19/11
to scal...@googlegroups.com
I would really like to try the initial plan before looking for
alternatives. The only way to keep Scala and GWT decoupled is if we
can develop a stable intermediate format.

I agree that GWT needs .class files for the Jribble files. My thinking
was that, for now, the source language would have to provide both
.class and .jribble files. Whenever CompilingClassLoader is asked for
bytecode, it checks if the requested class has a .jribble file
associated with it. If so, it loads the bytecode via the main class
loader GWT was run under. If not, it would do whatever it does
currently.

GWT already does (or did) something similar for Emma support. Perhaps
investigate how Emma works, and then add a judicious "||
isFromJribble(requestedClass)" in the right place?


-Lex

Stephen Haberman

unread,
Jul 20, 2011, 11:09:19 PM7/20/11
to scal...@googlegroups.com

> I would really like to try the initial plan before looking for
> alternatives.

Of course.

I just thought we were getting to the point where the initial plan was a
hitting problem: lack of super sourcing so that our scala-library fork
can have a trimmed-down API.

> The only way to keep Scala and GWT decoupled is if we can develop a
> stable intermediate format.

Agreed, we need a stable contract between GWT and Scala.

A file format like jribble is a potential contract, with a number of
very nice properties, but it's not the only way.

A stable API for GWT to invoke 3rd party compilers is another potential
contract. And, AFAICT, the best way to support super sourcing.

> GWT already does (or did) something similar for Emma support.

It looks like emma magic is only done for non-super sourced types.

Which makes sense--for super sourced types, the bytecode from the main
classloader (which came from the real .java file) would potentially be
too different from the GWT/ecj bytecode (which came from the emulated
.java file) to switch out and not avoid wonky behavior.

(Where as, with non-super sourced types, there is only one source .java
file, so GWT has a fairly good chance at mapping between the ecj and
javac bytecode.)

> Perhaps investigate how Emma works, and then add a judicious "||
> isFromJribble(requestedClass)" in the right place?

Given the multiple "!unit.isSuperSource" checks around the emma code, I
don't think it will help us.

And, even if it did fix the CompilingClassLoader issue, it won't fix the
TypeOracle failing to resolve our manually-massaged scala-library
jribble files (which are really just a form of manual super sourcing)
against the original/non-massaged scala-library bytecode.

- Stephen

Lex Spoon

unread,
Jul 23, 2011, 12:54:28 AM7/23/11
to scal...@googlegroups.com
Instead of working out a solution to mismatched .jribble and .class
files, let's get rid of the mismatches. That is, if we trim something
from .jribble, we need to make the same trimming from the .class
files. This can be done multiple ways. One simple way is to have
alternate .scala files for key classes; those .scala files can then be
consistently compiled to both .class and .jribble. There are other
ways, too.

Regarding super-source, my general feel is that it is not the most
robust and stable feature in GWT. We should try not to rely on it if
we can avoid it. To give you an idea, I was in the discussions about
how in the world to support Emma with GWT, and I remember, after much
discussion, when we decided just not to bother for super-source code.
Part of our reasoning was the self-fulfilling expectation that
super-source is not used that much.

Lex

Stephen Haberman

unread,
Jul 23, 2011, 1:40:43 AM7/23/11
to scal...@googlegroups.com

> That is, if we trim something from .jribble, we need to make the same
> trimming from the .class files.

Right, that is the crux of the issue.

> One simple way is to have alternate .scala files for key classes;

Right. Like super sourcing. :-)

> those .scala files can then be consistently compiled to both .class
> and .jribble.

So, if we fork scala.Option.scala, and compile it, the output includes
scala.Option.jribble and scala.Option.class. But there is already
a .class file from the non-forked scala.Option. Where do we put this
2nd .class file so that the GWT side can find it when loading
Option.jribble?

> Regarding super-source, my general feel is that it is not the most
> robust and stable feature in GWT.

Given it's used for the java.lang emulation, I am going to give it the
benefit of the doubt on at least stability.

> Part of our reasoning was the self-fulfilling expectation that
> super-source is not used that much.

I totally agree super-sourcing isn't used much. As a GWT user, there
are two reasons I've avoided it:

1) Maintaining two versions of a class is duplication that should
be avoided, and is usually/hopefully possible if you control the class.

2) Writing super source files sucks because the IDE doesn't
understand the rerooted package structures, and that the duplicate
type declarations aren't really duplicates.

Applied to scalagwt, for 1), we don't control the scala library types,
so are having forking/duplication forced on us whether we like it or
not, and for 2) well, yeah, it will suck.

- Stephen


Lex Spoon

unread,
Jul 25, 2011, 9:24:17 AM7/25/11
to scal...@googlegroups.com
On Sat, Jul 23, 2011 at 1:40 AM, Stephen Haberman
<ste...@exigencecorp.com> wrote:
> So, if we fork scala.Option.scala, and compile it, the output includes
> scala.Option.jribble and scala.Option.class. But there is already
> a .class file from the non-forked scala.Option. Where do we put this
> 2nd .class file so that the GWT side can find it when loading
> Option.jribble?

In a different jar. There will be scala-library.jar, for use on a JVM,
and scala-library-jar, for use with GWT. Their contents will be mostly
the same, but, from the sound of things, might have some differences.

Note that this is exactly the same way that Scala supports .NET. It
has a separate scala-library build. That build does not just differ in
output format. It also uses different .scala files for some parts of
the core library.

I appreciate the clever idea that we can use super-source if we need
to. Ideas like that make me glad this project is open-source. However,
we'll have a simpler, more robust system--at least on the GWT side--if
the Scala library comes in as ordinary source code. Start with the
simplest solution, and give that ground grudgingly.


By the way:

> we don't control the scala library types

We actually have a lot of influence on the Scala library. Nobody is
currently working on it, but one angle we have to make Scala+GWT work
better is to refactor the Scala library. For example, it would be
great if using scala.List didn't also pull in the parallel
collections. Imagine if some key methods indirected through
scala.runtime.*, and we replaced those files for
scala-library-jribble.jar....

Lex

Stephen Haberman

unread,
Jul 25, 2011, 11:51:51 AM7/25/11
to scal...@googlegroups.com

> In a different jar. There will be scala-library.jar, for use on a JVM,
> and scala-library-jar, for use with GWT.

Which comes first on the classpath?

If the JVM one is first, that's what GWT will use for the TypeOracle
bytecode. If the GWT one is first, that's what the compiler/server-side
code will use.

The for-GWT .class files would have to have their name mangled so that
both JVM and GWT library classes can co-exist peacefully on a single
classpath (e.g. scala.Option.class and scala.Option.gwt.class).

Think of GWT--it doesn't ship with its own rt.jar that has the bytecode
of it's emulated java.* (or other non-java) classes. Same thing, that
would cause classpath collision issues.

> However, we'll have a simpler, more robust system--at least on the GWT
> side--if the Scala library comes in as ordinary source code.

Simple, yes, but it's also incorrect. Compiling the .scala files outside
of GWT means the .jribble files will have been resolved against the
server-side classpath. I.e. Class.getInterfaces will be compiled, but
blows up later. That is simple, but not robust.

I would also not be surprised if embedding the compiler makes it
simpler to squirrel away the GWT-specific bytecode (into GWT's existing
caching/archive infrastructure) instead of worrying about how to have
dual scala-library.jars on the same classpath.

> We actually have a lot of influence on the Scala library.

True. That will be awesome, but it's a lot of work, and, in the end,
we'll surely need to emulate some classes here and there.

- Stephen

Stephen Haberman

unread,
Jul 26, 2011, 12:41:20 AM7/26/11
to scal...@googlegroups.com

> Start with the simplest solution, and give that ground grudgingly.

I should have added, I really do understand prefering the simpler
solution. I'm fine (grudgingly, ha) staying with jribble files for now,
providing we find a nice place to put the forked-but-not-super-sourced
scala-library bytecode.

My concern still stands about the client- vs. server-side classpath
mismatch, but that's less important right now.

Regardless of whether jribble files or an ExtraCompilationUnitProcessor
API is what we use in the end, both will need the forked scala-library
as .scala files to get the forked bytecode.

Greg, I imagine this is something you would know most about, given
you're maintaining the .jribble scala library fork. What's the outlook
on moving to/maintaining a .scala version of the scala library fork?

It probably makes the idea of doing some of the transforms via AST
rewriting impractical, unless it was done .scala source -> .scala
source.

- Stephen

Grzegorz Kossakowski

unread,
Jul 26, 2011, 5:33:20 AM7/26/11
to scal...@googlegroups.com
On 26 July 2011 06:41, Stephen Haberman <ste...@exigencecorp.com> wrote:

> Start with the simplest solution, and give that ground grudgingly.

I should have added, I really do understand prefering the simpler
solution. I'm fine (grudgingly, ha) staying with jribble files for now,
providing we find a nice place to put the forked-but-not-super-sourced
scala-library bytecode.

My concern still stands about the client- vs. server-side classpath
mismatch, but that's less important right now.

Regardless of whether jribble files or an ExtraCompilationUnitProcessor
API is what we use in the end, both will need the forked scala-library
as .scala files to get the forked bytecode.

I presume that we have a solution (even hacky) to feed GWT internals with forked class files but still keep original class files in a classpath, right? Jribble library depends on scala-library.jar (let's not go into the same discussion if it's a good idea or not for now).
 
Greg, I imagine this is something you would know most about, given
you're maintaining the .jribble scala library fork. What's the outlook
on moving to/maintaining a .scala version of the scala library fork?

It probably makes the idea of doing some of the transforms via AST
rewriting impractical, unless it was done .scala source -> .scala
source.

Yep. We'll eventually fork Scala library but this is a major, major effort. I tried this idea but seriously, this is hard. I might write down report on all issues I encountered so far but I'm busy with fixing bugs at the moment.

--
Grzegorz Kossakowski

Stephen Haberman

unread,
Jul 26, 2011, 10:56:48 AM7/26/11
to scal...@googlegroups.com

> I presume that we have a solution (even hacky) to feed GWT internals
> with forked class files but still keep original class files in a
> classpath, right?

Not really. The only thing I can come up with, without embedding scalac
in GWT, is to do something like:

* Run scalac target=jvm on the .scala files in the scala-library fork
* With a script, rename all of the .class files to .gwt.class
* Change CompilationStateBuilder.stealJribbleUnits to probe for a
.gwt.class file before falling back to a non-forked .class file for
all jribble units

This should work for now. If we ended up embedding scalac in GWT (so
ignore this for now, but just talking through it), it'd look something
like:

* Invoke ScalaLibrary.gwt.xml precompile
* GWT finds .scala files and .java files (e.g. the right Class.java file
for the emulated JRE library)
* CompilationStateBuilder.stealJribbleUnits calls
EmbeddedScalaApi.compileUnits(java + scala files)
* When EmbeddedScalaApi finishes, any compile errors are stored into the
appropriate CompilationUnits for reporting to the user, otherwise the
bytecode is put into the CompilationUnits (not on disk, although it
could go to disk in a tmp location if necessary)
* GWT invokes JdtCompiler.compileUnits(java files) as normal
* GWT precompile makes a ScalaLibrary.gwtar with all of the mini ASTs +
bytecode cached so that scala-gwt users don't have to recompile any
of it when their app starts up

> Jribble library depends on scala-library.jar (let's not go into the
> same discussion if it's a good idea or not for now).

Agreed on not rehashing that discussion right now.

> Yep.

Cool.

> We'll eventually fork Scala library but this is a major, major effort.

Yeah--I think it will end up being the hardest, most time-consuming part
of scala-gwt.

> I tried this idea but seriously, this is hard. I might write down
> report on all issues I encountered so far but I'm busy with fixing
> bugs at the moment.

The issue report sounds like a good idea, but understood that you have
other things to fix right now.

- Stephen


Lex Spoon

unread,
Jul 30, 2011, 10:05:25 AM7/30/11
to scal...@googlegroups.com
On Mon, Jul 25, 2011 at 11:51 AM, Stephen Haberman
<ste...@exigencecorp.com> wrote:
>
>> In a different jar. There will be scala-library.jar, for use on a JVM,
>> and scala-library-jar, for use with GWT.
>
> Which comes first on the classpath?

That's an immediate issue, but it shouldn't be by the time we merge.

One consideration is that many on the GWT team have long lobbied for a
-classpath argument to gwtc so as to separate the classpath for the
compiler's implementation from the classpath for the program being
compiled. This is by far the cleanest approach, and I daresay it is
standard compiler construction. If you need a solution immediately,
this is probably the best angle.

Second best is to solve this problem the same way GWT does for its
other dependencies: rewrite the Scala library with jarjar for use
within GWT. If for some reason they push back about a -classpath
argument, this way should work. Based on some Googling, Scala does
continue to work after being jarjared.

Best of all, though, is not to depend on Scala in the GWT code. I'll
post more on this later, but I still believe that this is too small of
a problem to merit giving them a Scala dependency. They aren't going
to want it, and I don't want to be in the position of arguing they
should.

Lex

Stephen Haberman

unread,
Jul 31, 2011, 12:01:57 AM7/31/11
to scal...@googlegroups.com

> > Which comes first on the classpath?
>
> That's an immediate issue, but it shouldn't be by the time we merge.

Hm.

> One consideration is that many on the GWT team have long lobbied for a
> -classpath argument to gwtc so as to separate the classpath for the
> compiler's implementation from the classpath for the program being
> compiled.

Sure, but that's compiler-classpath vs. user-classpath. We have to
further subdivide the user-classpath into client-user-classpath and
server-user-classpath.

For example, scalagwt-sample running dev mode would need both the real
scala-library (for user server classes) and forked scala-library (for
user client classes) bytecode both coexisting on the user-classpath.

> Second best is to solve this problem the same way GWT does for its
> other dependencies: rewrite the Scala library with jarjar for use
> within GWT.

No, jarjar wouldn't work. jarjar doesn't just change package names, it
rewrites the references to classes within the rewritten packages. So
scala.jarjar.Option would end up having pointers to scala.jarjar.Some
(say), when really we still want the pointer to be scala.Some. Otherwise
the TypeOracle/GWT will get even more confused and we'd have to
un-rewrite the references somehow.

(GWT uses jarjar for purely compiler-side dependencies, e.g. it's guava
fork, which is only used by the compiler and never ends up as JavaScript
in user programs.)

> Best of all, though, is not to depend on Scala in the GWT code. I'll
> post more on this later, but I still believe that this is too small of
> a problem to merit giving them a Scala dependency.

Right. If you recall, one of my first threads on the list was making the
GWT patch as palatable to the GWT team as possible. So, yay for no Scala
dependency in GWT core.

However, I've already proposed an API in GWT core that can be
implemented in a non-primary jar that does include a scalac dependency.

The scalac API needed would be roughly: hand it some bytecode that
forms the user client-side classpath, hand it some source .scala files,
and be given back the resulting bytecode + ASTs (possibly jribble ASTs).

I don't think there is anything wrong with this approach. The GWT team
embeds ecj. There is a reason they do so. We'll have the best chance of
providing the most robust experience by doing the same.

That being said, I'm perfectly willing to keep hacking along with our
current approach until we get feedback from the GWT team. If they can
solve our dual client-/server-user classpath conflicts while keeping
the .jribble approach, I will be happy. I am just anticipating they
will recommend using super sourcing like GWT projects do today.

- Stephen

Lex Spoon

unread,
Jul 31, 2011, 9:57:24 AM7/31/11
to scal...@googlegroups.com
On Sun, Jul 31, 2011 at 12:01 AM, Stephen Haberman > Sure, but that's

compiler-classpath vs. user-classpath. We have to
> further subdivide the user-classpath into client-user-classpath and
> server-user-classpath.

Yes, potentially. For our case, though, all we care about is the
compilation classpath. That one would have scala-library-jribble.jar,
while the others would have scala-library.jar.

If a particular developer is using multiple versions of
scala-library.jar, meaning multiple versions of Scala, then they are
beyond the abilities of the Scala+GWT project to help them.


>> Second best is to solve this problem the same way GWT does for its
>> other dependencies: rewrite the Scala library with jarjar for use
>> within GWT.
>
> No, jarjar wouldn't work. jarjar doesn't just change package names, it
> rewrites the references to classes within the rewritten packages.

[...]


> (GWT uses jarjar for purely compiler-side dependencies, e.g. it's guava
> fork, which is only used by the compiler and never ends up as JavaScript
> in user programs.)

Yes, that's what I had in mind for the Jribble support as well.
Rewrite the compiler's Jribble support. The code being compiled would
use scala.* in its normal location.


Lex

Stephen Haberman

unread,
Aug 1, 2011, 6:22:10 PM8/1/11
to scal...@googlegroups.com

> > We have to further subdivide the user-classpath into
> > client-user-classpath and server-user-classpath.
>
> Yes, potentially. For our case, though, all we care about is the
> compilation classpath. That one would have scala-library-jribble.jar,
> while the others would have scala-library.jar.

I could see that working for web mode, assuming a classpath param
existed.

Dev mode still seems problematic. Unless the classpath param existed,
and the real scala-library was put into WEB-INF/lib, and the
jetty/tomcat webapp class loader did the right thing.

Nonetheless, we will still run in to issues like:

https://github.com/scalagwt/scalagwt-scala/issues/13

Where UnifyAst blows because the user's code was compiled against the
wrong bytecode. Besides worrying about the GWT classpath (which we've
been talking about the -classpath param for), we also have to worry
about the scalac classpath, insofar as in a project like:

* com/myapp/client/Foo.scala
* com/myapp/server/Bar.scala

Bar.scala should be able to use String.format, but Foo.scala should
report String.format usage as a compile error. Right?

Embedding the scalac compiler, ecj style, and hand-feeding it .java
and .scala files that come from the resource oracle would solve this.
The global classpath is no longer an issue, and we just let GWT's
CompilationUnits manage/cache the bytecode just like it does for .java
files.

> If a particular developer is using multiple versions of
> scala-library.jar, meaning multiple versions of Scala, then they are
> beyond the abilities of the Scala+GWT project to help them.

Agreed.

- Stephen

Grzegorz Kossakowski

unread,
Aug 3, 2011, 9:19:11 AM8/3/11
to scal...@googlegroups.com
On 2 August 2011 00:22, Stephen Haberman <ste...@exigencecorp.com> wrote:
Nonetheless, we will still run in to issues like:

https://github.com/scalagwt/scalagwt-scala/issues/13

Where UnifyAst blows because the user's code was compiled against the
wrong bytecode. Besides worrying about the GWT classpath (which we've
been talking about the -classpath param for), we also have to worry
about the scalac classpath, insofar as in a project like:

* com/myapp/client/Foo.scala
* com/myapp/server/Bar.scala

Bar.scala should be able to use String.format, but Foo.scala should
report String.format usage as a compile error. Right?


See my input on this:


Still, you'd need separate classpath for server and client side packages, that's true. However, our answer should be: have two different projects (e.g. in Eclipse) and then you can easily setup the classpath in your preferred way.

This might feel like a unnecessary contraint to people but we can sell it very easily: with approach I outlined we can configure Scala IDE to use special classpath for client project and Scala IDE will properly (and instantly) report errors when you try to refer to things that are not supported in GWT either in Java or Scala library. AFAIK, GWT doesn't do it for Java files and only reports problems like in #13 when GWT compiler is being used and it report in it's own console window. We could do much better in Scala+GWT. :-)


Embedding the scalac compiler, ecj style, and hand-feeding it .java
and .scala files that come from the resource oracle would solve this.
The global classpath is no longer an issue, and we just let GWT's
CompilationUnits manage/cache the bytecode just like it does for .java
files.

I talked briefly to Scala IDE guys and they said that if we are going to embed scala compiler into GWT and do whatever magic we might introduce then there's little chance we can get decent support from them. Also, I've been warned that scala doesn't have any stability guarantees and tend to change APIs very frequently contrary to ECJ. I don't want to introduce this complexity to GWT people.

There's another argument for separation: we'll be able to merge in jribble backend into main scala repository (but probably not into distribution) with all tests running in standard test suite. I can convince Scala team to care about jribble backend provided it's reasonably covered by tests but there's no chance they'll ever want integrate GWT into any nightly tests. It's simply too big, too complex thing that they won't want to learn.

--
Grzegorz Kossakowski

Stephen Haberman

unread,
Aug 3, 2011, 12:39:45 PM8/3/11
to scal...@googlegroups.com

> https://groups.google.com/d/topic/scalagwt/AUp4r9Qu9Zg/discussion

Hm, sounds intriguing, I'll look in to it when I get a chance.

> Scala IDE will properly (and instantly) report errors when you try to
> refer to things that are not supported in GWT either in Java or Scala
> library.

That does sound cool. Will the Scala IDE really be okay with a
non-standard scala-library on the project's classpath? I thought it was
fairly insistent on it's SCALA_CONTAINER having a scala-library that
exactly matches whatever version of the scala-compiler the Scala IDE is
using. And that you'd get undefined behavior if you put your own version
of the scala-library on the classpath before SCALA_CONTAINER.

Or have they fixed this restriction?

I had assumed not, given they are still shipping Scala IDEs that are
built for specific scala-library usage (e.g. this update site if you
want to use scala 2.9 in your project, that update site if you want to
use scala trunk).

> I talked briefly to Scala IDE guys and they said that if we are going
> to embed scala compiler into GWT and do whatever magic we might
> introduce then there's little chance we can get decent support from
> them.

Sure, but I don't think we'd need very much support. I don't know what
you mean by "whatever magic", but see below for what we'd really need.

> Also, I've been warned that scala doesn't have any stability
> guarantees and tend to change APIs very frequently contrary to ECJ. I
> don't want to introduce this complexity to GWT people.

Yes, this sounds scary, but really, I don't think it would be a big
deal. It's not like GWT needs all the API complexity of, say, the Scala
IDE knowing the intricate internals of the scala compiler.

All GWT would need is a stable SimpleScalaCompilerApi interface that
looked like:

addClasspathEntry(byte[] classBytes);
addJavaSource(String contents);
addScalaSource(String contents);
List[JRibbleAsts] compile();

As long as that lived in your jribble jar in the scala tree, that's all
the Scala team would have to support and keep from breaking. Well, and
the JRibbleAsts classes (which would now not have to be serialized).

> There's another argument for separation: we'll be able to merge in
> jribble backend into main scala repository (but probably not into
> distribution) with all tests running in standard test suite.

Right--we could still do this with embedding. SimpleScalaCompilerApi
could have a handful of tests and that's it.

> I can convince Scala team to care about jribble backend provided it's
> reasonably covered by tests but there's no chance they'll ever want
> integrate GWT into any nightly tests. It's simply too big, too complex
> thing that they won't want to learn.

Right, that's not anything I was proposing.

There seems to be confusion around the entire embedding concept--it
can be done without introducing direct dependencies of GWT (core) on
Scala or Scala on GWT. It's just making extremely simple interfaces
(ExtraCompilationUnitCompiler on the GWT side and ScalaCompilerApi on
the Scala side), and then having us provide the glue code.

(Okay, realistically the glue code would be in a non-core GWT jar,
e.g. gwt-scala.jar, that implemented ExtraCompilationUnitCompiler
by calling the ScalaCompilerApi and, just as we are now, transforming
the scala-side JribbleAsts to GWT-side ASTs.)

I can't see it being more than a handful of code (AST transformation
aside, and which we have already anyway). If I had a ScalaCompilerApi
like I outlined above, I could do a spike in fairly short order.

That being said, Greg, I'll look into your new post about the boot
classpath and go from there. If you could let me know about the Scala
IDE <-> scala-library version restriction situation, that'd help
convince me that putting a non-standard scala-library in Scala GWT
Eclipse projects would actually work out well.

- Stephen


Grzegorz Kossakowski

unread,
Aug 3, 2011, 5:09:46 PM8/3/11
to scal...@googlegroups.com
On 3 August 2011 18:39, Stephen Haberman <ste...@exigencecorp.com> wrote:
> Scala IDE will properly (and instantly) report errors when you try to
> refer to things that are not supported in GWT either in Java or Scala
> library.

That does sound cool. Will the Scala IDE really be okay with a
non-standard scala-library on the project's classpath? I thought it was
fairly insistent on it's SCALA_CONTAINER having a scala-library that
exactly matches whatever version of the scala-compiler the Scala IDE is
using. And that you'd get undefined behavior if you put your own version
of the scala-library on the classpath before SCALA_CONTAINER.

Or have they fixed this restriction?

I had assumed not, given they are still shipping Scala IDEs that are
built for specific scala-library usage (e.g. this update site if you
want to use scala 2.9 in your project, that update site if you want to
use scala trunk).

One need to understand Scala's compiler design (both proper compiler and presentation compiler) to see why compiler and standard library are so coupled. Roughly speaking, when compiler is being initialized it has to build some internal structures (Symbols) for standard things like Predef.scala, Xml stuff (being used when you use xml literals), etc. In order to build those data structures compiler needs to read class files and parse binary data. It assumes quite a lot about interactions between different class files when deserializing this data (things are done lazily here) so little change in some class file might blow up everything.

However, people are already working on improving this situation. We should have some means to tell if particular version of library will work fine with particular version of a compiler. I understand those things quite well so I think I'm capable of preparing gwt version of scala's library that will work with corresponding version of scala compiler.

Anyway, definitive answers will come next week when Martin is back from vacations. We talked about this issue briefly in a past and he expressed an interest in solving those problems.
 
> I talked briefly to Scala IDE guys and they said that if we are going
> to embed scala compiler into GWT and do whatever magic we might
> introduce then there's little chance we can get decent support from
> them.

Sure, but I don't think we'd need very much support. I don't know what
you mean by "whatever magic", but see below for what we'd really need.

Magic for setting up a classpath when running scala compiler. If we want a decent IDE support we need to replicate this logic in IDE itself.
 
> Also, I've been warned that scala doesn't have any stability
> guarantees and tend to change APIs very frequently contrary to ECJ. I
> don't want to introduce this complexity to GWT people.

Yes, this sounds scary, but really, I don't think it would be a big
deal. It's not like GWT needs all the API complexity of, say, the Scala
IDE knowing the intricate internals of the scala compiler.

All GWT would need is a stable SimpleScalaCompilerApi interface that
looked like:

   addClasspathEntry(byte[] classBytes);
   addJavaSource(String contents);
   addScalaSource(String contents);
   List[JRibbleAsts] compile();

As long as that lived in your jribble jar in the scala tree, that's all
the Scala team would have to support and keep from breaking. Well, and
the JRibbleAsts classes (which would now not have to be serialized).


Oh, ok. Indeed I misunderstood you. You want to introduce some stable, minimal interface that both sides will target. That makes much more sense now.

However, I'm still uncomfortable with letting gwt controlling Scala compiler. For example, how your interface will allow people to use CPS (continuations) plug-in?

Using scalac from command line has been proven for years so it's pretty safe option for us. Actually, one cool idea is that you can build such an interface without anybody's help. In implementation you can still fork VM and just run scala compiler as from command line. This way you can try this out without grabbing anybody's attention. If this approach turns out to be sound we might switch to different implementation that does not fork VM process but really embed scalac.

*But*, *but*, Stephen, I really value all your hard work you've done so far and would prefer to see you working on other things. This problem with code splitter is an excellent example of something really important. There was a bug and I'm glad you fixed it. This way we move the whole project forward without me being drawn into every single problem.

> There's another argument for separation: we'll be able to merge in
> jribble backend into main scala repository (but probably not into
> distribution) with all tests running in standard test suite.

Right--we could still do this with embedding. SimpleScalaCompilerApi
could have a handful of tests and that's it.

> I can convince Scala team to care about jribble backend provided it's
> reasonably covered by tests but there's no chance they'll ever want
> integrate GWT into any nightly tests. It's simply too big, too complex
> thing that they won't want to learn.

Right, that's not anything I was proposing.

There seems to be confusion around the entire embedding concept--it
can be done without introducing direct dependencies of GWT (core) on
Scala or Scala on GWT. It's just making extremely simple interfaces
(ExtraCompilationUnitCompiler on the GWT side and ScalaCompilerApi on
the Scala side), and then having us provide the glue code.

(Okay, realistically the glue code would be in a non-core GWT jar,
e.g. gwt-scala.jar, that implemented ExtraCompilationUnitCompiler
by calling the ScalaCompilerApi and, just as we are now, transforming
the scala-side JribbleAsts to GWT-side ASTs.)

I can't see it being more than a handful of code (AST transformation
aside, and which we have already anyway). If I had a ScalaCompilerApi
like I outlined above, I could do a spike in fairly short order.


That part is more clear, however I'm still skeptical.

 
That being said, Greg, I'll look into your new post about the boot
classpath and go from there. If you could let me know about the Scala
IDE <-> scala-library version restriction situation, that'd help
convince me that putting a non-standard scala-library in Scala GWT
Eclipse projects would actually work out well.

I'll by trying this out tomorrow or on Friday. Definitive answers will come next week. I'm fairly optimistic about this.

--
Grzegorz Kossakowski

Stephen Haberman

unread,
Aug 3, 2011, 10:46:45 PM8/3/11
to scal...@googlegroups.com

> However, I'm still uncomfortable with letting gwt controlling Scala
> compiler. For example, how your interface will allow people to use CPS
> (continuations) plug-in?

Good question. I imagine there'd be some config/system property that passed
settings on to scalac. So you'd invoke DevMode/Compiler with
-Dgwt.scalac.flags=... where the value looked a lot like what people
pass to the CLI of scalac. Or something like that.

> Actually, one cool idea is that you can build such an interface
> without anybody's help.

That is a good idea, actually. I might give it a shot.

> would prefer to see you working on other things.

Fair enough, but what sort of things? AFAIK resolving the dual bytecode
issue in one way or another is the last hard problem on the GWT side.

Well, that and manifests.

Knock on wood though; I would of course expect more issues to come up as
more people/projects use scalagwt.

> That part is more clear, however I'm still skeptical.

:-)

- Stephen

James Nelson

unread,
Jun 10, 2012, 8:18:30 PM6/10/12
to scal...@googlegroups.com
I'm not sure if this post is old news or not,
but, speaking from personal experience with extensive use of super-sourcing... 

The best way to handle multiple classpaths seems to be maven.
To use multiple classpaths on client and server, you setup a module for each, plus a common module for shared code.
Gwt launches it's codeserver from your client-only classpath, server launches with pure scala.
Both include shared code, but gwt mangles it during generate-sources lifecycle.

Your client-only classpath can support a target/generated-sources folder in which you emit JIT-mangled java sources, as super sources.
Make sure your import statemtents and source packaging are mangled correctly; gwt erases packages like com.google.gwt.emul from emulated source.

With this generated source folder, gwt can run it's codeserver and talk to your mangled scala library.

On the server, you just run plain scala and ignore all gwt-hackery.
The potential gotchas here are:
a) Gwt codeserver must run in a different jvm than server.  Ray Cromwell just dropped SuperDevMode, which does exactly this for you.  You can likely tweak it to use your ExtraCompilationUnitBuilder.
b) Maven dependency (although, with archetypes, eclipse support, version control and source discovery, this can be a good thing}
c) Must run generate-sources maven goal wherever an object serialized between client and server changes
d) Must export deploy artifacts {serialization schema} to server module instead of client {gwt compiler option -out}

For the price of maven dependency, you get decent eclipse support, maven handles classpath, you keep "pure .jribble" format, eclipse projects can be generated from archetype, and gwt can run headless in it's own jvm.

If you've noticed the lockup and asynchronicity issues caused from running the codeserver and serverserver in the same jvm,
you will notice significant performance improvements with such a setup.

Check out https://plus.google.com/110412141990454266397/posts/DMKGiEKJo3p if you want to try out SuperDevMode
Reply all
Reply to author
Forward
0 new messages