Any plans for supporting Scala ?

[Maxime Lévesque]

Has there ever been discussions about supporting scala ?
What I like most about GWT is that it allows to write rich AJAX client
using a strongly typed language (java), which is pretty unique.
I have come to enjoy Scala very much, being able to write GWT
apps in Scala would be totally awesome.

I would think that compiling from scala source to javascript would
actually be easyer than from java, scala closures for example
could translate almost directly to javascipt closures.

I assume that significant part of your java->javascript translater
could be resused, and the fact that scala lives in the JVM would
certainly help.

What kind of effort would it take ?

[Rob Heittman]

I think Lex Spoon knows a thing or two about Scala.  I'm really surprised, nay shocked, that he hasn't ported GWT to Scala yet, on his lunch break!

j/k  :-)

2009/10/1 Maxime Lévesque <maxime....@gmail.com>

[Maxime Lévesque]

   Lex, are you there ?

 Could you crank out a Scala compiler for GWT in your next lunch break ? ;)

 Max

[Joel Webber]

I don't want to speak for Lex, but I will say that there are plenty of people out there (myself included) who would love to see a Scala frontend to GWT. And Maxime, I completely agree that a lot of Scala constructs would translate more effectively to Javascript than if they were translated through Java (or bytecode) first. This is one reason we've not really resonated with the suggestion that "it would be better if GWT just translated bytecode, so we could get languages like Scala for free". I'm afraid there would be too much useful information lost in the process, and it wouldn't be all that useful if it produced poor code (making large apps work in Javascript is hard enough as it is!).

This implies that it would take a significant change to the compiler infrastructure to efficiently support Scala, though, because we'd need a more expressive intermediate representation. I'll let those with more compiler experience than myself speak to the structure it should take, though.

Cheers,

joel.

2009/10/2 Maxime Lévesque <maxime....@gmail.com>

[John Tamplin]

2009/10/1 Maxime Lévesque <maxime....@gmail.com>

Has there ever been discussions about supporting scala ?

There have been some water cooler discussions about it, but it seemed like there were too many tasks left to do to make Java-based GWT everything we want it to be to undertake such a project.

People have used Scala on the server with App Engine, if that is of interest.

 

 I assume that significant part of your java->javascript translater
could be resused, and the fact that scala lives in the JVM would
certainly help.

That would help for Development Mode (previously called Hosted Mode), but doesn't really matter for compiling.
 

 What kind of effort would it take ?

We already build our own Java AST from the JDT AST, so the primary things would be:

• wire up a Scala compiler
• maybe have to do something for Java interop since we wouldn't necessarily have class files for dependencies at the time of compilation
  
• generate AST nodes from the Scala compiler
• update code generation passes to account for new AST nodes or new uses of existing ones
• probably minor modifications to Development Mode to know how to generate bytecode on demand for Scala
  

--
John A. Tamplin
Software Engineer (GWT), Google

[Ray Ryan]

I don't suppose there is an existing scala -> java source translator?

On Oct 2, 2009 7:04 AM, "John Tamplin" <j...@google.com> wrote:

2009/10/1 Maxime Lévesque <maxime....@gmail.com>

Has there ever been discussions about supporting scala ?

There have been some water cooler discussions about it, but it seemed like there were too many tasks left to do to make Java-based GWT everything we want it to be to undertake such a project.

People have used Scala on the server with App Engine, if that is of interest.

  > >  I assume that significant part of your java->javascript translater > could be resused, and th...

That would help for Development Mode (previously called Hosted Mode), but doesn't really matter for compiling.
 

> >  What kind of effort would it take ?

We already build our own Java AST from the JDT AST, so the primary things would be:

• wire up a Scala compiler
• maybe have to do something for Java interop since we wouldn't necessarily have class files for dependencies at the time of compilation
  
• generate AST nodes from the Scala compiler
• update code generation passes to account for new AST nodes or new uses of existing ones
• probably minor modifications to Development Mode to know how to generate bytecode on demand for Scala
  

--
John A. Tamplin
Software Engineer (GWT), Google

--~--~---------~--~----~------------~-------~--~----~ http://groups.google.com/group/Google-Web-To...

[Joel Webber]

scalac + a decompiler ought to do the trick, roughly. But you'd still end up with a bunch of big ugly Java constructs for things like functions, case classes, pattern matching, and the like. And while I'd love to see what would happen to that code if you ran it through gwtc, I'm guessing it would be rather suboptimal relative to what you'd get if you took the (scala -> (some better IR) -> js) path.

[Ian Petersen]

On Fri, Oct 2, 2009 at 8:06 AM, Joel Webber <j...@google.com> wrote:
> scalac + a decompiler ought to do the trick, roughly. But you'd still end up
> with a bunch of big ugly Java constructs for things like functions, case
> classes, pattern matching, and the like. And while I'd love to see what
> would happen to that code if you ran it through gwtc, I'm guessing it would
> be rather suboptimal relative to what you'd get if you took the (scala ->
> (some better IR) -> js) path.

It might be worth it to make scalac -> decompiler -> gwtc a supported
option, just to find out how much interest there really is in Scala ->
JS. If there's only three people in the world who want Scala -> JS,
it's really up to Lex if he wants to spend his lunches supporting
them. On the other hand, there might be enough people interested in
Scala -> JS that it's worth putting a hold on the Java -> JS polish to
get proper support for Scala working RSN.

Just a thought.

Ian

[Joel Webber]

I think any interested party ought to be able to try the decompiler route and see what happens. God knows how weird it might look, but I see no obvious reason that it shouldn't work. Takers?

[Joel Webber]

And I believe that should be a reasonable proxy for what it would look like to go from any statically-typed JVM language to Javascript without the benefit of a new IR. If it doesn't suck, then maybe I'm wrong about needing a new IR. I doubt it, but I've been wrong plenty of times before :)

[John Tamplin]

On Fri, Oct 2, 2009 at 11:52 AM, Joel Webber <j...@google.com> wrote:

I think any interested party ought to be able to try the decompiler route and see what happens. God knows how weird it might look, but I see no obvious reason that it shouldn't work. Takers?

Are there any free decompilers that do fully correct bytecode->source translation?  When I last looked, there were a lot of old (apparently) unsupported ones, and a few that mostly worked.  They were plenty good enough to figure out what the original code was trying to do, but not suitable for modifying the code and compiling it to functional code.

[Maxime Lévesque]

  Scalac has an option to produce desugared java like output,
(I say "java like", because it is "almost" compilable Java, but quite).
It's a horror scene from a readability standpoint, you get to see
all the tricks Scalac uses  to support closures, implicit conversions, comprehensions
and what not on the java VM. Looking at stack traces produced by Scala programs
gives you another hint or two. My take is that it is beyond suboptimal,
in the sense that the complexity of making it work might be not much
less than doing the ScalaAst -> GwtAst translation, but I could be wrong.

  In terms of interest in a Scala GWT, it's a chicken and egg thing,
my opinion is that once people get to see the expressivity of Scala applied
to GWT, the interest should grow.
One particular pattern that would illustrate this would be a builder
similar to : http://groovy.codehaus.org/Swing+Builder.
I like what I see in this example, with GWT-Scala you would get
the same level of expressivity, with the added benefit of strong typing
(for me that is a big one) and a superior UI framework (...yes I am a fan of GWT...).

 Scalac exposes it's API via plugin, it could save the most boring
work for a ScalaAst -> GwtAst translator :
 http://www.nabble.com/Creating-AST-trees-in-compiler-plugins-td22995976.html

 I'm no compiler Guru, nor a GWT one, but it would seem that if the
Gwt AST was enriched to support at least closures,
(might be usefull one day : http://www.javac.info/ ...one week after never...
but that is another topik...;))
possibly some other constructs that translate easily from Scala to JS
(if there are other) the translator would be easier to write.
... at least in theory !

  Max

[Miroslav Pokorny]

The main hurdle remains that the compiler works on source and not
bytecode, was there not a plan to change this sometime in the future.

If it worked on bytecode one could use any language as it would not
matter. Fir the moment it's too late got that now, unless some dies
the work of building the ast from bytecode instead of source.

[John Tamplin]

On Fri, Oct 2, 2009 at 10:53 PM, Miroslav Pokorny <miroslav...@gmail.com> wrote:

The main hurdle remains that the compiler works on source and not
bytecode, was there not a plan to change this sometime in the future.

If it worked on bytecode one could use any language as it would not
matter. Fir the moment it's too late got that now, unless some dies
the work of building the ast from bytecode instead of source.

The problem with that, as you can see by inspecting decompiler output, is that it is hard to recover the original source constructs from the bytecode, and you likely wind up taking lower-level constructs to build the JS from.  In IHM, recovering source-level generic info from the bytecode signatures was a huge nightmare, and had to have heuistics based on which compiler generated the bytecode to do everything.

Also, most of the languages that compile to bytecode use reflection to translate the source language in at least some cases, so it would not actually help you get those languages translated to GWT.

[Lex Spoon]

Let me try and sum up where things are.

Regarding manpower, I spent a month or two spending 20% time on this,
following all of the threads discussed in this email. From where
things are, I would estimate a 3-week full time effort to get a basic
version working that, for example, translated function literals the
same way that inner classes are translated. Getting structural method
calls to work would take a little while longer, but they aren't used
that much in most Scala code. Getting Scala function literals
translated to JavaScript function literals would require updating
GWT's IR analogous to what Max describes, and nobody has really
thought through a specific plan for that.

If anyone wants to work on this, I can give all the relevant Scala and
GWT pointers. Just let me know. It would involve writing some Scala
code, but would not require any deep knowledge of compilers.

Let me now update all the subthreads I see on here with what I found.
Please ping if I have overlooked something.

Decompilation does not work on Scala. I tried all the ones I could
get my hands on, and they all fail. It's been about a year, so if
anyone knows of a decompiler breakthrough in that time, it might be
possible now. However, it would take large changes. The decompiled
output I looked at was fine for 90% of the code, but the remaining 10%
included some really big problems that needed solving.

Generating Java source code from scalac might be possible, but it's
terribly difficult. Java source code does not want to be a target
language. Java rejects all kinds of code that are logical but that
wouldn't make sense to write by hand. When I started on this path, I
enumerated all the problems I knew of and found a solution to all of
them. However, along the way, more of them popped up. None of them
were ever fatal by themselves, but there are a bunch of them.

A better tactic would be to define a modified Java source language
that does not have the restrictions. It would be pretty much the same
as Java, but with a few changes. A partial list of changes would be:

1. There is no rule in constructors about calling the super()
constructor as the syntactically first thing in a constructor.
2. There is a comma operator expression just like in JavaScript.
3. Any expression can be used in an expression statement, even useless
things like literals.
4. The types would all be Java erased types.
5. Probably imports would be removed.
6. Overloading and overriding would be as in Java byte code, including
override on return type.
7. That probably implies that method calls are also as in Java byte
code, and specify the full method signature.

This plan has the huge advantage that everything in the chain is under
our control. The Scala compiler could generate this language, and GWT
could read it.

Also, per John's comments about decompiling Java, note that we could
plan to update this language to support more source languages than
just Java. It should be much easier for them to emit what they mean
than for GWT to reverse engineer what their bytecode intended.

Lex Spoon

[John Tamplin]

On Mon, Oct 5, 2009 at 12:10 PM, Lex Spoon <sp...@google.com> wrote:

This plan has the huge advantage that everything in the chain is under
our control.  The Scala compiler could generate this language, and GWT
could read it.

Also, per John's comments about decompiling Java, note that we could
plan to update this language to support more source languages than
just Java.  It should be much easier for them to emit what they mean
than for GWT to reverse engineer what their bytecode intended.

Why would that be easier than just parsing Scala and building an extended GWT AST from the Scala AST?  It seems like inventing a new language (even if it is close to Java) and modifying tools on both sides to use this would be more work.

[Ray Cromwell]

if I understand, Lex is not really talking about a new syntax/text representation, but an intermediate Java AST that has none of the restrictions that Java does. The current GWT AST lifts many of these restrictions too (you can build 'illegal' code in it that compiles fine to JS, but would not work with javac if tostring'ed). The problem with using the GWT AST is that if you want to hand something simple to Groovy, or Nice, or Clojure, it doesn't separate cleanly. Ideally, you'd want to hand them a gwtast.jar with a simple, well defined set of constructors, and promise them that if they use this to generate code,  it is guaranteed to compile.

Essentially, you want a portable intermediate representation, and currently, Java syntax on disk doesn't have enough fidelity.

[Rob Heittman]

I'd be willing to work on such a project once GWT 2.0 final lands, which will help stabilize the rest of my world and give me some free time to play.

On Mon, Oct 5, 2009 at 12:10 PM, Lex Spoon <sp...@google.com> wrote:

[Lex Spoon]

On Mon, Oct 5, 2009 at 12:23 PM, John Tamplin <j...@google.com> wrote:
> Why would that be easier than just parsing Scala and building an extended
> GWT AST from the Scala AST?  It seems like inventing a new language (even if
> it is close to Java) and modifying tools on both sides to use this would be
> more work.

Parsing isn't enough. It would also be necessary to type check, so
that the meaning of the ASTs can be understand. After that, it would
be necessary to desugar the Scala ASTs down into Java equivalents.
That adds up to the bulk of a Scala compiler. Especially the type
checking part would take a lot of time to reimplement.

The next best thing would be to call into the existing Scala compiler.
However, doing it that way means that we have to figure out a way to
supply people with compatible versions of the GWT compiler and Scala
compiler. If we don't come up with a stable API between the two code
bases, then we'll have to figure out a way to supply users binaries of
each language that work well together. We can do that, but it will
mean people have an extra constraint when they choose which version of
GWT and which version of Scala they want to use. Coming up with an
API looks better.

The text format would essentially be such an API. It is unlikely to
change much over time, because both code bases are tracking Java's
glacial motion. Further, it shouldn't be too time consuming to
develop, because it's only use would be to make two known code bases
be able to talk to each other.

Lex

[Lex Spoon]

On Mon, Oct 5, 2009 at 12:37 PM, Ray Cromwell <cromw...@gmail.com> wrote:
>
> if I understand, Lex is not really talking about a new syntax/text
> representation, but an intermediate Java AST that has none of the
> restrictions that Java does.

I was thinking it would be a text format, but it could be a binary
format if that looks easier.

Lex