Massive refactor: enable multiprocess GWT compiles

[Scott Blum]

Hi Bob, Daniel,

The massive attached patch really has one purpose: enable GWT compiles to run across multiple processes or machines.

Algorithm outline:

1) Precompile.

Produces a "Precompilation" data structure, which is serializable.  This Precompilation contains:

- a unified (all permutations), preoptimized AST where no deferred binding decisions have yet been mode

- the JJSOptions which produced the AST; these are not modifiable by subsequent compiler passes

- produce metadata regarding the number of permutations, and what the deferred binding decisions are for each permutation

- generated artifacts

When Precompile is run from the command line, it produces a "precompilation.ser" (under .gwt-tmp) which contains the Java serialized data.  It also generates a file "permCount.txt" which simply contains an integer of the number of permutations; this is intended to facilitate build tools dynamically determining the number of permutations for sharding.

Precompile takes exactly the same set of command line flags that GWTCompiler does.

2) CompilePerms

Compile one or more permutations into JavaScript.  The input is the "Precompilation" (but the generated artifacts are ignored during this step).  The output is a String JS file.  When run from the command line, the default is to compile all perms, but the "-perms" option lets you specify exactly which ones to compile.  The output files follow the scheme "permutation-0.js", "permutation-1.js".

CompilePerms takes a greatly restricted set of command line options, since the compiler options passed into the Precompile are preserved, and things like '-gen' no longer matter.

CompilePerms does not depend on the user's project being available on the classpath; all of the dependencies are serialized.  The only thing it needs is an exact matching gwt-dev.jar.  Each permutation is totally independent, so this step could be run across a network on a cluster of machines, for example.

3) Link

Link takes in the Precompilation and all of the output JS files and performs a link.  The output is the familiar GWTCompiler output directory.  Like CompilePerms, it takes very few options.  It will fail if CompilePerms has not been run for every permutation.

To play with it:

Create 3 launch configs; or better yet copy an existing GWTCompiler launch config.  The first copy, don't change anything except make the class to run "Precompile".  The second copy, remove any JJS options except -out and -logLevel and run CompilePerms.  Same with the third copy but run "Link".

Some consequences:

- GWTCompiler is now a simple wrapper around the other three phases.  Its command line behavior should be mostly indistinguishable, sans some altered logging and different temporary files.

- As part of this patch, we now official collapse permutations with identical rebind results.  This collapse is more effective because we only consider 'live' rebinds that are still reachable after preoptimizing the unified AST.  As a consequence, you generally can't be sure how many perms there will be before running the precompile.  Bob, please double check that the way I collapse perms doesn't invalidate any linker assumptions, since wildly different property values could be merged.

- I had to severely refactor how command line options are processed.  In particular, GWTCompiler and GWTShell no longer extend ToolBase, but rather composite it.  This turns out to be good, though, since we have a lot less code duplication.

- I intended for Precompile, CompilePerms, and Link to be public APIs.  This will allow build tool writers to integrate at a very deep level if desired and build arbitrarily sophisticated build tools.  There are basically 3 different APIs you can choose from:

  1) Pass a bunch of arguments to main() and run it like a command line.

  2) Construct an instance with an Options object and call the appropriate instance method.

  3) Call "plain data" static methods.  If you go this route, we write nothing to disk and deal in plain data... you're responsible for the plumbing.  This would be the option you'd go if you wanted to simply remote parts of the process over a network, for example.

PLEASE CAREFULLY REVIEW AND PUSH BACK ON THESE APIS!

Brief summary of changes and motivations:

com.google.gwt.core.ext.linker

1) ArtifactSet is now serializable, which has a trickle-down effect on this package.  I hope I documented well enough why I need to do some of the crazy serialization tricks here, but feel free to call BS.

2) Refactor of StandardLinkerContext to make it more programmable.

com.google.gwt.dev.cfg

- All the changes here are to make Property easily Serializable.

com.google.gwt.jjs

- JavaToJavaScriptCompiler: this isn't as bad as it looks, mostly it's just shuffled around to make the entire class static, since there's really no need for instance data anymore, and to provide a serializable form of all compilation data for per-permutation compiles.

- The rebind-related stuff is to a) only consider "live" rebinds to allow permutation collapsing and b) decouple RebindOracle from the per-permutation compiles to reduce dependenceies.

- "-validateOnly" is no longer a JJSOption, it's just a flag to Precompile (or GWTCompiler).

Everything else is where the real meat of the patch lies, all the reshuffling and refactoring and whatnot.  The actual deltas are a big mess in part, but I hope you'll forgive that; what's important is getting a good end result, so please help ensure we're on track for that!  Last note: I consider this patch more "solid and functional" than "utterly polished"... that's where you guys come in.

Thanks,

Scott