Hi Bob, Daniel,
The massive attached patch really has one purpose: enable GWT compiles to
run across multiple processes or machines.
Algorithm outline:
1) Precompile.
Produces a "Precompilation" data structure, which is serializable. This
Precompilation contains:
- a unified (all permutations), preoptimized AST where no deferred binding
decisions have yet been mode
- the JJSOptions which produced the AST; these are not modifiable by
subsequent compiler passes
- produce metadata regarding the number of permutations, and what the
deferred binding decisions are for each permutation
- generated artifacts
When Precompile is run from the command line, it produces a
"precompilation.ser" (under .gwt-tmp) which contains the Java serialized
data. It also generates a file "permCount.txt" which simply contains an
integer of the number of permutations; this is intended to facilitate build
tools dynamically determining the number of permutations for sharding.
Precompile takes exactly the same set of command line flags that GWTCompiler
does.
2) CompilePerms
Compile one or more permutations into JavaScript. The input is the
"Precompilation" (but the generated artifacts are ignored during this step).
The output is a String JS file. When run from the command line, the
default is to compile all perms, but the "-perms" option lets you specify
exactly which ones to compile. The output files follow the scheme
"permutation-0.js", "permutation-1.js".
CompilePerms takes a greatly restricted set of command line options, since
the compiler options passed into the Precompile are preserved, and things
like '-gen' no longer matter.
CompilePerms does not depend on the user's project being available on the
classpath; all of the dependencies are serialized. The only thing it needs
is an exact matching gwt-dev.jar. Each permutation is totally independent,
so this step could be run across a network on a cluster of machines, for
example.
3) Link
Link takes in the Precompilation and all of the output JS files and performs
a link. The output is the familiar GWTCompiler output directory. Like
CompilePerms, it takes very few options. It will fail if CompilePerms has
not been run for every permutation.
To play with it:
Create 3 launch configs; or better yet copy an existing GWTCompiler launch
config. The first copy, don't change anything except make the class to run
"Precompile". The second copy, remove any JJS options except -out and
-logLevel and run CompilePerms. Same with the third copy but run "Link".
Some consequences:
- GWTCompiler is now a simple wrapper around the other three phases. Its
command line behavior should be mostly indistinguishable, sans some altered
logging and different temporary files.
- As part of this patch, we now official collapse permutations with
identical rebind results. This collapse is more effective because we only
consider 'live' rebinds that are still reachable after preoptimizing the
unified AST. As a consequence, you generally can't be sure how many perms
there will be before running the precompile. Bob, please double check that
the way I collapse perms doesn't invalidate any linker assumptions, since
wildly different property values could be merged.
- I had to severely refactor how command line options are processed. In
particular, GWTCompiler and GWTShell no longer extend ToolBase, but rather
composite it. This turns out to be good, though, since we have a lot less
code duplication.
- I intended for Precompile, CompilePerms, and Link to be public APIs. This
will allow build tool writers to integrate at a very deep level if desired
and build arbitrarily sophisticated build tools. There are basically 3
different APIs you can choose from:
1) Pass a bunch of arguments to main() and run it like a command line.
2) Construct an instance with an Options object and call the appropriate
instance method.
3) Call "plain data" static methods. If you go this route, we write
nothing to disk and deal in plain data... you're responsible for the
plumbing. This would be the option you'd go if you wanted to simply remote
parts of the process over a network, for example.
PLEASE CAREFULLY REVIEW AND PUSH BACK ON THESE APIS!
Brief summary of changes and motivations:
com.google.gwt.core.ext.linker
1) ArtifactSet is now serializable, which has a trickle-down effect on this
package. I hope I documented well enough why I need to do some of the crazy
serialization tricks here, but feel free to call BS.
2) Refactor of StandardLinkerContext to make it more programmable.
com.google.gwt.dev.cfg
- All the changes here are to make Property easily Serializable.
com.google.gwt.jjs
- JavaToJavaScriptCompiler: this isn't as bad as it looks, mostly it's just
shuffled around to make the entire class static, since there's really no
need for instance data anymore, and to provide a serializable form of all
compilation data for per-permutation compiles.
- The rebind-related stuff is to a) only consider "live" rebinds to allow
permutation collapsing and b) decouple RebindOracle from the per-permutation
compiles to reduce dependenceies.
- "-validateOnly" is no longer a JJSOption, it's just a flag to Precompile
(or GWTCompiler).
Everything else is where the real meat of the patch lies, all the
reshuffling and refactoring and whatnot. The actual deltas are a big mess
in part, but I hope you'll forgive that; what's important is getting a good
end result, so please help ensure we're on track for that! Last note: I
consider this patch more "solid and functional" than "utterly polished"...
that's where you guys come in.
Thanks,
Scott
|
