Mavenizing GWT

[Thomas Broyer]

Hi Rajeev [Cc: GWT-Contrib],

I saw you created a BuildingWithMaven wiki page earlier today, which I
suppose is about mavenizing the GWT build process.

I thought about it a little these past weeks.

The major pain point IMO is about dependencies, particularly those
that have been patched: ECJ and Flute come to mind.

As we're talking about mavenization, I'd also like to see artifacts
reorganized to favor reuse (dependency) over duplication:
- common: classes shared between the compiler, server code and user lib
- requestfactory-shared
- requestfactory-client: depends on requestfactory-shared
- requestfactory-server: depends on requestfactory-shared and common
- requestfactory-apt: depends on requestfactory-shared (and maybe common)
- gwt-extension-api: provides the API for generators and linkers
- gwt-compiler: equivalent to the current "compiler.standalone" task
in gwt-dev, depends on common and gwt-extension-api
- gwt-dev: depends on gwt-compiler, contains the dev-mode, equivalent
to the current gwt-dev feature-wise
- gwt-shared: everything shared by gwt-user and gwt-servlet today
- gwt-user: depends on gwt-extension-api and gwt-shared, equivalent
to the current gwt-user feature-wise, minus the server-side code
- gwt-servlet: depends on gwt-shared and common, equivalent to the
current gwt-servlet feature-wise, minus the request-factory stuff

The only breaking changes would be that gwt-servlet no longer contains
request-factory stuff. That shouldn't break much people, and is only a
matter of adding requestfactory-server as an additional dependency.

The gwt-maven-plugin could depend on gwt-compiler only and dynamically
download gwt-dev only if running GWTTestCase or the dev mode (or
depend directly on gwt-dev as it currently does, that's probably
simpler to implement). The split between gwt-compiler and gwt-dev is
not strictly necessary, but as there's already a "compiler.standalone"
task, I suppose some people have a need for it.

I believe the gwt-user, gwt-servlet and gwt-dev could still be made to
conditionally (using a Maven profile) produce all-in-one JARs as we
know them today, for people not using Maven for their projects.

What do you think?

--
Thomas Broyer
/tɔ.ma.bʁwa.je/

[Ray Cromwell]

I agree with most of this, but I just wanted to mention that I think
breaking up the dependency chain will be a little tricky for maven
when it comes to tests. Maven general practice is that a given module
contains its tests and runs them. The problem with gwt-dev and
gwt-user is that many of the gwt-user tests can't run unless gwt-dev
has been built, but gwt-dev IIRC has some tests that depend on
gwt-user. So it may be that you need to make separate
gwt-dev-tests/gwt-user-tests subprojects.

Personally, I'd like to break up GWT-user further into lots tiny
pieces, like Core, I/O, Emulation, RPC, Widgets, DOM/Media/HTML, etc
We can still build an uber gwt-user.jar, but there are lots of
projects that don't use everything, and being able to move stuff off
of the classpath speeds up the compiler and dev-mode. There are some
projects like PlayN for example, that pretty much only use Core.

-Ray

> --
> http://groups.google.com/group/Google-Web-Toolkit-Contributors

[Hilco Wijbenga]

On 16 March 2012 20:26, Ray Cromwell <cromw...@google.com> wrote:
> I agree with most of this, but I just wanted to mention that I think
> breaking up the dependency chain will be a little tricky for maven
> when it comes to tests. Maven general practice is that a given module
> contains its tests and runs them. The problem with gwt-dev and
> gwt-user is that many of the gwt-user tests can't run unless gwt-dev
> has been built, but gwt-dev IIRC has some tests that depend on
> gwt-user.  So it may be that you need to make separate
> gwt-dev-tests/gwt-user-tests subprojects.
>
> Personally, I'd like to break up GWT-user further into lots tiny
> pieces, like Core, I/O, Emulation, RPC, Widgets, DOM/Media/HTML, etc
> We can still build an uber gwt-user.jar, but there are lots of
> projects that don't use everything, and being able to move stuff off
> of the classpath speeds up the compiler and dev-mode. There are some
> projects like PlayN for example, that pretty much only use Core.

Yes, that would be wonderful!

[Thomas Broyer]

On Saturday, March 17, 2012 4:26:01 AM UTC+1, Ray Cromwell wrote:

I agree with most of this, but I just wanted to mention that I think
breaking up the dependency chain will be a little tricky for maven
when it comes to tests. Maven general practice is that a given module
contains its tests and runs them. The problem with gwt-dev and
gwt-user is that many of the gwt-user tests can't run unless gwt-dev
has been built, but gwt-dev IIRC has some tests that depend on
gwt-user.  So it may be that you need to make separate
gwt-dev-tests/gwt-user-tests subprojects.

I've started working on the mavenization, but haven't yet come to tests, as simply breaking up dev-ext is not as easy as it sounds!

First, third-party dependencies:

• ECJ/JDT 3.4.2 is not available as a Maven dependency; maybe it's time to update to a newer version? See http://code.google.com/p/google-web-toolkit/issues/detail?id=3823 and http://code.google.com/p/google-web-toolkit/issues/detail?id=6960
• ICU4J 4.4.2 is not available as a Maven dependency.
• sourcemap is a subset of the closure compiler; it would be great to have a proper artifact that closure-compiler would depend on.
• streamhtmlparser is a subset of jsilver.
• sourcemap, closure-compiler, guava, protobuf and streamhtmlparser have been rebased, so they would have to be either published as com.google.gwt.* Maven artifacts, or possibly jarjar'd as part of the build (as Guice does for ASM, CGLib, Guava and JSR305). I think I'd prefer the latest, as it makes it easier to upgrade to a newer version of the dependency, but it might not work for all of them (e.g. streamhtmlparser)
• There are also forked dependencies: Rhino JS parser, to support JSNI references, and ASM (AFAICT because until ASM 4.0 they didn't provide binary compatibility and there were conflicts with third-party libs –possibly EMMA?–, see http://code.google.com/p/google-web-toolkit/source/detail?r=1962). ASM is not really a fork, and Scott suggested that it had been included into GWT because JarJar wasn't yet used at the time http://gwt-code-reviews.appspot.com/699802 It's been modified a bit since then, but only to fix bugs that I believe (but haven't checked) have been fixed upstream. Now that ASM 4.0 provides binary compatibility (under some conditions), it could be used instead, or we could simply use ASM as a third-party dependency and bundle it with JarJar at build-time (see above).

Next, dependencies within the code:

• most primary linkers in com.google.gwt.core.linker (which I'd likely have bundled in dev-ext) make use of JsToStringGenerationVisitor.javaScriptString. It's easy enough to move the method to com.google.gwt.util.tools.shared.StringUtils to break the dependency between dev-ext and the compiler internals. The JavaScriptStringTest makes use of com.google.gwt.dev.js.rhino.TokenStream though, so I made a rhino-parser-with-jsni artifact out of com.google.gwt.dev.js.rhino, used as a scope=test dependency by dev-ext.
• there are classes in com.google.gwt.core.ext that depend on compiler internals (linker/impl/Standard*, linker/*MetricsArtifact and soyc/**), so they should rather be in gwt-dev than in dev-ext. It would be a breaking change to move them into other packages, so I suggest we just split the package in the two JARs. I'd like SelectionScriptLinker to be part of dev-ext, so excluding c.g.g.core/ext/linker/impl/** is not really an option.
• there are a few dependencies from com.google.gwt.core.linker to com.google.gwt.util (but not util.tools.shared) and com.google.gwt.dev.util; again that might break too many people moving classes around.
• GeneratorContext references ResourceOracle, so there's some tweaking to do in inclusion patterns yet again.
• There are also dependencies from RequestFactory to ASM, but only for 2 classes (Type and Method) and for methods that don't even use ASM bytecode processing. I've started simplifying the code to remove the dependency.

I'm still in the very early stage (dev-ext doesn't even compile yet, so basically I only have common and rhino-parser-with-jsni passing), and I haven't yet looked at the tests dependencies you were talking about.

Personally, I'd like to break up GWT-user further into lots tiny
pieces, like Core, I/O, Emulation, RPC, Widgets, DOM/Media/HTML, etc
We can still build an uber gwt-user.jar, but there are lots of
projects that don't use everything, and being able to move stuff off
of the classpath speeds up the compiler and dev-mode. There are some
projects like PlayN for example, that pretty much only use Core.

Agreed, but I'll postpone that until I have something working ;-)