Discussion on changing gwt release groupid

[Colin Alworth]

We're in the last phases of refactoring out the various GWT modules from the gwt-user.jar each into their own github.com/gwtproject/ repositories and jars, so they can be released separately and managed directly on github rather than only through gerrit. These will be new jars, each with a org.gwtproject.* groupId, and with package structure also following org.gwtproject.*. As of their first version they are built and tested to be compatible with both GWT 2.9 and also J2CL, and will be as close to drop-in replacements for their original classes as possible, save for GWT.create(...) calls and changing package structure in your imports.

With this change nearly complete, Google has requested that the main GWT release no longer come from from them - as long as the groupId includes com.google, the release to maven central or sonatype snapshots can only be initiated by a Googler. When this change is completed, we would not need the CLA any longer, nor would the compiler need to be hosted on gerrit (though I assume that we would be welcome to continue there?). Google has indicated that we do not need to change package structures within these jars, and just releasing them on their own groupId is sufficient.

The primary complication this will add is for large, long-lived projects, using multiple parts of the GWT ecosystem: whereas adding a new dependency to a tree with a conflicting version is expected and handled (maven will ensure only one groupId+artifactId is present on the classpath, selecting the "nearest" version to your project in the tree, while gradle will pick the "highest" version, and ivy is configurable), adding a new dependency with a different _groupId_ will retain both classpath entries, which can clearly cause conflicts. Historically, we have seen issues with this kind of conflict, mostly between gwt-dev and gwt-user, but this change is likely to cause two conflicting versions of gwt-dev or two versions of gwt-user, in addition to conflicting gwt-dev/gwt-user.

One proposed solution is that we just switch from GWT 2.9 being com.google.gwt to GWT 2.10 being org.gwtproject. Resolution for this would occur within each project that uses GWT - if you deliberately stay with com.google.gwt, any dependency that gets an update could need an explicit exclusion added to it to prevent it from bringing in any org.gwtproject dependencies. Any missed exclusion on a dependency could result in familiar mixed-classpath errors like we sometimes see when gwt-user and gwt-dev disagree, though instead of mitigating by requiring that projects list both of these artifacts, every single dependency would need to apply exclusions to it.

Another solution could be to release 2.9.0 again under the new groupId, and likewise provide the next several releases under both groupIds. This would have the advantage of allowing a project to list both groupIds to manage all dependencies to the same version, but does require that Google continue to publish releases for the duration of some transition period.

A similar solution that I can see would be to use the "relocation" feature in maven, though I've rarely seen it used, and there is apparently some ongoing discussion as to whether it should be used. This would enable Google to release a new version which only tells Maven to switch to the org.gwtproject release instead - downstream projects and libraries would manage this in the same way as the second solution. Depending on the specifics, Google might release a matching trivial relocation "release" for each org.gwtproject release for a transition period, or might just release one time and be done. More details on the feature: https://maven.apache.org/guides/mini/guide-relocation.html

Are there other technical considerations I'm missing as to how this might be achieved in the most painless way possible?

[Jens]

I know you can force Gradle to swap out dependencies on the fly, e.g. replace com.google.gwt with org.gwtproject releases. If that would also be possible with Maven/Ivy/Bazel then it is just a matter of documentation.

If that is not possible, or not desired, then Google could probably publish a single, empty com.google.gwt:..:99.0 release without any dependencies in its pom. Our new published org.gwtproject:...:2.10 release would depend on com.google.gwt:...:99.0 which means all old GWT libraries should be upgraded via Maven/etc to 99.0 automatically as soon as you add org.gwtproject:...:2.10 to your dependencies. Since the 99.0 release is empty and the classes are provided by our new release, there should not be any class path conflicts. New GWT libraries would directly depend on our new release which means as soon as you try to migrate your code to one of the new GWT libs, our new org.gwtproject release will be pulled in and provide the code for everything.

[Thomas Broyer]

In addition to the relocation for gwt-dev and gwt-user (and other JARs), the BOM (org.gwtproject:gwt) might then list both the org.gwtproject *and* the "relocated" com.google.gwt.

How I think that would work out: people switch to using the org.gwtproject:gwt:2.10.0 BOM in their <dependencyManagement> with <scope>import</scope>, that would then import the dependency management rules that com.google.gwt:gwt-user should now be using 2.10.0 (and "nearest version" should now use that version), and of course org.gwtproject:gwt:2.10.0; and they'd switch their <dependencies> to org.gwtproject too. Now, if they have a dependency that references com.google.gwt:gwt-user:2.9.0, dependency management should actually resolve it to 2.10.0, which is relocated to org.gwtproject:gwt-user, so you should end up with a single JAR in your classpath.

So I would go with Google pushing a single 2.10.0 with relocation, and us pushing an org.gwtproject whose BOM references com.google.gwt in addition to org.gwtproject. I think the BOM could continue referencing com.google.gwt 2.10.0 even for later versions (same as above: com.google.gwt:gwt-user:2.9.0 would be "upgraded" to 2.10.0 by dependency management from the BOM, which relocates to org.gwtproject:gwt-user:2.10.0, but dependency management "upgrades" it in turn to org.gwtproject:gwt-user:2.11.0)

This should all be testable locally by tweaking the local repository (as explained in the mini guide).

[Colin Alworth]

@Thomas, it sounds like you think relocation should be well supported then? My primary concern was the quote on the linked page, but if this is well supported, then I think we're on the same page. From the linked page:

2020 rework in progress, see discussion on dev mailing list, still need analysis of issues, definition of improvements, and of course implementation...

I'd also suggest going one step further and release a o.g:*:2.9.0 which just has a relocation to point at c.g.g:*:2.9.0, and a BOM as you describe, so that projects can begin migrating right away, even before the first org.gwtproject-only release. We don't even necessarily need to wait until 2.10 this way, but other 2.9.x releases could move to gwtproject, and just have some canned maven files ready to have Google ship following each release (for some limited window of time/versions)?

I do think we need to push more than just the BOM (you may not have been suggesting this) for c.g.g releases, as there are plenty of projects out there not using the BOM.

Just to be sure I am clear, you are suggesting that o.g BOM includes o.g dependencies as well as c.g.g, while the c.g.g BOM would only reference c.g.g dependencies (which themselves would be relocated to o.g anyway). If so, I think I see a conflict here:

• SomeLibrary depends on c.g.g:gwt-user:2.9.0 or earlier, with or without BOM
  
• MyProject depends on o.g:gwt-user:2.10.0 or later, without a BOM, and also on SomeLibrary

Without a BOM, and without o.g:gwt-user:2.10.0 depending on some empty c.g.g:gwt-user (and making that conflict apparent), the project would end up with both old and new gwt-user on the classpath at the same time.

@Jens

Our new published org.gwtproject:...:2.10 release would depend on com.google.gwt:...:99.0 which means all old GWT libraries should be upgraded via Maven/etc to 99.0 automatically as soon as you add org.gwtproject:...:2.10 to your dependencies.

I suspect this will not work except in gradle, which picks the highest version in the case of a conflict. Maven picks the "nearest to your project", so:

• SomeLibrary depends on c.g.g:gwt-user:2.9.0 (or earlier, with or without a BOM)
  
• MyProject depends on SomeLibrary (so c.g.g:gwt-user is now 2 steps away)
• MyProject depends on o.g:gwt-user:2.10 without a BOM, which depends on c.g.g:gwt-user:99.0 (which is also 2 steps away)

It isn't clear to me which will win automatically. As above, we need the user to use a BOM if they update to the org.gwtproject implementation to work reliably, or for the ecosystem to be ahead of applications, and push updates to long-stable releases. Which, for some of these will require more relocations, work by Google (which is what we're trying to avoid in the first place) such as for Gin, GwtQuery, possibly others like GwtMockito.

I feel like I might be being dense here, is this cleaner than I'm imagining it? I'll try spinning up a few sample projects and artifacts with dummy groupIds to test with as time permits for some concrete experiments.

[Jens]

I suspect this will not work except in gradle, which picks the highest version in the case of a conflict. Maven picks the "nearest to your project", so:

• SomeLibrary depends on c.g.g:gwt-user:2.9.0 (or earlier, with or without a BOM)
  
• MyProject depends on SomeLibrary (so c.g.g:gwt-user is now 2 steps away)
• MyProject depends on o.g:gwt-user:2.10 without a BOM, which depends on c.g.g:gwt-user:99.0 (which is also 2 steps away)

Hmm, so MyProject would need to declare c.g.g:gwt-user:99.0 as a direct dependency as soon as it wants to use o.g:gwt-user:2.10 to make sure the "distance" is 1. A bit unfortunate but doable when clearly communicated in documentation. But maybe that relocation solution is more straight forward then for the end user.

[Michael S.]

Hi,

An additional groupID Release sound like a valid part. The Maven reloaction feature should normally work but so existing applications can easier migrate to newer groupID.

This strategy is also use for the migration from javax.* to jakarta* I think.

There stragy is:

1. All libs are cloned in new groupId

2. Update to newer Version only the new Group Ids (with old package structure)

3. rename packages

4. Release new Features

Maybe this is also usable for GWT migration

[Thomas Broyer]

On Thursday, June 11, 2020 at 10:19:00 PM UTC+2, Colin Alworth wrote:

@Thomas, it sounds like you think relocation should be well supported then? My primary concern was the quote on the linked page, but if this is well supported, then I think we're on the same page. From the linked page:

2020 rework in progress, see discussion on dev mailing list, still need analysis of issues, definition of improvements, and of course implementation...

I'd also suggest going one step further and release a o.g:*:2.9.0 which just has a relocation to point at c.g.g:*:2.9.0, and a BOM as you describe, so that projects can begin migrating right away, even before the first org.gwtproject-only release. We don't even necessarily need to wait until 2.10 this way, but other 2.9.x releases could move to gwtproject, and just have some canned maven files ready to have Google ship following each release (for some limited window of time/versions)?

I'm worried that this two way relocation (o.g:*:2.9.0 relocated to c.g.g:*:2.9.0, then c.g.g:*:2.10.0 relocating "back" to o.g:*:2.10.0) might cause more problems than it solves.

Also, referencing a relocating POM will print a warning in the console (at least for Maven, AFAICT) so you don't actually want to use o.g before it actually exists.

 

I do think we need to push more than just the BOM (you may not have been suggesting this) for c.g.g releases, as there are plenty of projects out there not using the BOM.

Just to be sure I am clear, you are suggesting that o.g BOM includes o.g dependencies as well as c.g.g, while the c.g.g BOM would only reference c.g.g dependencies (which themselves would be relocated to o.g anyway).

I would have had the c.g.g BOM relocate as well, actually.

But first and foremost, I'm suggesting someone tests this (and/or variants) within his local Maven repository (or even better, some actual Maven repository accessed through HTTP; should be as easy as copy/pasting some files in another directory and running a simple web server there).

 

If so, I think I see a conflict here:

• SomeLibrary depends on c.g.g:gwt-user:2.9.0 or earlier, with or without BOM
  
• MyProject depends on o.g:gwt-user:2.10.0 or later, without a BOM, and also on SomeLibrary

Without a BOM, and without o.g:gwt-user:2.10.0 depending on some empty c.g.g:gwt-user (and making that conflict apparent), the project would end up with both old and new gwt-user on the classpath at the same time.

Absolutely. When migrating to o.g, you would have to add a dependencyManagement to upgrade c.g.g to the relocating version, either right in your POM or through the o.g BOM.

Indeed we could make o.g:gwt-user:2.10.0 have a dependency on an empty c.g.g:gwt-user:2.10.0 (but then, not relocating, or you'll have warnings AFAICT) but that wouldn't even work in all cases with Maven, depending on the order the dependencies to SomeLibrary and gwt-user are declared in MyProject. And if we can't have a c.g.g:gwt-user:2.10.0 that relocates to to o.g, then it's going to surprise people who automatically update to c.g.g:gwt-user:2.10.0 following some tooling advice and don't understand why the JAR is empty.

One thing we could possibly do to detect such cases would be to add some code the Main classes (Compiler, DevMode, CodeServer, JUnitShell) that would look for some duplicate resource on the classpath, or even look for all the com/google/gwt/dev/About.properties content and warn or fail if it finds differing gwt.version properties in there (unfortunately this is in gwt-dev which has fewer risks of being duplicated than gwt-user).

So, my plan would be:

• test relocation, without overthinking it, and detecting limitations; then we can refine the plan
• add a check to detect duplicated gwt-user on the classpath when running GWT tools

[Colin Alworth]

Yep, sounds like the test stuff I had in mind - for a quick demo I'll set up a repo, put some "libraries" and "gwt" jars/boms into it, and then make a bunch of sample projects. The "gwt jars" will have some service loader wiring, and the sample projects will check to see which jars they end up with, so that we can see which configurations cause which surprises.

Roughly, I'll deploy

 * LibraryWithNewGwt

 * LibraryWithOldGwt

 * LibraryWithBomAndNewGwt

 * LibraryWithBomAndOldGwt

 * LibraryWithTransitiveOldGwt (two flavors of this, one for each of the old gwt above)

 * LibraryWithTransitiveNewGwt (again, two flavors)

then try to build projects with old or new gwt, plus 1-2 of the libraries above. If a project ever detects more than one gwt version, we'll fail that, and go from there.

I'm worried that this two way relocation (o.g:*:2.9.0 relocated to c.g.g:*:2.9.0, then c.g.g:*:2.10.0 relocating "back" to o.g:*:2.10.0) might cause more problems than it solves.

Also, referencing a relocating POM will print a warning in the console (at least for Maven, AFAICT) so you don't actually want to use o.g before it actually exists.

Good point - perhaps we just push a o.g:gwt:2.9.0 bom then, and encourage switching to just that? Same idea as before - projects can future proof themselves and move to this BOM, then only have to change its version in the BOM declaration to be correctly updated. This does miss out on some of the benefits though. Will hold off exploring until the above experiment has some results to share.

One thing we could possibly do to detect such cases would be to add some code the Main classes (Compiler, DevMode, CodeServer, JUnitShell) that would look for some duplicate resource on the classpath, or even look for all the com/google/gwt/dev/About.properties content and warn or fail if it finds differing gwt.version properties in there (unfortunately this is in gwt-dev which has fewer risks of being duplicated than gwt-user).

So, my plan would be:

• test relocation, without overthinking it, and detecting limitations; then we can refine the plan
  
• add a check to detect duplicated gwt-user on the classpath when running GWT tools
  

Yeah, I was thinking about this - It would be nice if we could go back in time and add some nice version sentinel  that can be scanned for across all jars, but About.properties is as good as anything. Confirming gwt-dev is current doesn't tell you much (plenty of libraries only use gwt-user), but hopefully it will catch the important cases. Probably should augment the above samples to have another axis: gwt-user, or gwt-user+gwt-dev on each library.

[Thomas Broyer]

Fwiw, I started setting up some tests here: https://github.com/tbroyer/gwt-relocation-tests

Feel free to contribute issues or pull requests.

For now, Maven actually fails because the repository only includes POMs and not JARs. I'll add empty JARs soon.

[Colin Alworth]

Nice, I have something very similar. My main finding is putting relocation in the BOM doesn't work, unless you _also_ include the previous version's dependencyManagement tag, so that it tells the projects which include the BOM "please update c.g.g" instead of just "relocate to o.g, which will say please include o.g", as that will skip the c.g.g version bump.

Will push shortly, compare notes.

-- 

  Colin Alworth

  co...@colinalworth.com

--

You received this message because you are subscribed to the Google Groups "GWT Contributors" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-web-toolkit-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-web-toolkit-contributors/ef206a74-1c02-421b-a71a-3af43da0bf8ao%40googlegroups.com.

[Thomas Broyer]

On Sunday, June 14, 2020 at 10:07:48 PM UTC+2, Colin Alworth wrote:

Nice, I have something very similar. My main finding is putting relocation in the BOM doesn't work, unless you _also_ include the previous version's dependencyManagement tag, so that it tells the projects which include the BOM "please update c.g.g" instead of just "relocate to o.g, which will say please include o.g", as that will skip the c.g.g version bump.

Indeed.

That's what I was suggesting in https://groups.google.com/d/msg/google-web-toolkit-contributors/L2RMqglOEXo/fBGwNB2kCQAJ

“In addition to the relocation for gwt-dev and gwt-user (and other JARs), the BOM (org.gwtproject:gwt) might then list both the org.gwtproject *and* the "relocated" com.google.gwt.”

[Colin Alworth]

Agreed, I was testing to confirm this. It appears to not make a difference in the samples I have so far if that BOM includes the relocation though, but there are a lot of permutations to go through, I'm mostly automating the "easier" ones at this time.

-- 

  Colin Alworth

  co...@colinalworth.com

--

You received this message because you are subscribed to the Google Groups "GWT Contributors" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-web-toolkit-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-web-toolkit-contributors/4f853f49-167b-4635-b185-9ae9d48948b5o%40googlegroups.com.

[Colin Alworth]

My repo of tests, with some notes on problems it has encountered while testing https://github.com/Vertispan/gwt-groupid-relocation-test

-- 

  Colin Alworth

  co...@colinalworth.com

[Thomas Broyer]

FYI, I've made a couple more tests, and added the results to the README: https://github.com/tbroyer/gwt-relocation-tests

Unsurprisingly, the "dumb" resolution rules ("nearest definition") of Maven makes it irrecoverable for projects still on c.g.g that depend on libraries transitively bringing in o.g (they'll have a mix of both, unless they use exclusions); whereas for Gradle some things can be done to make it work the same as if everything was still on c.g.g.

The one last test I'd like to do is the one you suggested, with an o.g:*:2.9.0 relocating to c.g.g:*:2.9.0. That could make it possible for Maven, in those above cases, to downgrade all o.g:*:2.10.0 to c.g.g:*:2.9.0 in one go, through a dependencyManagement rule (or switching to the o.g:gwt:2.9.0 BOM instead of c.g.g:gwt:2.9.0), rather than chasing every transitive and adding exclusions everywhere.

On Sunday, June 14, 2020 at 11:27:45 PM UTC+2, Colin Alworth wrote:

My repo of tests, with some notes on problems it has encountered while testing https://github.com/Vertispan/gwt-groupid-relocation-test

-- 

  Colin Alworth

  co...@colinalworth.com

On Sun, Jun 14, 2020, at 3:21 PM, Colin Alworth wrote:

Agreed, I was testing to confirm this. It appears to not make a difference in the samples I have so far if that BOM includes the relocation though, but there are a lot of permutations to go through, I'm mostly automating the "easier" ones at this time.

-- 

  Colin Alworth

  co...@colinalworth.com

On Sun, Jun 14, 2020, at 3:16 PM, Thomas Broyer wrote:

On Sunday, June 14, 2020 at 10:07:48 PM UTC+2, Colin Alworth wrote:

Nice, I have something very similar. My main finding is putting relocation in the BOM doesn't work, unless you _also_ include the previous version's dependencyManagement tag, so that it tells the projects which include the BOM "please update c.g.g" instead of just "relocate to o.g, which will say please include o.g", as that will skip the c.g.g version bump.

Indeed.

That's what I was suggesting in https://groups.google.com/d/msg/google-web-toolkit-contributors/L2RMqglOEXo/fBGwNB2kCQAJ

“In addition to the relocation for gwt-dev and gwt-user (and other JARs), the BOM (org.gwtproject:gwt) might then list both the org.gwtproject *and* the "relocated" com.google.gwt.”

--

You received this message because you are subscribed to the Google Groups "GWT Contributors" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-web-toolkit-contributors+unsubscribe@googlegroups.com.

[Thomas Broyer]

On Monday, June 15, 2020 at 7:44:34 PM UTC+2, Thomas Broyer wrote:

FYI, I've made a couple more tests, and added the results to the README: https://github.com/tbroyer/gwt-relocation-tests

Unsurprisingly, the "dumb" resolution rules ("nearest definition") of Maven makes it irrecoverable for projects still on c.g.g that depend on libraries transitively bringing in o.g (they'll have a mix of both, unless they use exclusions); whereas for Gradle some things can be done to make it work the same as if everything was still on c.g.g.

The one last test I'd like to do is the one you suggested, with an o.g:*:2.9.0 relocating to c.g.g:*:2.9.0. That could make it possible for Maven, in those above cases, to downgrade all o.g:*:2.10.0 to c.g.g:*:2.9.0 in one go, through a dependencyManagement rule (or switching to the o.g:gwt:2.9.0 BOM instead of c.g.g:gwt:2.9.0), rather than chasing every transitive and adding exclusions everywhere.

That worked great; but then I remembered dependency constraints are transitive in Gradle, contrary to dependencyManagement in Maven, and are published in Gradle Module Metadata; so this needs more tests with Gradle.

Also, if people know and use Ivy, SBT, Buildr, etc. it would be good to test with those, just to make sure (SBT seems to be using Ivy, or optionally Aether –the same as Maven– through a plugin).

There's also Coursier, which is used by Bazel's rules_jvm_external.