Classloader issues for server-side code in DevMode with GWT 2.6

[Thomas Broyer]

Hi all,

The update from Jetty 6 to Jetty 8 wasn't without regressions. One of them is a classloader issue.

First, when starting DevMode, sometimes (IIRC, not for all projects, might be because I tried with a WEB-INF/jetty-web.xml), it starts with:

[WARN] Server resource 'org/eclipse/jetty/xml/configure_6_0.dtd' could not be found in the web app, but was found on the system classpath

   [WARN] Adding classpath entry 'file:/home/tbr/.m2/repository/com/google/gwt/gwt-dev/2.6.0/gwt-dev-2.6.0.jar' to the web app classpath for this session

Which is… bad! (it's gwt-dev.jar here!)

and is btw probably the cause of https://code.google.com/p/google-web-toolkit/issues/detail?id=8526 (the other known regression)

Next, there are issues with javax.* APIs, such as JDO: https://code.google.com/p/google-web-toolkit/issues/detail?id=8585

This is because Jetty switched isSystemClass from "javax.servlet + javax.xml" to "all of javax.":

http://grepcode.com/file/repo1.maven.org/maven2/org.mortbay.jetty/jetty/6.1.11/org/mortbay/jetty/webapp/WebAppContext.java#95

vs. http://grepcode.com/file/repo1.maven.org/maven2/org.eclipse.jetty.aggregate/jetty-all/8.1.12.v20130726/org/eclipse/jetty/webapp/WebAppContext.java#112

We already have issues with the way we do class loading in the webapp in DevMode, e.g. https://code.google.com/p/google-web-toolkit/issues/detail?id=4649

In the end, I wonder if we shouldn't just basically revert https://code.google.com/p/google-web-toolkit/source/detail?r=4944, except issuing a warning when we find the class in the system classpath (but without automatically adding the JAR to the classpath and instead just saying they should move it to WEB-INF/lib). Maybe we could add a switch (in the form of a system property) to allow loading from the system classloader (not restricted to Jetty system and server classes).

What do you think?

[Thomas Broyer]

On Sunday, March 9, 2014 1:09:23 AM UTC+1, Thomas Broyer wrote:

Hi all,

The update from Jetty 6 to Jetty 8 wasn't without regressions. One of them is a classloader issue.

First, when starting DevMode, sometimes (IIRC, not for all projects, might be because I tried with a WEB-INF/jetty-web.xml), it starts with:

[WARN] Server resource 'org/eclipse/jetty/xml/configure_6_0.dtd' could not be found in the web app, but was found on the system classpath

   [WARN] Adding classpath entry 'file:/home/tbr/.m2/repository/com/google/gwt/gwt-dev/2.6.0/gwt-dev-2.6.0.jar' to the web app classpath for this session

Which is… bad! (it's gwt-dev.jar here!)

and is btw probably the cause of https://code.google.com/p/google-web-toolkit/issues/detail?id=8526 (the other known regression)

Tried the "Hello" sample, it starts with:

[WARN] Server class 'org.eclipse.jetty.servlet.listener.ELContextCleaner' could not be found in the web app, but was found on the system classpath

   [WARN] Adding classpath entry 'file:/C:/Users/MRO/Downloads/gwt-2.6.0/gwt-2.6.0/gwt-dev.jar' to the web app classpath for this session

   For additional info see: file:/C:/Users/MRO/Downloads/gwt-2.6.0/gwt-2.6.0/doc/helpInfo/webAppClassPath.html

[Jens]

In the end, I wonder if we shouldn't just basically revert https://code.google.com/p/google-web-toolkit/source/detail?r=4944, except issuing a warning when we find the class in the system classpath (but without automatically adding the JAR to the classpath and instead just saying they should move it to WEB-INF/lib). Maybe we could add a switch (in the form of a system property) to allow loading from the system classloader (not restricted to Jetty system and server classes).

What do you think?

IMHO embedded jetty should behave like any other production server. If something is missing in WEB-INF/lib and is not provided by the server itself it should fail. And if people have production servers that provide additional API's (that shouldn't be in WEB-INF/lib) they should use -noserver instead.

-- J.

[Goktug Gokdogan]

To be honest I don't even understand full extend of the problem, I mostly followed what is already there. We don't use most of this stuff internally.

I trust you making the best choice for external users and I'm fine with it as long as it doesn't break internal users.

--
http://groups.google.com/group/Google-Web-Toolkit-Contributors
---
You received this message because you are subscribed to the Google Groups "GWT Contributors" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-web-toolkit-co...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Thomas Broyer]

As I was writing a memo about the issue, it happened to me that there could be an easy workaround: special-case org.eclipse.jetty. classes to allow loading them from the system classloader as a fallback, without adding the containing classpath entry (most likely gwt-dev.jar) to the webapp classloader.

That said, given that some users don't understand the warning (https://groups.google.com/d/topic/google-web-toolkit/GSKlMwFKh3c/discussion), I'm tempted to refactor the classloading logic anyway, but possibly for GWT.next (leaving only the small workaround in 2.6.1).

BTW, I think we also need to refactor the InstalledHelpInfo to possibly link to the same files hosted at gwtproject.org; the issue here being that there could be a discrepancy between the version of GWT being used and the doc hosted at gwtproject.org (which would always be for the latest stable version; what about RCs? trunk/master? we could possibly just have a header in the docs saying for which version they apply to, and that they might not apply to other versions of GWT). The problem with InstalledHelpInfo as done currently is that it only links to additional doc if you use the SDK, but nowadays many people use dependency managers (Maven, Gradle, Ivy, etc.) so they don't have the additional info that could help them better understand the problem(s).

To unsubscribe from this group and stop receiving emails from it, send an email to google-web-toolkit-contributors+unsubscribe@googlegroups.com.

[gslender]

Does any of the changes in 2.6 in relation to the Jetty server and class loaders mentioned here impact users who are building GWT apps with the GAE dev server? I've yet to fully explore 2.6 with the current GAE releases, and haven't really followed the issue you've outlined here either, but wanted to know how internally the Google team members manage inter-op between GAE and GWT in this area (and to the same extent the Google Eclipse tools).

Thanks

Grant

[Thomas Broyer]

It does not apply to GAE as they just hook into DevMode extension points replacing the JettyLauncher with their own: https://code.google.com/p/googleappengine/source/browse/trunk/java/src/main/com/google/appengine/tools/development/gwt/AppEngineLauncher.java

[gslender]

Thanks Thomas. Can you also explain how things operate for GAE when and if I were to employ SuperDev mode - I thought there would be no impact as the GWT module is loaded by redirecting the HTML script to the compiler web server, but when I looked at this before I ran into some issues where the "ip address" serving the hosted page (in GAE) needed to match the compiler server (SuperDev Mode) and until I was using the same "localhost" it wouldn't work.

Do you know if much sociability testing between 2.6 and GAE is occurring? If I run into issues should I post them with GAE or GWT? Who would be best/fastest to address them?

G

[Thomas Broyer]

On Friday, March 14, 2014 6:03:15 AM UTC+1, gslender wrote:

Thanks Thomas. Can you also explain how things operate for GAE when and if I were to employ SuperDev mode

Heavily depends whether you're using GWT-RPC or not.

 

- I thought there would be no impact as the GWT module is loaded by redirecting the HTML script to the compiler web server, but when I looked at this before I ran into some issues where the "ip address" serving the hosted page (in GAE) needed to match the compiler server (SuperDev Mode) and until I was using the same "localhost" it wouldn't work.

Yes, this is a security measure added in 2.6.0. It's a bit hidden in the release notes:

• Security: with 2.6 we believe it's safe to turn on the Super Dev Mode hook and leave it on in production. As an extra precaution, we recommend setting the devModeUrlWhitelistRegexp configuration property to ensure that it can only load JavaScript from localhost and your developers' machines in your own domain.

 

Do you know if much sociability testing between 2.6 and GAE is occurring?

I don't think so; from our part at least. We do try to at least try the samples before doing a (non-bugfix) release but that's all.

There's a reason for doing release candidates: crowd-sourcing the testing! ;-)

 

If I run into issues should I post them with GAE or GWT? Who would be best/fastest to address them?

If you run into issues with GWT, then post in the GWT users group https://groups.google.com/d/forum/google-web-toolkit (or issue tracker if you believe it's a bug)

If we determine this is due to GAE, then you could post/report there.

Of course, if it's obvious that it comes from using GAE, then go directly to GAE groups/issue trackers ;-)

[Thomas Broyer]

+1 The problem is determining what "provided by the server" actually means though. Servers generally just use their classpath, but in our case the classpath generally (because of Eclipse, and because of how DevMode loads client code too) contains the classes that also are in WEB-INF/lib, so delegation to the "server classpath" just won't work (or rather, it'll work "too well", and won't detect that "something is missing in WEB-INF/lib".

I've written a memo on the issue, and possible solutions: https://docs.google.com/document/d/1bWfafaPA0m0Z1Swodnx7m3QTv31OdqFkE7aeadN_2BU/edit?usp=sharing

It's still work in progress re. the solutions.

I have a sent a patch re. the first solution: https://gwt-review.googlesource.com/6651 Still testing it a bit but sounds promising for inclusion in 2.6.1.

[Jens]

+1 The problem is determining what "provided by the server" actually means though. Servers generally just use their classpath, but in our case the classpath generally (because of Eclipse, and because of how DevMode loads client code too) contains the classes that also are in WEB-INF/lib, so delegation to the "server classpath" just won't work (or rather, it'll work "too well", and won't detect that "something is missing in WEB-INF/lib".

I've written a memo on the issue, and possible solutions: https://docs.google.com/document/d/1bWfafaPA0m0Z1Swodnx7m3QTv31OdqFkE7aeadN_2BU/edit?usp=sharing

Wouldn't it be a ton easier to just launch Jetty as a separate process using ProcessBuilder when DevMode is launched? Of course this new process should not have DevMode class path. Then its pretty much like -noserver but users do not have to setup a jetty server themselves.

Also I am curious why r4944 says its too much of a burden to get the class path right and why the old behavior causes integration problems. If class path is wrong and GWT "repairs" it, it will fail later in production. I don't really see the benefit. But I have to say that I always use -noserver so maybe I am missing something.

-- J.

[Thomas Broyer]

On Mon, Mar 17, 2014 at 1:54 AM, Jens <jens.ne...@gmail.com> wrote:

+1 The problem is determining what "provided by the server" actually means though. Servers generally just use their classpath, but in our case the classpath generally (because of Eclipse, and because of how DevMode loads client code too) contains the classes that also are in WEB-INF/lib, so delegation to the "server classpath" just won't work (or rather, it'll work "too well", and won't detect that "something is missing in WEB-INF/lib".

I've written a memo on the issue, and possible solutions: https://docs.google.com/document/d/1bWfafaPA0m0Z1Swodnx7m3QTv31OdqFkE7aeadN_2BU/edit?usp=sharing

Wouldn't it be a ton easier to just launch Jetty as a separate process using ProcessBuilder when DevMode is launched? Of course this new process should not have DevMode class path. Then its pretty much like -noserver but users do not have to setup a jetty server themselves.

Relatively easy to do with the GWT SDK (just put a jetty.jar in there and use that as the single entry in a root URLClassLoader, don't even need ProcessBuilder), much less elsewhere (Maven, Gradle, Ivy, etc.).

BTW, this is more or less what the AppEngine launcher does currently (IIUC), setting up the classpath from known JARs from the AppEngine SDK.

 

Also I am curious why r4944 says its too much of a burden to get the class path right and why the old behavior causes integration problems. If class path is wrong and GWT "repairs" it, it will fail later in production. I don't really see the benefit.

As a user, having a big warning that I should move a JAR but can continue to work is probably better than having to restart DevMode (which could be slow, particularly back then). Not all users understand the warning though (as we recently saw in the gwt user forum; link in the memo), so breaking with a good error message would maybe be a better choice in the end.

Possibly due to Google Plugin for Eclipse too: Eclipse probably makes it very easy to use the project's build path as the DevMode classpath but maybe difficult to test upfront which JAR should be in WEB-INF/lib; not to mention that the project build path probably mixes client-side and server-side dependencies.

IIRC we were also migrating from the Tomcat-based HostedMode where everything was loaded from the classpath (no "war" at all, static resources such as your host page had to come from "public sources" of your module(s), servlets were defined in the gwt.xml).

This is only speculation though.

 

But I have to say that I always use -noserver so maybe I am missing something.

Same here.

I use DevMode's embedded server only for simple projects like issue repro cases; but all my projects are Maven-driven (maybe soon Gradle-driven) so it's very easy to get things correctly setup in WEB-INF/

We're "power users", Jens; I think newbies (people learning all of GWT, java webapps, servlets and even web dev and/or java) can easily mess everything up, because of the steep learning curve. We probably also need to update our docs (particularly start adding things about Maven and Gradle, and how to use them with Eclipse or IntelliJ IDEA –or Netbeans?)

.

--
Thomas Broyer
/tɔ.ma.bʁwa.je/

[Stephen Haberman]

Hi Thomas,

> (or rather, it'll work "too well", and won't detect that "something is
> missing in WEB-INF/lib".

I'm attempting to follow along, but classloader semantics are "fun", so
apologies if I'm wrong somewhere...

So, to clarify, if I have a dependency, say foo.jar, that isn't GWT, and
isn't Jetty, but my server-side code uses it, with your proposal, would
it now have to be in (say) src/main/webapp/WEB-INF/lib? Or could it
still come from the Eclipse classpath like it does today?

My two cents is that foo.jar being on the Eclipse classpath is just
fine/what I generally prefer anyway, even if it "works too well", as it
means I can skip the "make a war" or "put all jars into WEB-INF/lib"
steps while just developing.

That said, you are right that it could lead to false positives, but I
would really hope that developers are not using "it works on my machine
in dev mode against my exploded war" as their criteria for "will work in
production". Perhaps that is too optimistic.

- Stephen

[Thomas Broyer]

On Monday, March 17, 2014 4:50:47 PM UTC+1, Stephen Haberman wrote:

Hi Thomas,

> (or rather, it'll work "too well", and won't detect that "something is
> missing in WEB-INF/lib".

I'm attempting to follow along, but classloader semantics are "fun", so
apologies if I'm wrong somewhere...

So, to clarify, if I have a dependency, say foo.jar, that isn't GWT, and
isn't Jetty, but my server-side code uses it, with your proposal, would
it now have to be in (say) src/main/webapp/WEB-INF/lib?

Neither one (depending on your build tool though).

Maven (or Gradle) are built on the same rule that src/ is for sources, and is readonly as far as the build tool is concerned.

Your dependencies coming from Maven will ultimately go to target/${project.build.finalName}/WEB-INF/lib, and this (target/$project.build.finalName} is what you should use as your "-war" directory. See http://web.archive.org/web/20130619170526/https://developers.google.com/eclipse/docs/faq#gwt_with_maven (don't ask me why it has been removed from the GPE FAQ, I have no idea)

http://stackoverflow.com/q/5719118/116472 is an example of what will happen if you don't follow that rule.

So your foo.jar will be in target/${project.build.finalName}/WEb-INF/lib, put there by your build tool.

AFAIK, the gwt-gradle-plugin does the right thing too (but doesn't integrate with the GPE; or rather, the GPE doesn't integrate with anything else than Maven).

That's for "managed dependencies". For "manual dependencies", then yes, I'd argue you should put them in your WEB-INF/lib.

Or could it still come from the Eclipse classpath like it does today?

Jens proposal is to make it fail in this case.

I don't have a strong opinion though; I'd be fine just using the system classloader as the parent of the webapp classloader (the end result would be the same as if you deployed all your code and dependencies to Jetty's lib/ or lib/ext/ or Tomcat's shared/ ); I wonder though what newbies would prefer (or rather, what would be best for them, before they can even forge an opinion, by the principle of least surprise). I wonder whether we should output warnings when loading things from the classpath rather than WEB-INF/lib; knowing how things work and knowing what I do, I'd be fine leaving without warnings, I think newbies would need/prefer warnings.

 

My two cents is that foo.jar being on the Eclipse classpath is just
fine/what I generally prefer anyway, even if it "works too well", as it
means I can skip the "make a war" or "put all jars into WEB-INF/lib"
steps while just developing.

That said, you are right that it could lead to false positives, but I
would really hope that developers are not using "it works on my machine
in dev mode against my exploded war" as their criteria for "will work in
production". Perhaps that is too optimistic.

As I said, we're power users, we're used to "build tools" assembling our apps, how java webapps are organized with their WEB-INF/classese and WEB-INF/lib, etc. We can tolerate one behavior or the other, possibly even having a switch to choose the one we prefer for our environment.

The problem has more to do with newbies, of the kind that download the GPE, create a sample app, copy/paste some code from StackOverflow, downloading the needed dependencies manually, and finally publishing their app to AppEngine; all of that without exiting Eclipse, without any build tool other than Eclipse and the GPE.

Maybe it's a GPE issue after all (I never tried IntelliJ IDEA for webapps; I tend to prefer embedded servers these days, rather than wars deployed into servlet containers).

[Jens]

Maybe it's a GPE issue after all (I never tried IntelliJ IDEA for webapps; I tend to prefer embedded servers these days, rather than wars deployed into servlet containers).

You have the same issue in IntelliJ. IntelliJ can create an example GWT project that will have src + war folder and gwt-user on class path. However inside that war folder there is no WEB-INF/lib folder generated. Because that folder does not exist by default it is maybe even more likely that a newbie directly adds a library to the project/module instead of creating WEB-INF/lib and manually put the lib folder on class path.

However if you want to "export" that project to a war file within IntelliJ you would define a Java web application artifact and while doing so, IntelliJ creates red error messages (along with a single click "fix" button) which say which libraries are required by the project but are missing in the artifact. So you get the sense that your artifact will not work if you ignore and don't fix all these error messages.

If you want to deploy to AppEngine from within IntelliJ you need to add the AppEngine facet and this will again need an artifact.

-- J.

[Stephen Haberman]

> Neither one (depending on your build tool though).

Sure, I get that Maven/Gradle/Ant will go through a copy-to-target step
first, and so not use any src/... artifacts. So, for them, having the
Jetty classloader not use the project classpath makes sense.

But at least we way run DevMode in Eclipse (as vanilla .launch files
with DevMode as the main class), we pass -war src/main/webapp, so we
can run directly against src/ without any copy-to-target/build system
steps at all.

And so then all of our IvyDE-managed jars and project .class files come
from the regular Eclipse/system classpath that "Java Applications" get.

> I don't have a strong opinion though; I'd be fine just using the
> system classloader as the parent of the webapp classloader (the end
> result would be the same as if you deployed all your code and
> dependencies to Jetty's lib/ or lib/ext/ or Tomcat's shared/ );

Which is the behavior today, right? That is what I'd generally prefer,
given it allows the previous "only use Eclipse" sort of setup.

> I think newbies would need/prefer warnings.

I'd be fine with a warning that could be disabled... :-)

- Stephen

[Thomas Broyer]

On Tue, Mar 18, 2014 at 4:10 PM, Stephen Haberman <stephen....@gmail.com> wrote:

> Neither one (depending on your build tool though).

Sure, I get that Maven/Gradle/Ant will go through a copy-to-target step
first, and so not use any src/... artifacts. So, for them, having the
Jetty classloader not use the project classpath makes sense.

But at least we way run DevMode in Eclipse (as vanilla .launch files
with DevMode as the main class), we pass -war src/main/webapp, so we
can run directly against src/ without any copy-to-target/build system
steps at all.

And so then all of our IvyDE-managed jars and project .class files come
from the regular Eclipse/system classpath that "Java Applications" get.

Writing to a source dir still feels hideously bad to me, but that's your project and I'm not going to dictate the way you should do it (and neither should GWT)

> I don't have a strong opinion though; I'd be fine just using the
> system classloader as the parent of the webapp classloader (the end
> result would be the same as if you deployed all your code and
> dependencies to Jetty's lib/ or lib/ext/ or Tomcat's shared/ );

Which is the behavior today, right?

No.

Today, the parent classloader only contains bootstrap classes. But the webapp classloader will automatically add JARs/directories to the webapp classloader if the webapp tries to load a class that cannot be found in either WEB-INF/ or the JVM (and you should have a warning that it couldn't find the class in the webapp but found it in the system classpath and that it adds the given JAR/directory to the webapp for the current session).

What I'm proposing in the end of the memo is to allow this kind of things (with or without the warning) if we cannot compute the GWT SDK install path (i.e. you likely use managed dependencies); but if you use the GWT SDK (and thus probably do not use managed dependencies), then we should probably mandate that you put everything within your WEB-INF (the GPE already automatically configures Eclipse to output classes to WEB-INF/classes).

AFAICT, the GPE doesn't integrate with Ivy, so should I understand that you're not using the GPE? or are you adding the GWT SDK as a dependency to your Eclipse project? or you just configured the GPE to not warn about a "missing SDK"?

 

> I think newbies would need/prefer warnings.

I'd be fine with a warning that could be disabled... :-)

You already have it ;-)

…but with different semantics.

(disable by setting the gwt.nowarn.webapp.classpath system property to any value)

--
Thomas Broyer
/tɔ.ma.bʁwa.je/