PROPOSAL: a devserver to replace devmode

[Thomas Broyer]

Hi all,

I've talked several times about using a Webpack-like devserver instead of DevMode, and with a better experience (in most cases) than CodeServer. I had described it in more details in https://github.com/gwtproject/gwt/issues/9437#issuecomment-250926589

I spent some time this week-end to build a prototype: https://github.com/tbroyer/gwt-devserver

There are a few caveats with the current implementation:

• maybe intercepts a bit too much, so most GWT-RPC calls won't work (they require the gwt.codeserver.port system property on the server side anyway to load the serialization policies from the CodeServer). It could intercept only *.nocache.js requests but that would break some apps that reference resources from the GWT modules' public path (such as themes) from non-GWT resources; or it could only intercept requests for those resources, but that requires more work.
• loads modules only to get their rename-to; the CodeServer already does that, but doesn't expose the information through public APIs.

If there's consensus, such a tool could be integrated into GWT proper and directly use CodeServer's API to share information; and completely replace DevMode.

One big advantages (IMO) over DevMode, or CodeServer with -launcherDir, is that it doesn't overwrite the *.nocache.js with the compile-on-load stub.

One big drawback is GWT-RPC requires the gwt.codeserver.port system property (which requires running the server on the same machine) or using a custom serialization policy loader. We could add a -launcherDir where we'd write the serialization policies (and only those, possibly also the public resources, but not the *.nocache.js stub!)

Feedback welcome!

[Brandon Donnelson]

I like the sounds on this, although my initial thoughts feel like it's another layer. But I also know typically it takes me a bit longer to warm up to new approaches. I have trying to think how I can cut the code server out of the loop. 

So I've been wondering if I could start the gwt super dev mode compile, provide the user agent in the args, so the code server isn't needed. Then compile the initial compile into the web app directory. Skip the working directory, skip a temp directory, do it into the web app. There would be no need for injection, b/c it'd be acting like the real web app, minus the optimizations and multi permutation builds. So in that same process, listen for changes to the app, and then recompile after the change, or based on a trigger, like a change in file.

Here's how Eclipse web tools platform could take advantage of that. Compile into a web app directory, when ever contents are added or changed in that directory it refreshes the server and if you refresh the web page, the contents get loaded. 

Do you think we could get rid of the any additional web server (code server) and push the bits into the web app directory, or some directory we could push to the web app server? The arguments needed for the compiler could be provided in the program args, and changes could trigger the recompile. 

[Thomas Broyer]

On Monday, October 24, 2016 at 3:55:47 AM UTC+2, Brandon Donnelson wrote:

I like the sounds on this, although my initial thoughts feel like it's another layer. But I also know typically it takes me a bit longer to warm up to new approaches. I have trying to think how I can cut the code server out of the loop. 

So I've been wondering if I could start the gwt super dev mode compile, provide the user agent in the args, so the code server isn't needed. Then compile the initial compile into the web app directory. Skip the working directory, skip a temp directory, do it into the web app. There would be no need for injection, b/c it'd be acting like the real web app, minus the optimizations and multi permutation builds. So in that same process, listen for changes to the app, and then recompile after the change, or based on a trigger, like a change in file.

That'd be entirely possible yes (might require duplicating large portions of CodeServer if you want to do it "externally", but could be baked into GWT proper yes).

Webpack actually comes with both a "watch mode" and a devserver (which, in the case of Webpack, actually wraps the "watch mode", it does not compile on-demand like GWT's SDM); each has its use IMO, they're not mutually exclusive.

The devserver has the advantage that you can proxy to a live server on another machine without even touching that machine, or you can serve static files from a local directory without writing to that directory and messing with either an existing production compile, your SCM, or your build (e.g. a *.nocache.js from src/main/webapp overwriting the production compilation in target/, or your build skipping GWT compilation because it thinks it's up-to-date and you end up packaging and deploying a stub nocache.js). Webpack's devserver can also inject a "live reload" script (which the GWT devserver could possibly do too, if it also watched the input files) so the webpage is automatically refreshed whenever a change is detected.

The "watch mode" is great if you're OK with writing directly to the webapp, or you can configure the webapp to load the GWT module from another directory (this is what I do currently in my gwt-maven-archetypes).

With the devserver, you can actually debug with a production server and quickly test changes to the code; like we used to be able to with the legacy DevMode in -noserver mode (even though we were deploying some code that allowed the "devmode" to be triggered, which is why you had to whitelist your server in the browser plugin settings when doing so), and we can do with the SDM bookmarklets when you're able to use them (i.e. you only have permutations around the user.agent and locale properties).

BTW, the devserver approach also works around the HTTPS limitations of SDM, as it proxies everything through HTTP (and on localhost so you can safely use APIs that require a "secure context"). That said, HTTPS could be added to CodeServer (and the devserver, for use by other devices/VMs not on localhost) relatively easily.

Actually, GWT already watches files to update its internal PathPrefixSet and avoid a full re-scan of the classpath. I wonder what the cost would be to do the same thing in another class (hopefully the JVM or the OS would share resources when watching the same folder or file twice), or if it could be possible to reuse the ResourceAccumulatorManager (but there's currently no callback to trigger anything on change).

The thing is, a GWT compilation still is rather costly even with all optimisations disabled, so I'm not sure triggering a compile on file change is a good thing.

 

Here's how Eclipse web tools platform could take advantage of that. Compile into a web app directory, when ever contents are added or changed in that directory it refreshes the server and if you refresh the web page, the contents get loaded. 

My understanding is that this is how GWT 3.0 with J2Cl would work, because J2Cl is "only" a transpiler that can work at the file level, without knowledge of the whole classpath (I can be wrong on that last bit though).

 

Do you think we could get rid of the any additional web server (code server) and push the bits into the web app directory, or some directory we could push to the web app server? The arguments needed for the compiler could be provided in the program args, and changes could trigger the recompile.

As said above, technically, yes, we could. This is not mutually exclusive with a devserver though, and/but might prove too costly at runtime compared to the current "compile on-demand" approach.

[Jens]

Hmm generally I like having build-in proxy with GWT, as I usually use a proxy anyways at work just to mimic the production behavior (load balancers serving static GWT app files and proxy certain requests to servlet container). However I think there should be more options to be more configurable. 

Looking at our setup I would instantly ask for cookie rewriting (multiple ProxyPassReverseCookieDomain, ProxyPassReverseCookiePath), proxy rewrites per URL (we have multiple *.war deployed thus different contextPaths but all are accessed from the GWT app) and finally the ability to add custom headers (mostly caching, but x-forwarded-for / x-real-ip should also be there). Well and as configure options add up, a way to switch between different configurations might be handy, e.g. -configFile parameter.

Also GWT-RPC and GWT public resources should work as lot of people depend on it. I actually did a contribution to provide gwt.codeserver.host but it was abandoned in favor of writing GWT-RPC policy files to the -launcherDir on each compile. That way if -launcherDir points to an exploded war a local Jetty will just have them or you need to build the war and redeploy to a remote server (or maybe use some network share). See https://gwt-review.googlesource.com/#/c/9504/ for some discussion.

Next there are people embedding the module.nocache.js file directly into their host page (like me ;-) basically our host page is a servlet) and that should work also during development.

-- J.

[Thomas Broyer]

On Monday, October 24, 2016 at 11:23:07 AM UTC+2, Jens wrote:

Hmm generally I like having build-in proxy with GWT, as I usually use a proxy anyways at work just to mimic the production behavior (load balancers serving static GWT app files and proxy certain requests to servlet container). However I think there should be more options to be more configurable.

I'd tend to disagree.

 

Looking at our setup I would instantly ask for cookie rewriting (multiple ProxyPassReverseCookieDomain, ProxyPassReverseCookiePath),

Those cookies likely wouldn't "work" anyway as they wouldn't be shared with the other "subdomains" (unless your own machine shares the same parent-domain as your server, in which case you don't need to rewrite the cookies anyway). I believe that, similarly to those redirects where you need to generate a full URL (generally for SSO), your server should adapt its behavior depending on the Host or X-Forwarded-Host; either that or use another proxy that does the rewriting for you (most "dev proxies" don't: Webpack –unless you use your own middleware or onProxyRes event handler–, devd, mitmproxy, etc.)

ProxyPathReverseCookiePath is useless: the reverse proxy doesn't rewrite paths (on purpose! the goal is only to be able to intercept some requests to route them to the CodeServer, or serve the stub *.nocache.js)

 

proxy rewrites per URL (we have multiple *.war deployed thus different contextPaths but all are accessed from the GWT app)

Again, the reverse proxy doesn't rewrite paths, so everything should Just Work™.

What would "not work" would be running several modules each corresponding to a different "context", as -modulePathPrefix can only match one of them; you'd have to run 2 devservers, one for each module, with the appropriate -modulePathPrefix, and one proxying to the other which would proxy to the "real" server.

E.g. if you have a GWT app at http://example.com/context1/app1/app1.nocache.js and another at http://example.com/context2/app2/app2.nocache.js, you cannot use a single devserver for both modules; you'd have to use 2 devservers:

devserver -proxyTo http://example.com -port 8889 -codeServerPort 9877 -modulePathPrefix context1 com.example.app1.App1

devserver -proxyTo http://localhost:8889 -modulePathPrefix context2 com.example.app2.App2

This is a bit heavyweight, but only needed when you want to work on *both* modules at the same time (BTW, the same is true with "CodeServer with -launcherDir", as you'd need 2 different -launcherDir values, so 2 CodeServer instances)

 

and finally the ability to add custom headers (mostly caching, but x-forwarded-for / x-real-ip should also be there).

The Jetty (Async)ProxyServlet already adds X-Forwarded-* headers:

http://archive.eclipse.org/jetty/9.2.14.v20151106/xref/org/eclipse/jetty/proxy/ProxyServlet.html#79

http://archive.eclipse.org/jetty/9.2.14.v20151106/xref/org/eclipse/jetty/proxy/AbstractProxyServlet.html#495

What kind of caching-related headers would you want? and why?

For anything else, providing hooks that you can override in a custom subclass would likely be the way I'd do it (note that it's currently only a single <500 lines file, so forking is also an option; it obviously wouldn't be the case if the devserver were built into GWT proper)

 

Also GWT-RPC and GWT public resources should work as lot of people depend on it. I actually did a contribution to provide gwt.codeserver.host but it was abandoned in favor of writing GWT-RPC policy files to the -launcherDir on each compile. That way if -launcherDir points to an exploded war a local Jetty will just have them or you need to build the war and redeploy to a remote server (or maybe use some network share). See https://gwt-review.googlesource.com/#/c/9504/ for some discussion.

Hey, this is a currently just a prototype coded over the week-end ;-)

I'd personally prefer not writing anything to the webapp, but as acknowledged in the OP we could have a -launcherDir option for serialization policies and possibly public resources (but then why not just use CodeServer with -launcherDir directly? one difference could be that devserver would never overwrite a file for example…)

 

Next there are people embedding the module.nocache.js file directly into their host page (like me ;-) basically our host page is a servlet) and that should work also during development.

I can't think of any other way to make it work (I did that too in one app) than having a way to switch from inlined *.nocache.js to external *.nocache.js on your server. There's absolutely no risk shipping that in production (it's only a network optimisation to avoid one HTTP request) so it looks like an acceptable compromise to me.

…or you could just continue using CodeServer with -launcherDir.

[Jens]

Well it is just what I would ask for in order to replace local apache / nginx on developer machines with GWT devserver. It's not a problem to simply continue using a dedicated proxy either with CodeServer or a "no-op" devserver. 

Basically our setup is more a less the result of two decisions: Not serving static content from servlet containers (instead it's severed by load balancer/proxy) and using GWT-RPC. We need to be in control of which app version a customer gets served, even before logging in, and that backend requests will be redirected to a matching server app version (because of GWT-RPC policy files and rolling updates). Thats why our production mapping is https://app.example.com/<customer>/ => <private-backend>/<app>_<build>/ for backend requests and the load balancer does the mapping using URL rewriting.

During development we just have a single app so the rewriting is a lot simpler but it still exists. Given that our server code sets cookies for the production domain only (I guess some paranoid security decision), the cookie domain needs to be rewritten during development to a local ip / localhost.

Given that static content is served by the load balancer / proxy itself, all caching headers (*.nocache.*, *.cache.*) are configured outside the servlet container. For consistency cache headers for dynamic host page are also configured outside the servlet container. So we don't have any ServletFilter doing it.

Hey, this is a currently just a prototype coded over the week-end ;-)

I'd personally prefer not writing anything to the webapp, but as acknowledged in the OP we could have a -launcherDir option for serialization policies and possibly public resources

(but then why not just use CodeServer with -launcherDir directly? one difference could be that devserver would never overwrite a file for example…)

Well I could also ask: Why not just having a single solution that simply works? Personally I don't like having different "entry points" for development. IMHO its more straight forward (especially for beginners) to just have CodeServer or just have devserver, but not both (even with different behavior for stub.nocache.js !). 

Adding gwt.codeserver.host and renaming -launcherDir to -publicResourcesDir would be fine I guess.

 

Next there are people embedding the module.nocache.js file directly into their host page (like me ;-) basically our host page is a servlet) and that should work also during development.

I can't think of any other way to make it work (I did that too in one app) than having a way to switch from inlined *.nocache.js to external *.nocache.js on your server. There's absolutely no risk shipping that in production (it's only a network optimisation to avoid one HTTP request) so it looks like an acceptable compromise to me.

Well devserver could special case host page loading ("/" or a configurable host page url) and inspect the result of the backend server to decide wether or not some inline *.nocache.js file needs to be replaced. To keep it simply I would require special enclosing start / end comments in the dynamic host page to make it compatible with devserver. Then the content in between these comments will be replaced with the stub.nocache.js.

 

…or you could just continue using CodeServer with -launcherDir.

Single solution preferred ;-)

[Daniel Kurka]

I am seeing a lot of arguments pop up about GWT RPC, but I think it should not be considered for this discussion at all. In my mind GWT 2.8 will be the last release that has GWT RPC and people should start migrating. I think its perfectly fine do design a replacement to devmode without GWT RPC support.

--
You received this message because you are subscribed to the Google Groups "GWT Contributors" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-web-toolkit-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-web-toolkit-contributors/361aafb5-d40d-433c-a6ba-6209069e5550%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brandon Donnelson]

I find most of our customer base uses GWT RPC. The story on using this may make using gwt feel complicated. 

If we can just compile into the web app directory for the entire process, then there would be no need for a proxy. From what I could tell it would be 3 args, -war, -browser (or maybe some user agent) and module. I'll see if I can prototype the service this week and provide it for chewing on. Do you think this is even possible? 

-Brandon

To unsubscribe from this group and stop receiving emails from it, send an email to google-web-toolkit-contributors+unsubscribe@googlegroups.com.

[Brandon Donnelson]

Nevermind you answered my question above. 

[Brandon Donnelson]

I've discovered another nifty method of compiling for debugging. That's using the Compiler to skip the CodeServer process and compiling directly into the exploded war directory. The issue with this is there aren't any source maps, but there are some that could care less and want to go as quickly as possible. This means a compiler launcher could be fired up and compile the project incrementally into the the war directory. 

Example 

Compiler -incremental  -setProperty user.agent=safari -war /Users/branflake2267/Downloads/tmp

CodeServer is Hacky

By they way, the CodeServer is a bit of a hack where the right tooling could provide all the pieces. Most the problem is that the server is caching the resources so updates won't get reflected after a incremental compile. But tooling can solve this. The next issue is source maps provision, this could get loaded into the web resources. Then there wouldn't be a need for the code server, just tooling that will compile into the war directory and refresh the web server after compile. 

I've added a GWT Compile launcher to the GWT Eclipse Plugin. This should make the compiler path easier to work with. It improves classpath configuration for the compiler.