Move Super Dev Mode to the open source repository. (issue1727804)

[skyb...@google.com]

Reviewers: cromwellian,

Description:
Move Super Dev Mode to the open source repository.
Includes a simple "demo" target that works with the "Hello" sample app.
I've also made some linker options configurable, as necessary to make
this work.
Review by: cromw...@google.com

Please review this at http://gwt-code-reviews.appspot.com/1727804/

Affected files:
A dev/codeserver/BUILD
A dev/codeserver/build.xml
A dev/codeserver/java/com/google/gwt/dev/codeserver/AppSpace.java
A dev/codeserver/java/com/google/gwt/dev/codeserver/CodeServer.java
A dev/codeserver/java/com/google/gwt/dev/codeserver/CompileDir.java
A
dev/codeserver/java/com/google/gwt/dev/codeserver/CompilerOptionsImpl.java
A dev/codeserver/java/com/google/gwt/dev/codeserver/ModuleState.java
A dev/codeserver/java/com/google/gwt/dev/codeserver/Modules.java
A dev/codeserver/java/com/google/gwt/dev/codeserver/Options.java
A dev/codeserver/java/com/google/gwt/dev/codeserver/Recompiler.java
A dev/codeserver/java/com/google/gwt/dev/codeserver/SourceHandler.java
A
dev/codeserver/java/com/google/gwt/dev/codeserver/UnmodifiableCompilerOptions.java
A dev/codeserver/java/com/google/gwt/dev/codeserver/WebServer.java
A dev/codeserver/java/com/google/gwt/dev/codeserver/computeScriptBase.js
A dev/codeserver/java/com/google/gwt/dev/codeserver/dev_mode_on.js
A dev/codeserver/java/com/google/gwt/dev/codeserver/frontpage.html
A dev/codeserver/java/com/google/gwt/dev/codeserver/modulepage.html
M dev/core/src/com/google/gwt/core/linker/CrossSiteIframeLinker.java
M samples/hello/src/com/google/gwt/sample/hello/Hello.gwt.xml
M user/src/com/google/gwt/core/CrossSiteIframeLinker.gwt.xml

[cromw...@google.com]

LGTM, I didn't review the Java files because I assume they are mostly
unchanged since the last time.



http://gwt-code-reviews.appspot.com/1727804/diff/1/dev/codeserver/BUILD
File dev/codeserver/BUILD (right):

http://gwt-code-reviews.appspot.com/1727804/diff/1/dev/codeserver/BUILD#newcode1
dev/codeserver/BUILD:1: # Copyright 2012 Google Inc. All rights
reserved.
Do we really want the BUILD file in the public repo?

http://gwt-code-reviews.appspot.com/1727804/

[t.br...@gmail.com]

Most of my comments should be seen as "TODOs", as I really don't want to
delay landing: this is highly experimental anyway, and we know it'll
need some more work.


http://gwt-code-reviews.appspot.com/1727804/diff/1/dev/codeserver/java/com/google/gwt/dev/codeserver/AppSpace.java
File dev/codeserver/java/com/google/gwt/dev/codeserver/AppSpace.java
(right):

http://gwt-code-reviews.appspot.com/1727804/diff/1/dev/codeserver/java/com/google/gwt/dev/codeserver/AppSpace.java#newcode2
dev/codeserver/java/com/google/gwt/dev/codeserver/AppSpace.java:2: *
Copyright 2011 Google Inc.
2012 ?

http://gwt-code-reviews.appspot.com/1727804/diff/1/dev/codeserver/java/com/google/gwt/dev/codeserver/ModuleState.java
File dev/codeserver/java/com/google/gwt/dev/codeserver/ModuleState.java
(right):

http://gwt-code-reviews.appspot.com/1727804/diff/1/dev/codeserver/java/com/google/gwt/dev/codeserver/ModuleState.java#newcode49
dev/codeserver/java/com/google/gwt/dev/codeserver/ModuleState.java:49:
defaultProps.put("user.agent", "safari");
This is just a "default" for the very first compilation, right?
(but hard-coding things that way will prevent detecting the "right"
permutation later on, no?)

http://gwt-code-reviews.appspot.com/1727804/diff/1/dev/codeserver/java/com/google/gwt/dev/codeserver/Options.java
File dev/codeserver/java/com/google/gwt/dev/codeserver/Options.java
(right):

http://gwt-code-reviews.appspot.com/1727804/diff/1/dev/codeserver/java/com/google/gwt/dev/codeserver/Options.java#newcode134
dev/codeserver/java/com/google/gwt/dev/codeserver/Options.java:134:
private class SourceFlag extends ArgHandler {
Why not use ArgHandlerDir?

http://gwt-code-reviews.appspot.com/1727804/diff/1/dev/codeserver/java/com/google/gwt/dev/codeserver/Options.java#newcode148
dev/codeserver/java/com/google/gwt/dev/codeserver/Options.java:148:
return "The root of a directory tree containing GWT source code to
compile.";
Does this mean SuperDevMode no longer loads sources from the classpath?

http://gwt-code-reviews.appspot.com/1727804/diff/1/dev/codeserver/java/com/google/gwt/dev/codeserver/WebServer.java
File dev/codeserver/java/com/google/gwt/dev/codeserver/WebServer.java
(right):

http://gwt-code-reviews.appspot.com/1727804/diff/1/dev/codeserver/java/com/google/gwt/dev/codeserver/WebServer.java#newcode164
dev/codeserver/java/com/google/gwt/dev/codeserver/WebServer.java:164: //
This is a GET because a bookmarklet can call it from a different origin
(JSONP).
We should probably use CORS and XMLHttpRequest with a POST request,
instead of JSON-P. The bookmarklet could handle XDomainRequest for IE8/9
and even possibly create an HTML form if we want to support IE6/7.

White-listing origins from command-line arguments would also add some
security.

http://gwt-code-reviews.appspot.com/1727804/diff/1/dev/codeserver/java/com/google/gwt/dev/codeserver/WebServer.java#newcode456
dev/codeserver/java/com/google/gwt/dev/codeserver/WebServer.java:456:
private static String escapeHtmlCharacters(String line) {
Couldn't we use SafeHtml.fromString() here?

http://gwt-code-reviews.appspot.com/1727804/

[cromw...@google.com]

http://gwt-code-reviews.appspot.com/1727804/diff/1/dev/codeserver/java/com/google/gwt/dev/codeserver/Options.java
File dev/codeserver/java/com/google/gwt/dev/codeserver/Options.java
(right):

http://gwt-code-reviews.appspot.com/1727804/diff/1/dev/codeserver/java/com/google/gwt/dev/codeserver/Options.java#newcode148
dev/codeserver/java/com/google/gwt/dev/codeserver/Options.java:148:
return "The root of a directory tree containing GWT source code to
compile.";

On 2012/06/05 10:52:58, tbroyer wrote:
> Does this mean SuperDevMode no longer loads sources from the
classpath?

It picks up from the classpath, but this directory shadows the
classpath. The reason is, at least for Google, we always produce
source-jars in our internal build system, and you can't 'edit' them.
That is, when we launch superdevmode, it has the source, but it's been
compiled and put into jars on the CP. This directory allows us to
specify another location to look at, in addition to the classpath (which
is the fallback)

For external users, it may not be needed, although I could see if
superdevmode is launched from a Maven build you might run into the same
issue.

http://gwt-code-reviews.appspot.com/1727804/

[skyb...@google.com]

http://gwt-code-reviews.appspot.com/1727804/

[skyb...@google.com]

Ray, you can see the diffs by looking at internal code review. I updated
a bunch of javadoc, tweaked Recompiler to deal with running with a
different linker, and changed the regexps in WebServer to be slightly
tighter and a bit more readable. But sure, basically the same.

http://gwt-code-reviews.appspot.com/1727804/diff/1/dev/codeserver/BUILD
File dev/codeserver/BUILD (right):

http://gwt-code-reviews.appspot.com/1727804/diff/1/dev/codeserver/BUILD#newcode1
dev/codeserver/BUILD:1: # Copyright 2012 Google Inc. All rights
reserved.

On 2012/06/05 01:05:21, cromwellian wrote:
> Do we really want the BUILD file in the public repo?

I was using an out-of-date version of rietveld that didn't ignore these
files. Fixed.

http://gwt-code-reviews.appspot.com/1727804/diff/1/dev/codeserver/java/com/google/gwt/dev/codeserver/AppSpace.java
File dev/codeserver/java/com/google/gwt/dev/codeserver/AppSpace.java
(right):

http://gwt-code-reviews.appspot.com/1727804/diff/1/dev/codeserver/java/com/google/gwt/dev/codeserver/AppSpace.java#newcode2
dev/codeserver/java/com/google/gwt/dev/codeserver/AppSpace.java:2: *
Copyright 2011 Google Inc.

On 2012/06/05 10:52:58, tbroyer wrote:

> 2012 ?

Well, I started on some of these files last year, so I'm going to leave
it.

http://gwt-code-reviews.appspot.com/1727804/diff/1/dev/codeserver/java/com/google/gwt/dev/codeserver/ModuleState.java
File dev/codeserver/java/com/google/gwt/dev/codeserver/ModuleState.java
(right):

http://gwt-code-reviews.appspot.com/1727804/diff/1/dev/codeserver/java/com/google/gwt/dev/codeserver/ModuleState.java#newcode49
dev/codeserver/java/com/google/gwt/dev/codeserver/ModuleState.java:49:
defaultProps.put("user.agent", "safari");

On 2012/06/05 10:52:58, tbroyer wrote:

> This is just a "default" for the very first compilation, right?
> (but hard-coding things that way will prevent detecting the "right"
permutation
> later on, no?)

Yes, this is just for the first compile and the output isn't intended to
be run. Normally when you are running a separate server, the binding
properties will be computed by the nocache.js script of the
non-superdevmode version of the GWT app, and then they're saved in
session storage, so we always compile the same permutation.

However, I recently added support for running html files directly out of
the Super Dev Mode code server (for an all-static GWT app where a
separate server isn't needed). In that case, the initial compile *is*
used, which is the wrong permutation if you're not using Safari or
Chrome. And on the second compile, it will build all permutations, which
is another bug.

I'll be fixing this later. I'd like to see if there's a way to terminate
the initial compile early and not output a permutation. It's a useful
sanity check but shouldn't take too long.

http://gwt-code-reviews.appspot.com/1727804/diff/1/dev/codeserver/java/com/google/gwt/dev/codeserver/Options.java
File dev/codeserver/java/com/google/gwt/dev/codeserver/Options.java
(right):

http://gwt-code-reviews.appspot.com/1727804/diff/1/dev/codeserver/java/com/google/gwt/dev/codeserver/Options.java#newcode134
dev/codeserver/java/com/google/gwt/dev/codeserver/Options.java:134:
private class SourceFlag extends ArgHandler {

On 2012/06/05 10:52:58, tbroyer wrote:

> Why not use ArgHandlerDir?

This variant actually checks that the directory exists. (Probably
ArgHandlerDir should be fixed, but I didn't want to get into that.)

http://gwt-code-reviews.appspot.com/1727804/diff/1/dev/codeserver/java/com/google/gwt/dev/codeserver/Options.java#newcode148
dev/codeserver/java/com/google/gwt/dev/codeserver/Options.java:148:
return "The root of a directory tree containing GWT source code to
compile.";

On 2012/06/05 16:15:35, cromwellian wrote:
> On 2012/06/05 10:52:58, tbroyer wrote:
> > Does this mean SuperDevMode no longer loads sources from the
classpath?

> It picks up from the classpath, but this directory shadows the
classpath. The
> reason is, at least for Google, we always produce source-jars in our
internal
> build system, and you can't 'edit' them. That is, when we launch
superdevmode,
> it has the source, but it's been compiled and put into jars on the CP.
This
> directory allows us to specify another location to look at, in
addition to the
> classpath (which is the fallback)

> For external users, it may not be needed, although I could see if
superdevmode
> is launched from a Maven build you might run into the same issue.

Tweaked the help text a bit.

http://gwt-code-reviews.appspot.com/1727804/diff/1/dev/codeserver/java/com/google/gwt/dev/codeserver/WebServer.java
File dev/codeserver/java/com/google/gwt/dev/codeserver/WebServer.java
(right):

http://gwt-code-reviews.appspot.com/1727804/diff/1/dev/codeserver/java/com/google/gwt/dev/codeserver/WebServer.java#newcode164
dev/codeserver/java/com/google/gwt/dev/codeserver/WebServer.java:164: //
This is a GET because a bookmarklet can call it from a different origin
(JSONP).

On 2012/06/05 10:52:58, tbroyer wrote:

> We should probably use CORS and XMLHttpRequest with a POST request,
instead of
> JSON-P. The bookmarklet could handle XDomainRequest for IE8/9 and even
possibly
> create an HTML form if we want to support IE6/7.

> White-listing origins from command-line arguments would also add some
security.

Yeah, CORS is better and I'm not too concerned about old browsers. Users
would have to maintain the whitelist, though. If we want to wildcard
hostnames (e.g. everything under corp.google.com), I think that requires
a preflight request?

http://gwt-code-reviews.appspot.com/1727804/diff/1/dev/codeserver/java/com/google/gwt/dev/codeserver/WebServer.java#newcode456
dev/codeserver/java/com/google/gwt/dev/codeserver/WebServer.java:456:
private static String escapeHtmlCharacters(String line) {

On 2012/06/05 10:52:58, tbroyer wrote:

> Couldn't we use SafeHtml.fromString() here?

Currently, gwt-codeserver.jar only depends on gwt-dev.jar (and its
dependencies) and I'd rather not add the dependency until it enables a
more important feature.

http://gwt-code-reviews.appspot.com/1727804/

[Chris Price]

I've just tried to take the demo for a spin but I had a few problems
trying to convince the dev_mode_on code that super dev mode was
enabled. Hopefully I'm missing something but from a quick look over
the code I think something might be unavailable in the public version.

dev_mode_on contains the following conditional when showing/hiding the
compile bookmarklet -

var mod = modules_on_page[module_name];

var dev_mode_key = '__gwtDevModeHook:' + module_name;
var dev_mode_on = mod['superdevmode'] ||
window.sessionStorage[dev_mode_key];

if (!dev_mode_on && !mod.canRedirect) {
return 'This module doesn\'t have Super Dev Mode enabled';
}

In my case it keeps falling into the "This module doesn't have SDM
enabled" block and no SDM for me! However, if I force one of the
conditions to skip the block everything works nicely for that compile
and subsequent compiles. I'm assuming that the sessionStorage is
responsible for latching the behaviour, so I looked into what sets
window.__gwt_activeModules[module_name]['superdevmode'] and
window.__gwt_activeModules[module_name].canRedirect.

The only place I can find that sets
window.__gwt_activeModules[module_name]['superdevmode'] is a custom
version (although the original doesn't seem to be in the source tree)
of computeScriptBase.js which isn't used by the standard
CrossSiteIframeLinker. A comment in Recompiler suggests that it is
used by Google's server-side linker.

window.__gwt_activeModules[module_name].canRedirect seems only to be
set by the DevModeRedirectHook.js which is included by
CrossSiteIframeLinker when devModeRedirectEnabled is true. However,
Recompiler checks for the existence of this property (throwing if it
doesn't exist) before forcing it to false.

My current workaround is to prepend the following onto my dev mode on
bookmarklet -

window.__gwt_activeModules.hello.superdevmode = true;

Is this a genuine problem or am I going about this all wrong?

Chris

> --
> http://groups.google.com/group/Google-Web-Toolkit-Contributors

[Chris Price]

Just to finish this off in case anyone else has the same problem -
this was fixed in r11031.

https://code.google.com/p/google-web-toolkit/source/detail?r=11031

[Jens]

Just to finish this off in case anyone else has the same problem -
this was fixed in r11031.

https://code.google.com/p/google-web-toolkit/source/detail?r=11031

Thanks for the info. I had the same problem but now it works. Now lets hope the compilation process will become a lot faster in the future. A single permutation of our app compiles on the code server in ~45 seconds and thats not really acceptable during development compared to the "classic" DevMode.

-- J.

[Ray Cromwell]

How big is your app? (KLOC) Internally, we'd had success with <10s
refresh on some fairly largish apps. I'm wondering if something else
isn't going on. Note that the first compile usually takes long, but
subsequent ones are faster. Are you using UiBinder? One issue could be
generators using too much time.

> --
> http://groups.google.com/group/Google-Web-Toolkit-Contributors

[Ray Cromwell]

I just checked again, one of the internal apps we are trying
SuperDevMode against has about 400,000 lines of code and combines in
about 7-8 seconds. So maybe your app has 6x as much code? If not, it
gives us a clue that perhaps there's some other edge condition causing
the compile to be slow.

-Ray

[Paul Robinson]

I've just tried SuperDevMode for the first time. I have around 250KLOC, of which around 200KLOC are translated to javascript.

The code server reported 3 minutes for the initial compile (with 2 permutations), then 37 seconds for the first compile from the browser, and then consistently 18-19 seconds thereafter.

Paul

[Ray Cromwell]

Hmm, that's still not where we'd like to be. Our design goal is < 10s.
Are you using UiBinder, Gin, ImageResource, I18N, GWT RPC? I'm
wondering if one of the generators may or may not have an issue with
incremental compilation.

This may not do anything, but try it anyway, add these JVM flags
-Dgwt.persistentunitcache
-Dgwt.persistentunitcachedir=/tmp
-XX:MaxPermSize=512M
-Xmx2048m

Also add -Dgwt.speedtracerlog=/tmp/superdevmode.json

And send me the resulting file after a few recompile runs. This may
give me a hint as to which parts of the compiler are taking too long.

-Ray

> --
> http://groups.google.com/group/Google-Web-Toolkit-Contributors

[Paul Robinson]

I'm not using UiBinder or Gin.

I am using GWT RPC, I18N and ImageResource.

I have approximately:
- 100 methods across several ClientBundles returning an ImageResource.
- 700 I18N methods spread over about 30 interfaces (my initial 2 permutations were for the default locale)
- for GWT RPC, I have about 40 RPC methods. One of those is used for the command pattern and has a hierarchy of about 70 request classes that can be passed to it.

My initial VM args were:
-Xmx1024M -Xss8M

This is running on linux 64 bit, with enough spare RAM that it never used any swap space.

After changing to your suggested VM args, the initial compile was 62s (instead of 3 minutes), then compiling from the browser took 18.4s, 16.8s, 17.0s, 17.4s, 15.4s, 16.8s, 16.0s

I did add a "-Dgwt.speedtracerlog=/tmp/superdevmode.json" argument, but no file was produced.

HTH
Paul

[Ray Cromwell]

Opps, sorry, the speedtracer log is always on anyway. Look in the
super-dev-mode work dir, e.g.
/tmp/gwt-codeserver-*/modulename/speedtracer.html

I think the issue might be that I18N generators don't implement
IncrementalGenerator, and so they run every compile, even if nothing
in your interfaces or property files changed. We are talking about
having perhaps two buttons in the UI, "Compile" and "Run Generators
and Compile", or maybe checkbox. Then, we'd run the generators once on
the first compile, and subsequent recompiles would not re-run them.
Except when people are editing css, images, or i18n interfaces, they
can then do a full 'regenerate and recompile'. Later when all of the
standard GWT generators are taught to behave better it would be needed
less and less.

> --
> http://groups.google.com/group/Google-Web-Toolkit-Contributors

[Paul Robinson]

I've sent the speed tracer file directly to you.

Also, I have some http GET requests that the code server cannot find (quite reasonably). It would be nice if it could delegate anything it can't find to the external tomcat/jetty.

It's also objecting to some of the characters in the URL of some of my http GET requests. Is it possible c.g.g.dev.codeserver.WebServer's safe filename pattern could be an optional argument?

Thanks,
Paul

[Stefano Ciccarelli]

My app is 600000 lines of code and compiles in 120 seconds: a lot of uibinder, a lot of classes in rpc, gin… so a lots of generators. 

--
Sent with Sparrow

--

http://groups.google.com/group/Google-Web-Toolkit-Contributors

[Ray Cromwell]

How long does a regular compile take? We've identified an issue with
the RPC and ClientBundle generators running every time. We may have to
add an option to "Recompile without re-running generators" option
until we fix the caching.

-Ray

> --
> http://groups.google.com/group/Google-Web-Toolkit-Contributors

[Ray Cromwell]

BTW, can you send me the speedtracer.html file in your
/tmp/*codeserver/* directory?

On Fri, Jun 8, 2012 at 1:41 PM, Stefano Ciccarelli
<scicc...@gmail.com> wrote:

> --
> http://groups.google.com/group/Google-Web-Toolkit-Contributors

[Stefano Ciccarelli]

The full compile takes ~16 minutes for 8 permutations.

About the SuperDevMode the first compile takes ~260 seconds for 2 permutations, the subsequent compiles takes from 120-160 seconds each (1 permutation).

The speedtracer.html file is attached.

I have 4gb of ram and I'm using -Xmx1024m.

I'm sure the performance will improve, the real problem now is that it is a pain to use with gwt-rpc because I have to deal with generated *.gwt.rpc files.

-- 
Inviato con Sparrow

--

http://groups.google.com/group/Google-Web-Toolkit-Contributors

[Ray Cromwell]

looks like you have compiler.stackMode = emulated. It won't solve your
problem, but it should shave off some time to set it to 'native'. One
thing puzzles me is even the linker stage is taking 15 seconds for
you. That stage doesn't really do much but write files and create the
nocache.js script. There might be some degenerate issue there. Seems
like a significant amount of time is spend running RPC generators and
ClientBundle, even though it is not supposed to.

-Ray


On Sat, Jun 9, 2012 at 11:54 PM, Stefano Ciccarelli

[Stefano Ciccarelli]

I've tried setting compiler.stackMode=native and I gained 30 seconds for a single permutation, I don't need it during development so I now a compiles takes around 80/90 seconds.

Now I'm trying to implement the IncrementalGenerator interface on my generators but GeneratorContext.isGeneratorResultCachingEnabled() seems to be always false.

2012/6/10 Ray Cromwell <cromw...@google.com>

--
Nel mondo esistono 10 categorie di persone, quelle che capiscono il binario e quelle che non lo capiscono.

[Brian Slesinsky]

Not yet. SuperDevMode is still considered "experimental" for 2.5. But
we'll be working on it soon.

On Mon, Oct 29, 2012 at 12:07 PM, NK <nagink...@gmail.com> wrote:
> With release of GWT 2.5GA has this issue been resolved? I tried to use
> SuperDev mode ig 2.5 GA, , it takes to recompile my app around 40to 60 sec.