SDM "-XcompilePerFile": initial feedback

[Ivan Markov]

Dear GWT team (and John in particular),

Lately I was checking from time to time the progress of the SDM incremental compiler in GWT 2.7.0.

Let me first say - great work so far! 

This morning our project was - for the first time - compilable with the new incremental SDM code-path, without internal compiler errors or visible malfunctioning.

And this with NO changes to the codebase. Also the fact that I don't have to split our codebase into very many small acyclic modules (not that it is not a good practice anyway) is obviously a HUGE PLUS.

What I am observing in terms of recompilation times is encouraging (25 seconds with 2.7 vs. 47 with 2.6.1) but what is even more interesting is that the compilation itself (up until where it says "Compile of permutations succeeded") seems to take only 7 seconds, which is less than 30% of the recompilation time.

In other words, with the new incremental compiler, the biggest slowdown during recompilation seems to be the linker now, right?

If the linking time can go down to, say, a couple of seconds, then SDM is there (IMO).

So out of pure curiosity - are there obvious optimizations in the linker which are still pending?

For example, I noticed that a huge amount of time (> 50% of the whole linking time) is spend in "Saving source with extras", i.e. SourceSaver. Given that these sources are only needed by the browser during a sourcemap-based debugging session, can't they be served on the fly by the code server rather than being pregenerated on disk during the recompilation? There are debugging scenarios (in-IDE debugging with SDBG), where these source files are not even needed and never requested from the codeserver.

Also, is there any way to accelerate the linker which is in the control of the developer?

For example, can I turn off the above source files pregeneration to see how much time is saved during recompilation?

================= 

Finally one side question:

Does the incremental compiler preserve the names of the generated JavaScript types/functions across recompiles? Including the names of the types/functions which ARE recompiled (as long as the programmer did not change the name of the corresponding Java method/class, that is)?

Because if it does, this opens interesting possibilities:

(a) Javascript hot-swapping 

One disadvantage of the current SDM recompilation is that it triggers page reload in the browser.

If the application being debugged is a stateful, single-paged one, this means that even for a very small change, the developer essentially starts-over the debugging, as the application state is lost. 

Now, the debugging protocol of Blink allows the debugger to hot-swap a piece of JavaScript with a newer version, right?

If the incremental compiler preserves the names of the generated Javascript types/functions, then a debugger like SDBG can detect the SDM recompilation and can pro-actively push the newly generated JavaScript into the browser. Imagine how cool is this: you stay on the page, spot a bug (say, in an onclick event handler), fix the bug in the Java code, hit the Recompile button (or Eclipse/SDBG does it for you behind the scenes when you save your Java file!) and there you have it - you can click again and now the correct, recompiled event handler will be triggered. All of this without leaving the application page and starting over.

Of course, for the above to work, additional changes may be necessary. For example, contrary to production, the generated javascript file should probably always be named the same way. Perhaps, it should also not contain one big "eval" statement. Finally, for big changes, the above hot-swapping will likely not work, but the idea is not to make it work in all cases, but for the majority of the small, incremental changes.

(b) On the fly evaluation of Java expressions in the debugger

The challenge here is that the expression being evaluated must be compiled as if it is already part of the GWT application (i.e.  if it refers to application's classes, methods fields etc., it should be with their correct javascript names). 

This sounds like a perfect fit for the incremental compiler - how about a new HTTP service point in the codeserver? Say, something like http://localhost:<port>?eval=<Java-expression-or-if-required-whole-Java-type>, which outputs, as a response, a valid JavaScript block of code which can be evaluated by the debugger? SDBG (or other debugger) will then use the SDM codeserver as a compiler of sorts for the expression entered by the user during debugging.

I'm surely missing something - for example the evaluated expression needs to be capable of accessing the variables on the currently selected stackframe in the debugger, but if Eclipse's Java debugger is capable of doing that without special support in the Java debugging protocol, then it should be possible for our case as well.

What do you think?

[Ray Cromwell]

On Mon, Aug 18, 2014 at 7:14 AM, Ivan Markov <ivan....@gmail.com> wrote:

Dear GWT team (and John in particular),

In other words, with the new incremental compiler, the biggest slowdown during recompilation seems to be the linker now, right?

If the linking time can go down to, say, a couple of seconds, then SDM is there (IMO).

Yes, we are working on speeding up the linkers right now. Most of the linkers just write a file to disk, which should happen in less than a second.

 

For example, I noticed that a huge amount of time (> 50% of the whole linking time) is spend in "Saving source with extras", i.e. SourceSaver. Given that these sources are only needed by the browser during a sourcemap-based debugging session, can't they be served on the fly by the code server rather than being pregenerated on disk during the recompilation? There are debugging scenarios (in-IDE debugging with SDBG), where these source files are not even needed and never requested from the codeserver.

Actually, saving sources isn't strictly needed to make source maps work, especially for non-Googlers, so we can probably default this to false.

Finally one side question:

Does the incremental compiler preserve the names of the generated JavaScript types/functions across recompiles? Including the names of the types/functions which ARE recompiled (as long as the programmer did not change the name of the corresponding Java method/class, that is)?

Yes, within a session. If you restart SDM, then no. Hot-patching and expression evaluation are definitely things we've discussed (I want to build an Apple XCode/Swift Playground-like REPL for fun myself), but first we've got to make the normal SDM experience out of the box great, including IDE integration and super-fast recompile.  Once we get there, hopefully we can do the other things and reach a state that was better than DevMode was.

--
You received this message because you are subscribed to the Google Groups "GWT Contributors" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-web-toolkit-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-web-toolkit-contributors/5a2a4436-5c2d-4e60-9d5d-314db0045b7d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ivan Markov]

 

For example, I noticed that a huge amount of time (> 50% of the whole linking time) is spend in "Saving source with extras", i.e. SourceSaver. Given that these sources are only needed by the browser during a sourcemap-based debugging session, can't they be served on the fly by the code server rather than being pregenerated on disk during the recompilation? There are debugging scenarios (in-IDE debugging with SDBG), where these source files are not even needed and never requested from the codeserver.

Actually, saving sources isn't strictly needed to make source maps work, especially for non-Googlers, so we can probably default this to false.

This is likely to break the ChromeDevTools debugger, as it does need and does request those source files from the codeserver, isn't it?

(SDBG does not need these as it has the "real" source files in the Eclipse IDE.)

Therefore the perfect scenario would be if these files are indeed materialized by the codeserver lazily, on a file-by-file basis, once such a file is requested by the browser.

A compromise would be if there is an option to switch their generation off, but it defaults to ON (generating them).

Finally one side question:

Does the incremental compiler preserve the names of the generated JavaScript types/functions across recompiles? Including the names of the types/functions which ARE recompiled (as long as the programmer did not change the name of the corresponding Java method/class, that is)?

Yes, within a session. If you restart SDM, then no. Hot-patching and expression evaluation are definitely things we've discussed (I want to build an Apple XCode/Swift Playground-like REPL for fun myself), but first we've got to make the normal SDM experience out of the box great, including IDE integration and super-fast recompile.  Once we get there, hopefully we can do the other things and reach a state that was better than DevMode was.

I think within a session is good enough.

And yes, super-fast recompile is the most important thing of all. Everything else can wait.

 

[Roberto Lublinerman]

So out of pure curiosity - are there obvious optimizations in the linker which are still pending?

We have been thinking about strategies to incrementalize the work done by the linkers, one idea is to have a linker that outputs js for different Java classes to different resources and that could be incrementalized.

For example, I noticed that a huge amount of time (> 50% of the whole linking time) is spend in "Saving source with extras", i.e. SourceSaver. Given that these sources are only needed by the browser during a sourcemap-based debugging session, can't they be served on the fly by the code server rather than being pregenerated on disk during the recompilation? There are debugging scenarios (in-IDE debugging with SDBG), where these source files are not even needed and never requested from the codeserver.

I am currently working on this one. There are two options here: 1) serving source code directly without copying and 2) copying only "changed" sources. Option 1) would be the optimal in terms of compile time but it might result of a worse debugging experience (if you modify a source while debugging the source mapping will be out of date).

 

[John Stalcup]

On Monday, August 18, 2014 7:14:38 AM UTC-7, Ivan Markov wrote:

Dear GWT team (and John in particular),

Lately I was checking from time to time the progress of the SDM incremental compiler in GWT 2.7.0.

Let me first say - great work so far! 

This morning our project was - for the first time - compilable with the new incremental SDM code-path, without internal compiler errors or visible malfunctioning.

That's great to hear :). We're pretty happy with the recent progress as well. 

And this with NO changes to the codebase. Also the fact that I don't have to split our codebase into very many small acyclic modules (not that it is not a good practice anyway) is obviously a HUGE PLUS.

What I am observing in terms of recompilation times is encouraging (25 seconds with 2.7 vs. 47 with 2.6.1) but what is even more interesting is that the compilation itself (up until where it says "Compile of permutations succeeded") seems to take only 7 seconds, which is less than 30% of the recompilation time.

In other words, with the new incremental compiler, the biggest slowdown during recompilation seems to be the linker now, right?

Yes, we've come to the same conclusion and Roberto is working on the linker speed right now.

 

If the linking time can go down to, say, a couple of seconds, then SDM is there (IMO).

So out of pure curiosity - are there obvious optimizations in the linker which are still pending?

For example, I noticed that a huge amount of time (> 50% of the whole linking time) is spend in "Saving source with extras", i.e. SourceSaver. Given that these sources are only needed by the browser during a sourcemap-based debugging session, can't they be served on the fly by the code server rather than being pregenerated on disk during the recompilation? There are debugging scenarios (in-IDE debugging with SDBG), where these source files are not even needed and never requested from the codeserver.

Also, is there any way to accelerate the linker which is in the control of the developer?

For example, can I turn off the above source files pregeneration to see how much time is saved during recompilation?

================= 

Finally one side question:

Does the incremental compiler preserve the names of the generated JavaScript types/functions across recompiles? Including the names of the types/functions which ARE recompiled (as long as the programmer did not change the name of the corresponding Java method/class, that is)?

Yes, names in this mode are now consistent for the life of that SuperDevMode process (see https://gwt.googlesource.com/gwt/+/master/dev/core/src/com/google/gwt/dev/js/JsPersistentPrettyNamer.java). 

Because if it does, this opens interesting possibilities:

(a) Javascript hot-swapping 

One disadvantage of the current SDM recompilation is that it triggers page reload in the browser.

If the application being debugged is a stateful, single-paged one, this means that even for a very small change, the developer essentially starts-over the debugging, as the application state is lost. 

Now, the debugging protocol of Blink allows the debugger to hot-swap a piece of JavaScript with a newer version, right?

If the incremental compiler preserves the names of the generated Javascript types/functions, then a debugger like SDBG can detect the SDM recompilation and can pro-actively push the newly generated JavaScript into the browser. Imagine how cool is this: you stay on the page, spot a bug (say, in an onclick event handler), fix the bug in the Java code, hit the Recompile button (or Eclipse/SDBG does it for you behind the scenes when you save your Java file!) and there you have it - you can click again and now the correct, recompiled event handler will be triggered. All of this without leaving the application page and starting over.

Of course, for the above to work, additional changes may be necessary. For example, contrary to production, the generated javascript file should probably always be named the same way. Perhaps, it should also not contain one big "eval" statement. Finally, for big changes, the above hot-swapping will likely not work, but the idea is not to make it work in all cases, but for the majority of the small, incremental changes.

It does sound like an interesting idea. If at some point the browser reload time (as opposed to the compile time) is the longest part then we will start looking at ways to speed that up. One similarish idea we've discussed is to change the linkers so that the output JS exists as separates files. This would allow the browser to hold on to it's precompiled javascript cache for parts of the application that haven't changed (thus reload much faster). But as Ray mentions, we're pretty heads down on an earlier stage of development right now. So I probably won't be looking at something like this any time soon, but patches welcome.

What I am working on is trying to flesh out the "maturity" of this current approach. At the moment we do not notice when you change the content of resource files which were inputs to Generators and so as a result we fail to rerun those Generators when we should. Once this is corrected there will be a while spent on general bug finding.

After that we will probably start looking at remaining performance bottlenecks. Roberto is already looking at Linkers and we think that the next big opportunity after that will be in improving the TypeOracle construction time (see https://gwt.googlesource.com/gwt/+/master/dev/core/src/com/google/gwt/dev/javac/CompilationUnitTypeOracleUpdater.java and related classes). But there will probably be other opportunities as well.

[Ivan Markov]

18 август 2014, понеделник, 21:10:17 UTC+3, Roberto Lublinerman написа:

So out of pure curiosity - are there obvious optimizations in the linker which are still pending?

We have been thinking about strategies to incrementalize the work done by the linkers, one idea is to have a linker that outputs js for different Java classes to different resources and that could be incrementalized.

Sounds interesting! However if you mean separate JavaScript file per Java class, wouldn't that generate too many Javascript files in case I have, say, refactor-renamed a heavily used method? Another approach is maybe each time to output a single JavaScript file, which contains the delta of the changes to the previous JavaScript file (i.e. some new functions and some function redefinitions). Then the browser uses the old JavaScript + the new one... Anyway, just thinking loud, you sure know the topic better than me...

For example, I noticed that a huge amount of time (> 50% of the whole linking time) is spend in "Saving source with extras", i.e. SourceSaver. Given that these sources are only needed by the browser during a sourcemap-based debugging session, can't they be served on the fly by the code server rather than being pregenerated on disk during the recompilation? There are debugging scenarios (in-IDE debugging with SDBG), where these source files are not even needed and never requested from the codeserver.

I am currently working on this one. There are two options here: 1) serving source code directly without copying and 2) copying only "changed" sources. Option 1) would be the optimal in terms of compile time but it might result of a worse debugging experience (if you modify a source while debugging the source mapping will be out of date).

 

I think the out-of-date problem in option (1) does not happen so often in practice. 

Also, even if you do (2), the out-of-date problem is impossible to avoid if you use an in-IDE debugger like SDBG. Reason: SDBG is not really using the source files' copies you generate in the code server, but rather is opening and using directly the source files in the Eclipse project (and that's how it should be or else the integration between editing and debugging won't be so seamless). The DART Editor does it too, and I would assume JIDEA does the same.

So IMO the ideal solution would be if you serve the source files directly, and maybe keep the option to pregenerate the source files on disk if somebody is using an out-of-IDE debugger like ChromeDevTools and wants to avoid the out-of-date problem by all means.

[Ivan Markov]

(a) Javascript hot-swapping 

One disadvantage of the current SDM recompilation is that it triggers page reload in the browser.

If the application being debugged is a stateful, single-paged one, this means that even for a very small change, the developer essentially starts-over the debugging, as the application state is lost. 

Now, the debugging protocol of Blink allows the debugger to hot-swap a piece of JavaScript with a newer version, right?

If the incremental compiler preserves the names of the generated Javascript types/functions, then a debugger like SDBG can detect the SDM recompilation and can pro-actively push the newly generated JavaScript into the browser. Imagine how cool is this: you stay on the page, spot a bug (say, in an onclick event handler), fix the bug in the Java code, hit the Recompile button (or Eclipse/SDBG does it for you behind the scenes when you save your Java file!) and there you have it - you can click again and now the correct, recompiled event handler will be triggered. All of this without leaving the application page and starting over.

Of course, for the above to work, additional changes may be necessary. For example, contrary to production, the generated javascript file should probably always be named the same way. Perhaps, it should also not contain one big "eval" statement. Finally, for big changes, the above hot-swapping will likely not work, but the idea is not to make it work in all cases, but for the majority of the small, incremental changes.

It does sound like an interesting idea. If at some point the browser reload time (as opposed to the compile time) is the longest part then we will start looking at ways to speed that up. One similarish idea we've discussed is to change the linkers so that the output JS exists as separates files. This would allow the browser to hold on to it's precompiled javascript cache for parts of the application that haven't changed (thus reload much faster). But as Ray mentions, we're pretty heads down on an earlier stage of development right now. So I probably won't be looking at something like this any time soon, but patches welcome.

The javascript hot swapping is not (only) addressing the browser page reload time. Rather, it is addressing the frustration of the user that he has lost the state of the app due to the page reload, and now she has to potentially do lots of manual clicks and types *inside* the app until it is brought again to the state where the bug happens. But for sure this is a much lower priority than having a very fast incremental recompilation and relinking. Once I have some time myself, I may start looking into hot swapping in the context of SDBG.

(BTW, If outputting the 30+ MB javascript file does take some time then looking into outputting instead a "delta" javascript file as you hint is probably worth it even if we put aside my dreams for hot-code swapping via the debugger... but then I'm only speculating here...)

What I am working on is trying to flesh out the "maturity" of this current approach. At the moment we do not notice when you change the content of resource files which were inputs to Generators and so as a result we fail to rerun those Generators when we should. Once this is corrected there will be a while spent on general bug finding.

After that we will probably start looking at remaining performance bottlenecks. Roberto is already looking at Linkers and we think that the next big opportunity after that will be in improving the TypeOracle construction time (see https://gwt.googlesource.com/gwt/+/master/dev/core/src/com/google/gwt/dev/javac/CompilationUnitTypeOracleUpdater.java and related classes). But there will probably be other opportunities as well.

 Sounds great. I am already happy with the current 7 seconds recompile (as long as the 18 seconds of relinking go down to 2 or so :-) ), but faster is always better.

[John Stalcup]

Hey Ivan

We just finished fixing a couple of issues in this compile mode (one around JSOs and a couple around rerunning Generators).

Any more feedback you have would be appreciated. If you see any bugs please report them.

Thanks!

John

--
You received this message because you are subscribed to the Google Groups "GWT Contributors" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-web-toolkit-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-web-toolkit-contributors/1132c1d4-0ac7-4d1e-a1ad-7575e6dadab5%40googlegroups.com.

[Jens]

We just finished fixing a couple of issues in this compile mode (one around JSOs and a couple around rerunning Generators).

Any more feedback you have would be appreciated. If you see any bugs please report them.

Just tried it yesterday. Compile time of normal SDM was about 40-50 seconds and with per-file it dropped down to 17-20 seconds when changing a single file. From this 17-20 seconds it was about 5-8 seconds pure compilation and the rest was linking. Since I use an SSD I don't think linking was limited by IO. I tried using -Dgwt.speedtracerlog but it seems like recompilation using per-file does not emit any speedtracer log, only the first precompile does so.

During the initial precompile there was a warning about certain css files loaded via class loader, although I am not sure why these few files caused the warning. These css files were referenced using @Source of CssRessource and had a relative path. Other css files did work fine. Also a warning about source maps was emitted (Unable to find all source code needed by debuggers, X files form source maps weren't found).

So in general it seems to work well however linking slows it down quite a bit.

-- J.

[Jens]

Ok just tried it again today with latest trunk and single file changes caused recompiles of 3-6 seconds depending on the file usage while linking stays relatively constant at 5-6 seconds. So linking seems twice as fast now, probably because less work will be done as a log statement "writing source maps and extras" is now missing.

During precompile SDM logs that it has traversed about 140000 fields + methods and about 13000 types so you get an idea of the app size.

I think I will try using SDM with per-file compilation today to see how it works when I change multiple files etc.

-- J.

[Ivan Markov]

I can confirm similar results to the previous post:

- 3 to 6 seconds for recompiling changes to a single file

- 5 to 6 seconds for linking

- Changes to more types naturally cause the compile time to grow (e.g. 13 seconds for around 10 changed files) however the linking stays at 5 to 6 seconds

The improvement compared to my original post seems to be:

- 50% to 100% for a single file recompile

- 300% to 400% for the linking phase

The machine is an ageing core I7-2630QM

Regarding project size, SDM reports traversal of 76197 fields and methods and 6950 types.

As for bugs, I don't have anything to report, but I was not using 2.7 on a daily basis yet.

[Ivan Markov]

BTW using a more recent laptop seems to make a difference and nearly doubles the performance:

MacBook Pro model 2013 (I7 2.7 GHz):

- 2 to 3 seconds for recompiling changes to a single file

- 2 to 3 seconds for linking

- Changes to more types of course again results in a growth of the recompilation time; this time it is 8 instead of 13 seconds for the same set of around 10 changed files. Linking still 3 seconds.

I think you are getting there!

[Juan Pablo Gardella]

Great news!! Thanks a lot for the effort.

--
You received this message because you are subscribed to the Google Groups "GWT Contributors" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-web-toolkit-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-web-toolkit-contributors/c646cebb-005a-42fd-9f9d-62334e1a3189%40googlegroups.com.

[Arnaud TOURNIER]

Thanks to all of you who participated to this very good things !

I am now able (from the trunk) to have a great experience debugging with Gwt + Eclipse + Sdbg + Chrome.

I cannot be quantitatively very precise but the latest optimization (-XcompilePerFile) seems to give good results. My project on which i test that is not so big so the compilation takes about 10 seconds. Improvements are obvious !

Thanks a lot!

Le vendredi 29 août 2014 18:16:40 UTC+2, juan_pablo_gardella a écrit :

Great news!! Thanks a lot for the effort.

On 29 August 2014 13:02, Ivan Markov <ivan....@gmail.com> wrote:

BTW using a more recent laptop seems to make a difference and nearly doubles the performance:

MacBook Pro model 2013 (I7 2.7 GHz):

- 2 to 3 seconds for recompiling changes to a single file

- 2 to 3 seconds for linking

- Changes to more types of course again results in a growth of the recompilation time; this time it is 8 instead of 13 seconds for the same set of around 10 changed files. Linking still 3 seconds.

I think you are getting there!

--
You received this message because you are subscribed to the Google Groups "GWT Contributors" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-web-toolkit-contributors+unsubscribe@googlegroups.com.

[Daniel Kurka]

Hi Arnaud,

are those 10s after the changes we made or before?

-Daniel

To unsubscribe from this group and stop receiving emails from it, send an email to google-web-toolkit-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-web-toolkit-contributors/2e4c1342-cd60-48c8-b6c0-eb286350b36b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Google Germany GmbH

Dienerstr. 12
80331 München

Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Graham Law, Katherine Stephens

[Arnaud TOURNIER]

Basically the compile took 10s before and 9s now, but this is a very tiny project, so that's why i am saying the improvement is here but not quantitatively measurable on my particular project. I will send better stats when working with bigger projects.

Thanks

Arnuad

--
You received this message because you are subscribed to a topic in the Google Groups "GWT Contributors" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-web-toolkit-contributors/1EYSq2QZ51s/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-web-toolkit-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-web-toolkit-contributors/CALLujipnmokiRVxtpE1g%2BF%2BccZbkt0nZYf3jKsDm3ogvmN52%3DA%40mail.gmail.com.

[Seamus McMorrow]

Hi guys, 

I have just tried out the 2.7-SNAPSHOT. I have a fairly large multi-module maven project. 

For development, I am using STS 3.6, windows7, SSD, i7 3720qm 2.6Ghz, 

Starting SuperDevmode:

Unification traversed 147422 fields and methods and 11825 types. 11773 are considered part of the current module and 11773 had all of their fields and methods traversed.

Compilation succeeded -- 113.359s

Linking succeeded -- 2.768s

120.982s total -- Compile completed

Changing One File in one module project/jar:

Unification traversed 5282 fields and methods and 2440 types. 238 are considered part of the current module and 238 had all of their fields and methods traversed.

 Compilation succeeded -- 9.081s

 Linking succeeded -- 1.566s

 13.436s total -- Compile completed

Some other things to note:

When using GWT 2.6.1, a recompile used to take about 45 secs or so.

Sourcemap debugging in Chrome now works perfectly! I can now debug the javascript via java in chrome and all the line numbers match. (This was not the case with 2.6.1)

On the third compile I tried changing 3 files, however it gave me errors about not finding some class files.

I will try to keep using this, and report any other findings.

Keep up the great work guys, GWT development is going to get so much cooler. :)

Looking forward to the official release 

 

[John Stalcup]

Such a small time makes me think that you have particularly slow linkers. What's the time split between compiling and linking?

To view this discussion on the web visit https://groups.google.com/d/msgid/google-web-toolkit-contributors/CANjaDneM-U8n56Kjr36Ce5D7_HRvaCcCqExFM4QB%2B-TsQdKNTg%40mail.gmail.com.

[John Stalcup]

On Mon, Sep 1, 2014 at 3:55 AM, Seamus McMorrow <seamusm...@gmail.com> wrote:

Hi guys, 

I have just tried out the 2.7-SNAPSHOT. I have a fairly large multi-module maven project. 

For development, I am using STS 3.6, windows7, SSD, i7 3720qm 2.6Ghz, 

Starting SuperDevmode:

Unification traversed 147422 fields and methods and 11825 types. 11773 are considered part of the current module and 11773 had all of their fields and methods traversed.

Compilation succeeded -- 113.359s

Linking succeeded -- 2.768s

120.982s total -- Compile completed

The first compile in -XcompilePerFile is slower than a regular compile since it has to traverse a larger set of types and traverse them more fully (to ensure later cache accesses are correct). I'm guessing that normally you get about an 80s first compile? Eventually we should be able to make even this first compile faster by persisting the MinimalRebuildCache to disk and reusing it between different SDM process launches. Also (as Roberto pointed out recently) if we switche from whole-type linking to per-function linking then we could probably get away with doing the old-style partial-class traversal, which would mean faster first compiles and a little bit smaller output.

Changing One File in one module project/jar:

Unification traversed 5282 fields and methods and 2440 types. 238 are considered part of the current module and 238 had all of their fields and methods traversed.

 Compilation succeeded -- 9.081s

 Linking succeeded -- 1.566s

 13.436s total -- Compile completed

Some other things to note:

When using GWT 2.6.1, a recompile used to take about 45 secs or so.

Sourcemap debugging in Chrome now works perfectly! I can now debug the javascript via java in chrome and all the line numbers match. (This was not the case with 2.6.1)

On the third compile I tried changing 3 files, however it gave me errors about not finding some class files.

Can you post these error messages and whether they are related to a Generator? Are you weren't seeing the new warning about resources being loaded via ClassLoader instead of being registered in the .gwt.xml file and loaded via ResourceOracle?

 

I will try to keep using this, and report any other findings.

Keep up the great work guys, GWT development is going to get so much cooler. :)

Looking forward to the official release 

Great! Any bugs/repro cases you can report will help us get this to release-worthy, sooner. 

 

--
You received this message because you are subscribed to the Google Groups "GWT Contributors" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-web-toolkit-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-web-toolkit-contributors/f300a560-19e8-44bd-ba39-b2e48abb8522%40googlegroups.com.

[Arnaud TOURNIER]

Hi Daniel,

I made some more serious tests on a production project with which i used to develop using the classic DevMode.

Those are the numbers that come from the CodeServer with or without the -XcompilePerFile flag, and they are quite impressive :

Standard Compile per file Reduction

1st compile 101s 41s 59%

Change 1 string and recompile 16s 5,8s 64%

Change 1 string and recompile 18s 3,6s 80%

First compile Unification traversed 60072 fields and methods and 6179 types. 6139 are considered part of the current module and 6139 had all of their fields and methods traversed.

Second compile Unification traversed 2528 fields and methods and 2650 types. 124 are considered part of the current module and 124 had all of their fields and methods traversed.

I did not use the Chrome dev tools to debug the application but the SDBG plugin for Eclipse.

The experience is almost incredible, perfectly usable and much faster than the classic DevMode. The only bad thing is not having the Java fields inspected correctly (although one can guess the javascript equivalents most of the time) and little things that can be covered with time... Everything needs a compromise and having that debugging experience brought back to Eclipse is really what i was fearing to lose with the SuperDevMode. So i will adopt the SuperDevMode for a while !

Ray, Brian, John and Robert, Ivan, James and you all made a really impressive work ! The future of GWT is assured ! Now i can't wait to have the Java 8 features to be tried ;)

Now i have a question about the message "Unification traversed..." : first what means the message exactly (i need to see the code) and second why some many fields and methods are traversed if only change one file ?

Next i had ideas but no time to implement them, maybe i dump them here with the hope they can be useful ;)

- Recompile on file change. The time the user switches to the browser, the compile is almost finished.

- Tool to benchmark.

- Recompile on F5, standard SuperDevMode linker

- GWT plugin for eclipse : GWT tools + SDBG

- Sample including the experience

Thanks a lot again for all the work and the good things coming out !

Arnaud

To view this discussion on the web visit https://groups.google.com/d/msgid/google-web-toolkit-contributors/2e4c1342-cd60-48c8-b6c0-eb286350b36b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[John Stalcup]

Any new bugs to report?

To unsubscribe from this group and stop receiving emails from it, send an email to google-web-toolkit-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-web-toolkit-contributors/ecad7963-656b-48f2-b85e-ce56060359c0%40googlegroups.com.

[Jens]

Any new bugs to report?

Some things we have observed at work while using latest trunk SDM (compile per file):

1.) If we don't use -noprecompile then SDM does not start because it says it has found more than one permutation. If we use -noprecompile then SDM starts and the first compile through the bookmarklet works as well. Our trunk build from 1st Sep still worked in both situations. With current trunk the precompile says that it can not set the locale property to "en" because only "de" was allowed, then tries a fallback value and finally the precompile fails.

2.) From time to time it happens that a recompile through the bookmarklet breaks the app because some JS isn't emitted correctly. For example we have a factory that creates a specific Ginjector and that factory has rebind rules that are triggered by a form factor property. Just today a colleague has recompiled the app and got "DesktopGinjectorProvider is undefined". Another example is that a custom composite stopped working with error "undefined is not a function". Taking a look at the JS code is seems like the problem happened in the constructor function that wanted to call the clinit function for that class. The machine that had the issue also had Chrome beta so I wasn't able to debug the pure JS because of 4.)

3.) On one machine the first compilation after starting SDM with -noprecompile never finishes. The CodeServer java process does some heavy work for the first minute or so and then goes to sleep and thats it. I call it "SDM progress dots of death". The CodeServer arguments are exactly the same as on other machines which work without issue. I have no idea why this is happening. Also this only happens with latest trunk, a build from 1st Sep works.

4.) Not directly GWT related but you might want to know it: The new Chrome beta contains the change that Brian asked for which marks lines without source maps in gray so you know that you can not put a breakpoint on these lines. However for some reasons Chrome now also forbids setting breakpoints in the pure JS file (e.g. app-0.js) if no source map entry was found for that JS line. That can be quite annoying. For example I couldn't set a breakpoint inside a class constructor function because the whole JS function wasn't source mapped.

-- J.

[John Stalcup]

Thanks for the feedback Jens.

On Mon, Sep 8, 2014 at 1:31 PM, Jens <jens.ne...@gmail.com> wrote:

Any new bugs to report?

Some things we have observed at work while using latest trunk SDM (compile per file):

1.) If we don't use -noprecompile then SDM does not start because it says it has found more than one permutation. If we use -noprecompile then SDM starts and the first compile through the bookmarklet works as well. Our trunk build from 1st Sep still worked in both situations. With current trunk the precompile says that it can not set the locale property to "en" because only "de" was allowed, then tries a fallback value and finally the precompile fails.

I'm working on a CL right now to opt -XcompilePerFile into the -noprecompile behavior for just the reason. Should be in today.

2.) From time to time it happens that a recompile through the bookmarklet breaks the app because some JS isn't emitted correctly. For example we have a factory that creates a specific Ginjector and that factory has rebind rules that are triggered by a form factor property. Just today a colleague has recompiled the app and got "DesktopGinjectorProvider is undefined". Another example is that a custom composite stopped working with error "undefined is not a function". Taking a look at the JS code is seems like the problem happened in the constructor function that wanted to call the clinit function for that class. The machine that had the issue also had Chrome beta so I wasn't able to debug the pure JS because of 4.)

I've fixed two Gin generator related breakages like this so far. Are you sure this is occurring with HEAD? If so then there's definitely a bug in the "stale types marking" code that needs finding. If you can make a repro case I'll figure it out.

3.) On one machine the first compilation after starting SDM with -noprecompile never finishes. The CodeServer java process does some heavy work for the first minute or so and then goes to sleep and thats it. I call it "SDM progress dots of death". The CodeServer arguments are exactly the same as on other machines which work without issue. I have no idea why this is happening. Also this only happens with latest trunk, a build from 1st Sep works.

Hmmm, that's really strange. It doesn't sound -XcompilePerFIle related so I probably won't take a look at it for now. Maybe Brian will want to take a look.

4.) Not directly GWT related but you might want to know it: The new Chrome beta contains the change that Brian asked for which marks lines without source maps in gray so you know that you can not put a breakpoint on these lines. However for some reasons Chrome now also forbids setting breakpoints in the pure JS file (e.g. app-0.js) if no source map entry was found for that JS line. That can be quite annoying. For example I couldn't set a breakpoint inside a class constructor function because the whole JS function wasn't source mapped.

+skybrian 

-- J.

--
You received this message because you are subscribed to the Google Groups "GWT Contributors" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-web-toolkit-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-web-toolkit-contributors/5b8b733b-ef6c-4aaa-a071-d0bf4b027ca6%40googlegroups.com.

[John Stalcup]

On Mon, Sep 8, 2014 at 1:48 PM, John Stalcup <sta...@google.com> wrote:

Thanks for the feedback Jens.

On Mon, Sep 8, 2014 at 1:31 PM, Jens <jens.ne...@gmail.com> wrote:

Any new bugs to report?

Some things we have observed at work while using latest trunk SDM (compile per file):

1.) If we don't use -noprecompile then SDM does not start because it says it has found more than one permutation. If we use -noprecompile then SDM starts and the first compile through the bookmarklet works as well. Our trunk build from 1st Sep still worked in both situations. With current trunk the precompile says that it can not set the locale property to "en" because only "de" was allowed, then tries a fallback value and finally the precompile fails.

I'm working on a CL right now to opt -XcompilePerFile into the -noprecompile behavior for just the reason. Should be in today.

this is in now. 

[Jens]

I'm working on a CL right now to opt -XcompilePerFile into the -noprecompile behavior for just the reason. Should be in today.

Cool will try it tomorrow.

I've fixed two Gin generator related breakages like this so far. Are you sure this is occurring with HEAD? If so then there's definitely a bug in the "stale types marking" code that needs finding. If you can make a repro case I'll figure it out.

Currently I am not sure, need to ask colleague tomorrow. Can you tell me the commit hashes of your two fixes so I can see which of our builds contain them?

And a last question just popped up again: Is there any way to force embedded Jetty of CodeServer to not print out request logs at DEBUG level? Kind of annoying when all these progress requests show up between GWT logs.

-- J.

[John Stalcup]

On Mon, Sep 8, 2014 at 3:00 PM, Jens <jens.ne...@gmail.com> wrote:

I'm working on a CL right now to opt -XcompilePerFile into the -noprecompile behavior for just the reason. Should be in today.

Cool will try it tomorrow.

I've fixed two Gin generator related breakages like this so far. Are you sure this is occurring with HEAD? If so then there's definitely a bug in the "stale types marking" code that needs finding. If you can make a repro case I'll figure it out.

Currently I am not sure, need to ask colleague tomorrow. Can you tell me the commit hashes of your two fixes so I can see which of our builds contain them?

recentish ones are:

267ad5efd00aae9b0f69eca793891e9fdad28e45  "Opts compilePerFile into noPrecompile to avoid >1 permutation."

a43aa788cc2b78904c1bf6f0de9b8a1ebe78a6d2 "Fixes cascaded generator invalidation for inner classes."

9da4d05b62d7d3cb48ccd03de7b4ad676e2318bd "Makes SDM+perFileCompile correctly handle switching browsers."

7365570209426063bbc7fa4745e0b422e4fb4850 "Marks types generated by invalidated generators as "modified"."

And a last question just popped up again: Is there any way to force embedded Jetty of CodeServer to not print out request logs at DEBUG level? Kind of annoying when all these progress requests show up between GWT logs.

i don't know, +skybrian 

-- J.

--
You received this message because you are subscribed to the Google Groups "GWT Contributors" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-web-toolkit-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-web-toolkit-contributors/b841ccc9-4bea-4e73-9333-c9861308b33d%40googlegroups.com.

[Brian Slesinsky]

On Mon, Sep 8, 2014 at 3:00 PM, Jens <jens.ne...@gmail.com> wrote:

And a last question just popped up again: Is there any way to force embedded Jetty of CodeServer to not print out request logs at DEBUG level? Kind of annoying when all these progress requests show up between GWT logs.

i don't know, +skybrian 

Maybe by configuring java.util.Logger? What does the message look like? (I assume you're passing -logLevel DEBUG to CodeServer? If so I haven't used DEBUG level.)

- Brian

[Jens]

Maybe by configuring java.util.Logger? What does the message look like? (I assume you're passing -logLevel DEBUG to CodeServer? If so I haven't used DEBUG level.)

Never mind. Since we have a project with combined client/server code all server libs are on class path of SDM including SLF4J as we start SDM through IDE. So Jetty configures itself to use SLF4J and without a SLF4J config file it defaults to console logging with level DEBUG. Didn't recognized that SLF4J has been used.

-- J.

[Jens]

recentish ones are:

267ad5efd00aae9b0f69eca793891e9fdad28e45  "Opts compilePerFile into noPrecompile to avoid >1 permutation."

a43aa788cc2b78904c1bf6f0de9b8a1ebe78a6d2 "Fixes cascaded generator invalidation for inner classes."

9da4d05b62d7d3cb48ccd03de7b4ad676e2318bd "Makes SDM+perFileCompile correctly handle switching browsers."

7365570209426063bbc7fa4745e0b422e4fb4850 "Marks types generated by invalidated generators as "modified"."

My colleague used a build that only contained "Marks types generated by invalidated generators as "modified"." . We keep an eye on it if we still get random GIN / UiBinder issues with HEAD.

A bit more information about the issue that SDM compilation does not complete for one of my colleagues:

- Issue happens with HEAD as well as build from 1st Sep.

- Issue only happens if -noprecompile is used (which is now standard with HEAD but wasn't with the build of 1st Sep.)

- On my machine I get a warning about certain css files being loaded from class path. This happens between the log output "Compiling module ..." and "Compiling 1 permutation". My colleague does not see these warnings and he only sees the output "Compiling module ..." which takes forever. So either the InlineClientBundleGenerator never started or takes forever. Otherwise he should have seen the same warnings.

- It is not an endless loop because CPU usage drops down to 0% after a minute or so. 

- A small example app on my colleagues machine works without issues. So SDM is not totally broken on his machine. Only our main app does not work for him.

- For any reason it is reproducible on his machine, but so far it is impossible to reproduce on mine.

Maybe we find some time tomorrow for some real debugging of CodeServer.

-- J.

[Brian Slesinsky]

This sounds like it might be a deadlock. Could you jstack to get a thread dump of the running process? It should tell us where it's stuck.

[Jens]

This sounds like it might be a deadlock. Could you jstack to get a thread dump of the running process? It should tell us where it's stuck.

Yeah it is a deadlock: http://pastebin.com/WCVZc21E

-- J.

[Jens]

Yeah it is a deadlock: http://pastebin.com/WCVZc21E

Hmm although it is not a java level dead lock. Seems like GWT can not close a jar file? Strange.

-- J.

[Jens]

Yeah it is a deadlock: http://pastebin.com/WCVZc21E

Hmm although it is not a java level dead lock. Seems like GWT can not close a jar file? Strange.

Ok we just solved it. The Java run configuration had a VM parameter "-XstartOnFirstThread" and when removing it the CodeServer works as expected. The colleague copied that parameter from his classic DevMode "Web Application" run configuration.

It seems like that Eclipse adds this parameter automatically on Mac OS when creating a new "Web Application" run configuration (provided by GPE). That means most DevMode run configurations have that parameter set. This might affect the new -superDevMode feature of GWT 2.7 as well.

Maybe you can try to reproduce the issue by using Mac OS and adding the VM parameter "-XstartOnFirstThread" to "CodeServer" and "DevMode -superDevMode". We attached a debugger to CodeServer and the thread that hangs wants to close an input stream of the jar url connection gwt-user.jar!/.../disclosurePanelClosed.png. So to reproduce it your demo project should probably use a DisclosurePanel. If you can reproduce it, it might be a good idea to mention that behavior for Mac OS users in the 2.7 release notes.

-- J.

[Jens]

I've fixed two Gin generator related breakages like this so far. Are you sure this is occurring with HEAD? If so then there's definitely a bug in the "stale types marking" code that needs finding. If you can make a repro case I'll figure it out.

Hey John, I have to say that we still see issues with UiBinder + compilePerFile in GWT trunk from 09.09.2014. We have a UiBinder Composite, lets call it "MyComposite", that has 

<ui:style type="com.example.MyComposite.Style" field="style"> ... some css ... </ui:style> 

in its XML file and

interface Style extends CssResource {

  ....

}

@UiField

Style style;

in the corresponding Java class. This MyComposite is used in a different Composite (GenericCustomComposite) that has a sub class (SpecificCustomComposite).

If we add a new accessor method to the Style interface of MyComposite then compilation still works but the app fails at runtime with:

com.google.gwt.core.client.JavaScriptException: (TypeError) : undefined is not a function

at Unknown.UIObject_1_g$(app-0.js@10:19605)

at Unknown.Widget_1_g$(app-0.js@19:20155)

at Unknown.Composite_1_g$(app-0.js@17:109460)

at Unknown.GenericCustomComposite_1_g$(app-0.js@20:479016)

at Unknown.SpecificCustomComposite_1_g$(app-0.js@23:477194)

at Unknown.generateUiEntries_0_g$(app-0.js@29:221296)

at Unknown.setData_0_g$(app-0.js@14:221535)

at Unknown.success_60_g$(app-0.js@25:221911)

at Unknown.success_61_g$(app-0.js@10:221921)

at Unknown.onSuccess_7_g$(app-0.js@10:213650)

So instantiating of SpecificCustomComposite (whose super class uses the modified MyComposite) fails. I debugged it a bit with the following information (some stuff renamed):

function SpecificCustomComposite_1_g$(param1_0_g$, param2_0_g$, param3_0_g$){

  try {

    $clinit_SpecificCustomComposite_0_g$();

    // Here "this" points to __proto__: F_0_g$ which contains all the methods 

    // of SpecificCustomComposite_1_g as well as $init_13782_g$() called below.

    GenericCustomComposite_1_g$.call(this, param1_0_g$, param2_0_g$);

    this.$init_13782_g$();

    this. param3_1_g$ = param3_0_g$;

  }

   catch (e_0_g$) {

    throw e_0_g$;

  }

}

function GenericCustomComposite_1_g$(param1_0_g$, param2_0_g$){

  try {

    $clinit_GenericCustomComposite_0_g$();

    var someVar_0_g$;

    // Here "this" still points to __proto__: F_0_g$ which contains all the methods

    // of SpecificCustomComposite_1_g as well as $init_13782_g$().

    Composite_1_g$.call(this);

    this.$init_13781_g$(); // technically the correct init method but "this" does not have it.

    .... 

    this.initWidget_3_g$(this.container_66_g$);

  }

   catch (e_0_g$) {

    throw e_0_g$;

  }

}

function Composite_1_g$(){

  try {

    $clinit_Composite_0_g$();

    // "this" is still the same F_0_g$

    Widget_1_g$.call(this);

    this.$init_10766_g$(); // technically the correct init method but "this" does not have it.

  }

   catch (e_0_g$) {

    throw e_0_g$;

  }

}

function Widget_1_g$(){

  try {

    $clinit_Widget_0_g$();

    // "this" is still the same F_0_g$

    UIObject_1_g$.call(this);

    this.$init_10009_g$(); // technically the correct init method but "this" does not have it.

  }

   catch (e_0_g$) {

    throw e_0_g$;

  }

}

function UIObject_1_g$(){

  try {

    $clinit_UIObject_0_g$();

    // "this" is still the same F_0_g$

    Object_1_g$.call(this);

    this.$init_10008_g$(); // causes undefined error because "this" does not contain the init method of UIObject

  }

   catch (e_0_g$) {

    throw e_0_g$;

  }

}

So it seems like the "this" pointer somehow gets messed up after recompile.

The try/catch block in all these code blocks above are from our custom GWT build as we need to disable JIT on safari permutation (iOS 6.x JIT bugs) and wrapping everything with try/catch was the safest solution so far (even though it is not nice). To do so we have a custom Visitor in JavaToJavaScriptCompiler.PermutationCompiler.compilePermutation() that executes between calls to optimizeJs() and JsStackEmulator.exec(). 

Maybe this info helps you to spot the issue.

-- J.