Massive increase in compile time with GWT 2.3

[googelybear]

Hi,

I have recently updated from GWT 2.2.0 to 2.3 and noticed a
significant increas in compile time: Compile time with 2.2.0 was
45minutes and now with 2.3 it increased to 1h 12m - that's almost a
40% increase. I didn't change any settings at all.

Are others experiencing this as well?
Do you have any suggestions how to get the compile time down again?

thanks,
Dennis

[George Moschovitis]

45 minutes ?!?! what kind of app is that?

-g.

[googelybear]

technically the project consists of 4 separate apps, so 4 modules that
are compiled individually and it supports 4 locales. Is it unusual to
have such a large compile time?

On May 18, 1:14 pm, George Moschovitis <george.moschovi...@gmail.com>
wrote:

[Eric Andresen]

I've noticed a pretty significant jump as well.  My app used to be around 65 seconds, and it's up to around 135 seconds now, sometimes spiking up to 3-4 minutes.  I had just chalked it up to installing the full WindowBuilder and GAE plugins that I had skipped in the past, but maybe it is the SDK.

One thing I noticed was the upgrade clobbered my "-draftCompile" and "-localWorkers" settings in Eclipse, you might want to check those on your compile settings if you're on a multi-core processor.

[Thomas Broyer]

On Wednesday, May 18, 2011 1:39:00 PM UTC+2, googelybear wrote:

technically the project consists of 4 separate apps, so 4 modules that
are compiled individually and it supports 4 locales. Is it unusual to
have such a large compile time?

We have something like 65000 LOCs (ncloc metric from Sonar) spread across 2000 classes, with many generators involved (UiBinder, RequestFactory, Editor framework), and it compiles in 212,486s (output from the compiler; that's approx. 3½ minutes).

FYI, a compileReport from a month and a half back (before I added the Apache Wave editor to our app, which brings a whole lot of new files) says:

Source files: 3122

Initial Type Oracle Types: 4409

Final Type Oracle Types: 6187

GeneratedTypes: 1778

AST Referenced Types: 6187

Unreferenced Types: 3409

So yes, I'd say 45 minutes is unusual. You might want to tweak the number of workers, or the memory you allocate the java process (I'm using Maven, so it automatically uses as many workers as there are cores/processors on the machine –but it's a VM, so I don't even know how many of them we have/it thinks we have–, and I otherwise simply bumped the Xmx to 512m –yes, only 512m–).

Also, try using the "server JVM" (when using an Oracle/Sun JDK, pass the "-server" argument). I didn't made any measure, but Ray Cromwell found it made a difference (a few years back).

FYI, I'm using a custom GWT build out of trunk @ r9848 (w/ half a dozen patches applied), which is long after 2.2, but before 2.3 (and before the com.google.web.bindery fwiw), so I can't tell if I already made the "jump" in compile time or not (I don't remember noticing any jump in our metrics when I upgraded from our custom-built 2.2)

[Raphael André Bauer]

GWT 2.3 introduces a new permutation for IE9. This can explain your
compile time. If you want to support IE9 "natively" you have to accept
that I guess...

Best,

Raphael

[googelybear]

In the project I'm working on we have about 300k LOC and ~7000
classes. The GWT Compiler is invoked with the following arguments on
the build machine (this machine has 8 cores and 64gb of ram): -
Dgwt.localWorkers=8 -Xmx2048M -Xss1024M and there are 48 permutations
to compute. Do you guys have any idea what else we could tweak here?

On May 18, 4:02 pm, Raphael André Bauer
<raphael.andre.ba...@gmail.com> wrote:

[Thomas Broyer]

On Wednesday, May 18, 2011 5:37:36 PM UTC+2, googelybear wrote:

In the project I'm working on we have about 300k LOC and ~7000
classes. The GWT Compiler is invoked with the following arguments on
the build machine (this machine has 8 cores and 64gb of ram): -
Dgwt.localWorkers=8 -Xmx2048M -Xss1024M and there are 48 permutations
to compute. Do you guys have any idea what else we could tweak here?

Huh, OK, with such metrics, maybe 45 minutes is normal then...

(and I forgot that we actually only compile 2 permutations!)

Maybe try tweaking the -optimize level. I've read when it was being added that "-optimize 9" (the default) is not always worth it, and something like "-optimize 5" could give similar results (output size) for a, obviously, shorter compile time.

[Jeff Larsen]

What is the purpose of the build? 

Is it to deploy the actual code to a production/test server or is it to enable some sort of selenium/webdriver test framework.

If it is the latter, you could add -draftCompile which will not be highly obfuscated code, but it should be a quicker compile, especially with 48 perms. 

You could also link up multiple computers to do distrubted permutation computations. Ray Cromwell had a talk about this at last year's IO. 

Depending on your budget, a SSD drive could potentially help you out a lot here too. 

[Sanjiv Jivan]

Have you tried compilation using SSD? I'm my experience from last year, SSD's were great for reads but terrible for writes and compilation of medium to large projects actually took a fair bit longer on SSD's. 

It's possible the newer SSD's have gotten better but I would recommend doing some more research before making this expensive purchase with expectations of significant compile time speed improvements.

--
You received this message because you are subscribed to the Google Groups "Google Web Toolkit" group.
To post to this group, send email to google-we...@googlegroups.com.
To unsubscribe from this group, send email to google-web-tool...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-web-toolkit?hl=en.

[googelybear]

First of all: Thank you very much for all the feedback!

This build I am trying to optimize is compiled on our build server by
the continuous integration tool (hudson in our case triggered after
every commit). It is mainly used to run unit tests and for general
testing by the developers to get "instant" feedback (well, it used to
do that when we started). It is not a production build. But I don't
like to take too many things out, e.g. take out browsers then you can
no longer test it on different browsers and your feedback cycle - the
time until you notice something doesn't work after you implemented it
- gets longer). For the production build then it is absolutely OK to
take longer.

I tried using the draftCompile switch and it already reduces the build
time to 53minutes - that's a huge 25% decrease compared to the
previous 1h12min.

I will try to play with the optimize level now.

Regarding the SSD: This is not really an option for me as the build is
running on a server used by other projects/users so that would be a
major operation.

Please let me know if you have any other suggestions.

Dennis

[Magno Machado]

You could try the distributed build. Basically, you will have N machines and each will only compile a set of permutations, and after all you link everything together and you have your compiled app ready for use

http://code.google.com/p/google-web-toolkit/wiki/DistributedBuilds

--
Magno Machado Paulo
http://blog.magnomachado.com.br
http://code.google.com/p/emballo/

[Hilco Wijbenga]

On 19 May 2011 00:37, googelybear <googe...@gmail.com> wrote:
> This build I am trying to optimize is compiled on our build server by
> the continuous integration tool (hudson in our case triggered after
> every commit). It is mainly used to run unit tests and for general
> testing by the developers to get "instant" feedback (well, it used to
> do that when we started). It is not a production build. But I don't
> like to take too many things out, e.g. take out browsers then you can
> no longer test it on different browsers and your feedback cycle - the
> time until you notice something doesn't work after you implemented it
> - gets longer). For the production build then it is absolutely OK to
> take longer.

In general, I don't think it is a good idea to have one build for
(many) different purposes.

For unit tests you don't need all browsers so pick one and stick with
it. In fact, for unit tests you don't need any browser. :-) Your unit
test build can and should be very fast. This should be the most
stripped down version you can think of. Mind you, it would be even
better if you broke up your app into separate modules so that all the
unit testing is done in the small, fast module builds.

The second build would be for integration testing. For your automated
integration testing you don't need more than one browser either.
(Unless, of course, you have a very advanced setup testing multiple
browsers.) Run this build once or twice a day at a specific time (say
lunch time and dinner time). (The specific time is so that people know
about it and can try to make sure their change is (or is not)
included.)

If the automated integration test build is successful then kick off
the full build for all browsers. This need only happen once a day or
even once a week. This build is then used for manual testing. It
should be auto deployed to some QA/test environment. Most (test/QA)
people don't like working with a moving target (for obvious reasons),
hence the "build once a week" suggestion. Then, if QA says this build
is good, promote it to production; no need for another build. I.e.
assuming you follow the best practice of not including your
environment configuration in the WAR.

[Twentyseven]

Hello,

In fact I have exactly the same problem than Dennis.
We migrate our application from GWT 2.0 to 2.3 and th ecompilation
time has increased about 40%.
My problem is not how to optimize the compilation time but why this
huge difference between GWT versions.

Thank's

On 19 mai, 18:30, Hilco Wijbenga <hilco.wijbe...@gmail.com> wrote:

> On 19 May 2011 00:37, googelybear <googelyb...@gmail.com> wrote:
>
> > This build I am trying to optimize is compiled on our build server by
> > the continuous integration tool (hudson in our case triggered after
> > every commit). It is mainly used to run unit tests and for general
> > testing by the developers to get "instant" feedback (well, it used to
> > do that when we started). It is not a production build. But I don't
> > like to take too many things out, e.g. take out browsers then you can
> > no longer test it on different browsers and your feedback cycle - the

> >timeuntil you notice something doesn't work after you implemented it

> > - gets longer). For the production build then it is absolutely OK to
> > take longer.
>
> In general, I don't think it is a good idea to have one build for
> (many) different purposes.
>
> For unit tests you don't need all browsers so pick one and stick with
> it. In fact, for unit tests you don't need any browser. :-) Your unit
> test build can and should be very fast. This should be the most
> stripped down version you can think of. Mind you, it would be even
> better if you broke up your app into separate modules so that all the
> unit testing is done in the small, fast module builds.
>
> The second build would be for integration testing. For your automated
> integration testing you don't need more than one browser either.
> (Unless, of course, you have a very advanced setup testing multiple
> browsers.) Run this build once or twice a day at a specifictime(say

> lunchtimeand dinnertime). (The specifictimeis so that people know

[Juan Pablo Gardella]

Because GWT 2.3.0 include support to IE9, so you have another permutation.

Juan

2011/6/2 Twentyseven <ebart...@gmail.com>