[groovy-user] Groovyc OutOfMemoryError

[Clay McCoy]

We have a large Java project (1 million LOC, some individual classes with 20k loc), and we use Groovy mainly for testing. The use of Groovy is not a large percentage of the loc. Recently we have been having to increase memory substantially just to compile. It now to the point that we are scared to even edit a groovy file because we will likely have to increase memory. At this point we are up to 1344m. We compile using IntelliJ locally, and Ant on the CI server. We've had to increase memory in both places, but the Ant build seems to be more brittle.

We are using Groovy 1.6.1.

This is the Ant groovyc target. We started forking it because of the memory issues. The nested javac seems to be the preffered way rather than jointCompilationOptions.

<target name="compile">
<mkdir dir="${dir.classes}"/>
<echoproperties destfile="echoedCompile.properties"/>

<taskdef name="groovyc" classname="org.codehaus.groovy.ant.Groovyc" classpath="${classpath}"/>
<groovyc srcdir="${dir.src.java}" destdir="${dir.classes}" fork="yes" memoryMaximumSize="1344m" memoryInitialSize="1344m">
<classpath>
<pathelement path="${classpath}"/>
<pathelement path="${classpath.other.jars.to.include}"/>
</classpath>
<javac source="1.5" target="1.5" debug="on" debuglevel="lines,vars,source"/>
</groovyc>
</target>

This is the error that we get:

compile:
19-Jul-2009 17:00:19 [groovyc] Compiling 6224 source files to E:\bamboo-agent-home\xml-data\build-dir\DEV\Build\EJB\target\classes
19-Jul-2009 17:03:18 [groovyc] >>> a serious error occurred: Java heap space
19-Jul-2009 17:03:18 [groovyc] >>> stacktrace:
19-Jul-2009 17:03:18 [groovyc] java.lang.OutOfMemoryError: Java heap space
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.ast.ClassNode.<init>(ClassNode.java:315)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.ast.ClassNode.<init>(ClassNode.java:238)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.ast.ClassHelper.makeWithoutCaching(ClassHelper.java:156)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.ast.ClassHelper.make(ClassHelper.java:148)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.ast.ClassHelper.make(ClassHelper.java:137)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.vmplugin.v5.Java5.makeClassNode(Java5.java:316)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.vmplugin.v5.Java5.makeParameter(Java5.java:337)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.vmplugin.v5.Java5.makeParameters(Java5.java:330)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.vmplugin.v5.Java5.configureClassNode(Java5.java:279)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.ast.ClassNode.lazyClassInit(ClassNode.java:257)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.ast.ClassNode.getInterfaces(ClassNode.java:341)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.ast.ClassNode.declaresInterface(ClassNode.java:842)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.ast.ClassNode.implementsInterface(ClassNode.java:827)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.ast.ClassNode.isDerivedFromGroovyObject(ClassNode.java:817)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.classgen.AsmClassGenerator.isGroovyObject(AsmClassGenerator.java:2639)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.classgen.AsmClassGenerator.visitPropertyExpression(AsmClassGenerator.java:2617)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.ast.expr.PropertyExpression.visit(PropertyExpression.java:55)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.classgen.AsmClassGenerator.visitAndAutoboxBoolean(AsmClassGenerator.java:4029)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.classgen.AsmClassGenerator.makeCallSite(AsmClassGenerator.java:1955)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.classgen.AsmClassGenerator.makeCall(AsmClassGenerator.java:1789)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.classgen.AsmClassGenerator.makeCall(AsmClassGenerator.java:1775)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.classgen.AsmClassGenerator.makeInvokeMethodCall(AsmClassGenerator.java:1758)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.classgen.AsmClassGenerator.visitMethodCallExpression(AsmClassGenerator.java:2267)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.ast.expr.MethodCallExpression.visit(MethodCallExpression.java:63)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.classgen.AsmClassGenerator.visitAndAutoboxBoolean(AsmClassGenerator.java:4029)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.classgen.AsmClassGenerator.visitCastExpression(AsmClassGenerator.java:1701)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.classgen.AsmClassGenerator.assignmentCastAndVisit(AsmClassGenerator.java:3968)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.classgen.AsmClassGenerator.evaluateEqual(AsmClassGenerator.java:3920)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.classgen.AsmClassGenerator.visitBinaryExpression(AsmClassGenerator.java:1324)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.ast.expr.BinaryExpression.visit(BinaryExpression.java:49)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.classgen.AsmClassGenerator.visitAndAutoboxBoolean(AsmClassGenerator.java:4029)
19-Jul-2009 17:03:18 [groovyc] at org.codehaus.groovy.classgen.AsmClassGenerator.visitExpressionStatement(AsmClassGenerator.java:1305)
19-Jul-2009 17:03:20
19-Jul-2009 17:03:20 BUILD FAILED
19-Jul-2009 17:03:20 E:\bamboo-agent-home\xml-data\build-dir\DEV\Tools\AntScripts\Build\build.xml:402: Forked groovyc returned error code: 1
19-Jul-2009 17:03:20

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

[Guillaume Laforge]

Not answering your question directly so far, but I'm curious why
you're using 1.6.1 vs 1.6.3?

--
Guillaume Laforge
Groovy Project Manager
Head of Groovy Development at SpringSource
http://www.springsource.com/g2one

[Clay McCoy]

I looked at the release notes for 1.6.2 and 1.6.3 and didn't see anything that would blatantly address this. I'll give it a try.

Currently Groovy is viewed with some skepticism here because of this issue, and some previous inconsistencies with IntelliJ and Ant builds. We started having these problems when we upgraded to 1.6.0. That is the point where we had to fork, and the memory requirements have increased rapidly since then. We have moved the version forward and back a few times, usually in a panic.

We get a similar error in IntelliJ and keep having to increase the groovy.compiler.Xmx. It's currently at 1344m, which just seems wrong. We could use 640m in May with a very similar codebase.

I found the original issue for forking groovyc. The comments lack confidence in this functionality.
http://jira.codehaus.org/browse/GROOVY-2066

[Jochen Theodorou]

Clay McCoy schrieb:
[...]

> We get a similar error in IntelliJ and keep having to increase the
> groovy.compiler.Xmx. It's currently at 1344m, which just seems wrong.
> We could use 640m in May with a very similar codebase.

hmm... you could help us here a little in telling how memory is used.
What is it that takes so much memory in your case?

> I found the original issue for forking groovyc. The comments lack
> confidence in this functionality.
> http://jira.codehaus.org/browse/GROOVY-2066

why do they lack confidence?

bye blackdrag

--
Jochen "blackdrag" Theodorou
The Groovy Project Tech Lead (http://groovy.codehaus.org)
http://blackdragsview.blogspot.com/

[Mike Dillon]

Have you analyzed the heap at all to see what's in it when it fills up?
These options for Sun's VM are useful in this situation:

-XX:+HeapDumpOnOutOfMemoryError
-XX:+HeapDumpPath=/path/to/dump/directory

http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/clopts.html

We actually always run with this setting on, just in case. It comes in
very handy. If you don't have experience analyzing heap dumps, the
VisualVM tool that comes with Sun's VM is pretty handy.

-md

begin Clay McCoy quotation:

[Clay McCoy]

These were odd responses, you just turned my questions around on me.

>hmm... you could help us here a little in telling how memory is used.
>What is it that takes so much memory in your case?

We are just trying to compile the code. If I knew what the compiler was doing that takes so much memory I would be fixing it.

>why do they lack confidence?

Again, you tell me. These are the words from the issue:
"this is not really tested so it may have problems."
"For errors in the task (and I suspect there are some), new issues should be raised."

[Paul King]

I know as a general principal that for the runtime, 1.6 sacrifices
some startup time and memory usage for increased speed. I am not
sure whether compiler memory usage has changed that much although
again we did some things to speed it up which might have involved
keeping more info in memory, not sure. If Jochen can't remember
I can try to track it down from svm history. I certainly didn't
spot too much change between 1.6.1 and 1.6.3 memory wise for the
compiler when perusing history just now.

I'd like to help you try to find the culprit for your memory
issues but not sure the best place to start without access to
the code or some more info. If you had to take a guess, is there
one thing your code is doing that is a little unique, e.g. large
amount of code? accessing lots of other classes? something else?

Cheers, Paul.

[Russel Winder]

Clay,

On Wed, 2009-07-22 at 22:29 -0500, Clay McCoy wrote:
[ . . . ]

> >why do they lack confidence?
>
> Again, you tell me. These are the words from the issue:
> "this is not really tested so it may have problems."
> "For errors in the task (and I suspect there are some), new issues should be raised."

The moral of this story then is that Groovy should apply traditional Big
Company marketeering language (i.e. positive comments only, absolutely
nothing negative ever -- don't actually lie, just avoid telling people
information if it is negative) to its documentation rather than just
being open and honest?
--
Russel.
=============================================================================
Dr Russel Winder Partner
xmpp: rus...@russel.org.uk
Concertant LLP t: +44 20 7585 2200, +44 20 7193 9203
41 Buckmaster Road, f: +44 8700 516 084 voip: sip:russel...@ekiga.net
London SW11 1EN, UK m: +44 7770 465 077 skype: russel_winder

[Martin C. Martin]

Hi Clay,

Clay McCoy wrote:
> These were odd responses, you just turned my questions around on me.

I think what's happening is, the Groovy team doesn't know the answer to
your question, so they're asking you for a little extra information.
They haven't focused on the issue of memory consumption during
compilation much, but are happy to help you.

>> hmm... you could help us here a little in telling how memory is used.
>> What is it that takes so much memory in your case?
>
> We are just trying to compile the code. If I knew what the compiler was doing that takes so much memory I would be fixing it.

They just need you to add two command line arguments to the compiler,
and to send them the information that results. So, you don't have to
learn how to hack the compiler to fix it, they're happy to do that. But
they need more information before they can.

Now that I think about it, didn't Alex commit something about ClassNode
fields doing lazy allocation of ArrayLists or arrays? e.g. for a type
label, we never use the "constructors" field, or "interfaces," "fields,"
etc., so there's no need to allocate empty ArrayLists for those. Was
that committed into 1.5.x but not trunk? Or vice versa?

Best,
Martin

[Jochen Theodorou]

Clay McCoy schrieb:

> These were odd responses, you just turned my questions around on me.
>
>> hmm... you could help us here a little in telling how memory is
>> used. What is it that takes so much memory in your case?
>
> We are just trying to compile the code. If I knew what the compiler
> was doing that takes so much memory I would be fixing it.

Clay, we are happy to fix the issue once we know what is causing the
problem. The groovy build itself and many other projects seem not to
have the same problem to that extend, so it might be something special
to your project. If it is something special to your project we will have
a problem reproducing the issue. The last time someone complained about
memory usage we actually could track the issue down to the joint
compilation and that actually the java compiler was using much more
memory than before. But your case seems to be different, so we are
asking for memory profile so that we can optimize the memory usage were
it actually might be worth it. Of course for that we need to know the
hotspots, and your hotspots, not were we assume they might be.

>> why do they lack confidence?
>
> Again, you tell me. These are the words from the issue: "this is not
> really tested so it may have problems." "For errors in the task (and
> I suspect there are some), new issues should be raised."

Ok, then let me tell you, that since then several months have passed and
nothing really happened. This mode is widely used and I have no problem
using it too. Confident enough?

bye blackdrag

--
Jochen "blackdrag" Theodorou
The Groovy Project Tech Lead (http://groovy.codehaus.org)
http://blackdragsview.blogspot.com/

[Clay McCoy]

I think the honest comments are great. Unfortunately I am being bit by this
one in particular, and I'm just trying to get more info.

On 7/23/09 3:24 AM, "Russel Winder" <russel...@concertant.com> wrote:

> Clay,
>
> On Wed, 2009-07-22 at 22:29 -0500, Clay McCoy wrote:
> [ . . . ]
>>> why do they lack confidence?
>>
>> Again, you tell me. These are the words from the issue:
>> "this is not really tested so it may have problems."
>> "For errors in the task (and I suspect there are some), new issues should be
>> raised."
>
> The moral of this story then is that Groovy should apply traditional Big
> Company marketeering language (i.e. positive comments only, absolutely
> nothing negative ever -- don't actually lie, just avoid telling people
> information if it is negative) to its documentation rather than just
> being open and honest?

[Clay McCoy]

I will provide a memory profile soon, and anything else that could be helpful. Thanks.

[Mike Dillon]

I think an instance/memory histogram for all loaded classes should be a
decent start. After that, someone may want you to dig into the heap and
see what certain instances are being retained by. I'm assuming you can't
just make the code or heap dump available for others to reproduce
independently, so all the diagnosis must be done by proxy. Hence the
requests for you to provide this information :)

-md

begin Clay McCoy quotation:

[Clay McCoy]

Based on these comments I was able to convince my company to go ahead and give Groovy 1.6.3 a try. We took the groovyc memoryMaximumSize from 1344m all the way down to 512m and still didn't get an OutOfMemoryError. This is quite significant for us! Hopefully it will work as well for the IntelliJ build. Now I just have to force Gmaven to use 1.6.3 somehow.
Thank you for all the helpful comments. I don't have a heapDump because the upgrade worked so well, but I did permanently put that option in our Ant script. (
<compilerarg value="-XX:+HeapDumpOnOutOfMemoryError"/> in the javac nested in the groovyc task).

On 7/22/09 3:15 PM, "Guillaume Laforge" <glaf...@codehaus.org> wrote:

[Mike Dillon]

I'm not sure what the default heap dump path is, but you probably want
to set it to a known directory.

-md

begin Clay McCoy quotation: