Re: Out of memory exception from the javac compiler

[Reinier Zwitserloot]

We did make a tiny change here but I'm fairly sure the only place that code is live right now is in the edge build; is that the version of lombok you are using perhaps?

Lombok isn't building a parse tree, it's building a node tree. (javac has already done all the parsing). As java didn't itself end up using too much memory it is exceedingly unlikely that lombok's more compact tree did result in exceeding your heap. So, I'm guessing there's an infinite loop someplace instead, and you ran into a bug that our tests didn't catch.

How many files are in the tree you are compiling? It would help a lot if we can isolate the source file that is causing this issue. Catching OutOfMemoryError is very very tricky and not guaranteed to work, but at the very least I can build a custom version for you that will start emitting the names of the source files once it notices it's 50 nodes deep, which is highly unusual for a normal java file.

On Thursday, May 23, 2013 9:56:02 PM UTC+2, Doug wrote:

Just trying out lombok for the first time.  Added the jar and a first sample annotation (@ToString).  All looked good on a simple example program.

Then tried adding this class to a library in the main build of a project and got a memory exception from the compiler.  It appears lombok is building some sort of parse tree and uses too much memory to do it.

Any ideas?

Doug

E:\fp\Main\Source\Java>javac -version

javac 1.6.0_04

    [mkdir] Created dir: E:\fp\Main\Source\Java\tmpclasses

    [javac] Compiling 543 source files to E:\fp\Main\Source\Java\tmpclasses

    [javac]

    [javac]

    [javac] An annotation processor threw an uncaught exception.

    [javac] Consult the following stack trace for details.

    [javac] java.lang.OutOfMemoryError: Java heap space

    [javac]     at lombok.core.AST.buildWithField(AST.java:247)

    [javac]     at lombok.javac.JavacAST.drill(JavacAST.java:262)

    [javac]     at lombok.javac.JavacAST.buildStatementOrExpression(JavacAST.jav

a:257)

    [javac]     at lombok.javac.JavacAST.buildTree(JavacAST.java:157)

    [javac]     at lombok.javac.JavacAST.buildTree(JavacAST.java:63)

    [javac]     at lombok.core.AST.buildWithField0(AST.java:350)

    [javac]     at lombok.core.AST.buildWithField(AST.java:248)

    [javac]     at lombok.javac.JavacAST.drill(JavacAST.java:262)

    [javac]     at lombok.javac.JavacAST.buildStatementOrExpression(JavacAST.jav

a:257)

    [javac]     at lombok.javac.JavacAST.buildTree(JavacAST.java:157)

    [javac]     at lombok.javac.JavacAST.buildTree(JavacAST.java:63)

    [javac]     at lombok.core.AST.buildWithField0(AST.java:350)

    [javac]     at lombok.core.AST.buildWithField(AST.java:248)

    [javac]     at lombok.javac.JavacAST.drill(JavacAST.java:262)

    [javac]     at lombok.javac.JavacAST.buildStatementOrExpression(JavacAST.jav

a:257)

    [javac]     at lombok.javac.JavacAST.buildTree(JavacAST.java:157)

    [javac]     at lombok.javac.JavacAST.buildTree(JavacAST.java:63)

    [javac]     at lombok.core.AST.buildWithField0(AST.java:350)

    [javac]     at lombok.core.AST.buildWithField(AST.java:248)

    [javac]     at lombok.javac.JavacAST.drill(JavacAST.java:262)

    [javac]     at lombok.javac.JavacAST.buildStatementOrExpression(JavacAST.jav

a:257)

    [javac]     at lombok.javac.JavacAST.buildExpression(JavacAST.java:242)

    [javac]     at lombok.javac.JavacAST.buildLocalVar(JavacAST.java:209)

    [javac]     at lombok.javac.JavacAST.buildStatementOrExpression(JavacAST.jav

a:253)

    [javac]     at lombok.javac.JavacAST.buildTree(JavacAST.java:157)

    [javac]     at lombok.javac.JavacAST.buildTree(JavacAST.java:63)

    [javac]     at lombok.core.AST.buildWithCollection(AST.java:381)

    [javac]     at lombok.core.AST.buildWithField0(AST.java:355)

    [javac]     at lombok.core.AST.buildWithField(AST.java:248)

    [javac]     at lombok.javac.JavacAST.drill(JavacAST.java:262)

    [javac]     at lombok.javac.JavacAST.buildStatementOrExpression(JavacAST.jav

a:257)

    [javac]     at lombok.javac.JavacAST.buildTree(JavacAST.java:157)

BUILD FAILED

E:\fp\Main\Source\Java\build.xml:486: Compile failed; see the compiler error out

put for details.

[Reinier Zwitserloot]

Presumably the compiler still has the same max heap size configuration, but jumping up to 64-bit means that the same heap now takes up more memory, which could explain this result.

I assume that you're not on the edge release?

I'm going to work on a new edge release that will attempt to catch this. I guess I'll design it so that it'll always trigger on OOME, but catching OOME is not guaranteed to actually do anything according to the JVM spec. I'll add a -D switch which will also print unusually deep trees, with the -D parameter configuring the definition of 'unusually deep'.

I'll keep you posted when I finish this feature.

[Reinier Zwitserloot]

Anders, is that the entire stack trace? The whole thing? That's only 3 levels deep. If this is the full trace, it's not an issue of extremely deeply nested ASTs.

Doug's is also seemingly short. So, I've just done the reporting when OOMEs occur, I don't think (unless both of these stack traces are cut short or something really weird is going on) there's a point in adding a 'report if nesting is deeper than X' option. It's also just for javac for now.

Could you two grab http://projectlombok.org/download-edge.html and give this another shot? Hopefully the message of the OOME should include the filename and the position. The position will be rather cryptic; it's the raw character position, not split up in line / column. If the catch block on an OOME works at all, in general you should assume there's almost no heap space left so I don't want to risk creating new OOME conditions querying the line number table. Still, it's something to go on. 

On Wednesday, June 12, 2013 9:27:02 AM UTC+2, Anders Kobberup wrote:

Did you make a version that does emit the names of classes with unusual deep node trees? 

We are facing the same problem after moving our build server to a new machine, though in our case it is not a Java Heap Space, but a GC Overhead limit exceeded, but that is almost the same..

"if too much time is being spent in garbage collection: if more than 98% of the total time is spent in garbage collection and less than 2% of the heap is recovered, an OutOfMemoryError will be thrown."

The old build server were a 32 bit running java 1.6_xx whereas the new is 64bit running java 1.7_xx - i dont know if that helps. The new box has 2 gb heap where the old one only had 1.

Lombok versions tried: 0.11.6 and 0.11.8

 

An annotation processor threw an uncaught exception.

Consult the following stack trace for details.

java.lang.OutOfMemoryError: GC overhead limit exceeded

at lombok.javac.JavacAST.drill(JavacAST.java:263)

at lombok.javac.JavacAST.buildStatementOrExpression(JavacAST.java:257)

at lombok.javac.JavacAST.buildTree(JavacAST.java:157)

at lombok.javac.JavacAST.buildTree(JavacAST.java:63)

at lombok.core.AST.buildWithCollection(AST.java:381)

at lombok.core.AST.buildWithField0(AST.java:355)

at lombok.core.AST.buildWithField(AST.java:248)

at lombok.javac.JavacAST.drill(JavacAST.java:262)

at lombok.javac.JavacAST.buildStatementOrExpression(JavacAST.java:257)

at lombok.javac.JavacAST.buildTree(JavacAST.java:157)

at lombok.javac.JavacAST.buildTree(JavacAST.java:63)

at lombok.core.AST.buildWithField0(AST.java:350)

at lombok.core.AST.buildWithField(AST.java:248)

at lombok.javac.JavacAST.drill(JavacAST.java:262)

at lombok.javac.JavacAST.buildStatementOrExpression(JavacAST.java:257)

at lombok.javac.JavacAST.buildStatement(JavacAST.java:246)

at lombok.javac.JavacAST.buildMethod(JavacAST.java:226)

at lombok.javac.JavacAST.buildType(JavacAST.java:188)

at lombok.javac.JavacAST.buildCompilationUnit(JavacAST.java:169)

at lombok.javac.JavacAST.<init>(JavacAST.java:81)

at lombok.javac.JavacTransformer.transform(JavacTransformer.java:67)

at lombok.javac.apt.Processor.process(Processor.java:256)

at lombok.core.AnnotationProcessor$JavacDescriptor.process(AnnotationProcessor.java:117)

at lombok.core.AnnotationProcessor.process(AnnotationProcessor.java:169)

at com.sun.tools.javac.processing.JavacProcessingEnvironment.callProcessor(JavacProcessingEnvironment.java:793)

at com.sun.tools.javac.processing.JavacProcessingEnvironment.discoverAndRunProcs(JavacProcessingEnvironment.java:722)

at com.sun.tools.javac.processing.JavacProcessingEnvironment.access$1700(JavacProcessingEnvironment.java:97)

at com.sun.tools.javac.processing.JavacProcessingEnvironment$Round.run(JavacProcessingEnvironment.java:1029)

at com.sun.tools.javac.processing.JavacProcessingEnvironment.doProcessing(JavacProcessingEnvironment.java:1163)

at com.sun.tools.javac.main.JavaCompiler.processAnnotations(JavaCompiler.java:1108)

at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:824)

at com.sun.tools.javac.main.Main.compile(Main.java:439)

/Anders