[groovy-user] ConfigSlurper / GroovyClassLoader memory leak

[asperkins]

The following snippet of code will bring a JVM down by filling up the perm
gen heap.

for (int i = 0; i < 10000; i++) {
new ConfigSlurper().parse("foo = 'bar'");
println i
}

8078
8079
8080
Caught: java.lang.OutOfMemoryError: PermGen space
at groovy.util.ConfigSlurper.parse(ConfigSlurper.groovy:123)

A line from jmap -permstat

class_loader classes bytes parent_loader alive? type

0x9527de80 23 96536 0x9527d940 dead
groovy/lang/GroovyClassLoader$InnerLoader@0x91620470

For every iteration there will be an instance like the one above.

Our application using Groovy config files extensively. It also reloads these
files at a frequent rate. This problem is causing quite a bit of pain at the
moment.

Any ideas?

--
View this message in context: http://www.nabble.com/ConfigSlurper---GroovyClassLoader-memory-leak-tp20115910p20115910.html
Sent from the groovy - user mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

[Jochen Theodorou]

asperkins schrieb:

> The following snippet of code will bring a JVM down by filling up the perm
> gen heap.

[...]

> 0x9527de80 23 96536 0x9527d940 dead
> groovy/lang/GroovyClassLoader$InnerLoader@0x91620470
>
> For every iteration there will be an instance like the one above.
>
> Our application using Groovy config files extensively. It also reloads these
> files at a frequent rate. This problem is causing quite a bit of pain at the
> moment.

the inner loader itself is not bad... it is bad if the outer class
loader still holds a reference to the class in its cache. So that might
be the problem... for example GroovyShell#evaluate uses more or less the
same class loader structure, but there is no such problem...

what version of groovy are you using?

bye blackdrag

--
Jochen "blackdrag" Theodorou
The Groovy Project Tech Lead (http://groovy.codehaus.org)
http://blackdragsview.blogspot.com/
http://www.g2one.com/

[asperkins]

I've tried 1.5.6, 1.5.7, and 1.6 beta.
--
View this message in context: http://www.nabble.com/ConfigSlurper---GroovyClassLoader-memory-leak-tp20115910p20116898.html

Sent from the groovy - user mailing list archive at Nabble.com.

[Jochen Theodorou]

asperkins schrieb:

>
>
> Jochen Theodorou wrote:
>> asperkins schrieb:
>>> The following snippet of code will bring a JVM down by filling up the
>>> perm gen heap.
>> [...]
>>> 0x9527de80 23 96536 0x9527d940 dead
>>> groovy/lang/GroovyClassLoader$InnerLoader@0x91620470
>>>
>>> For every iteration there will be an instance like the one above.
>>>
>>> Our application using Groovy config files extensively. It also reloads
>>> these
>>> files at a frequent rate. This problem is causing quite a bit of pain at
>>> the
>>> moment.
>> the inner loader itself is not bad... it is bad if the outer class
>> loader still holds a reference to the class in its cache. So that might
>> be the problem... for example GroovyShell#evaluate uses more or less the
>> same class loader structure, but there is no such problem...

when looking at the source I see that there is an ExpandoMetaclass
created for this class... probably one that cannot be collected... in
that case it is no wonder the permgen gets polluted. If that is the
reason, then you could try doing the following... use parsed scripts as
input for ConfigSlurper#parse, after parse set the metaClass in the
registry (GroovySystem.getMetaClassRegistry) to null by using
setMetaClass(Class,MetaClass). After this it should work. Then the class
can be unloaded. Not a nice workaround, but well...

[Robert Fischer]

I what circumstances will an ExpandoMetaClass not get unloaded?

--
~~ Robert Fischer.
Smokejumper Consulting http://smokejumperit.com
Enfranchised Mind Blog http://enfranchisedmind.com/blog
LinkedIn Profile http://www.linkedin.com/in/robertfischer
Twitter Feed http://twitter.com/robertfischer

[asperkins]

Is this what you mean? ( if so, it results in a nullpointer exception )

GroovyClassLoader classLoader = new GroovyClassLoader();
Script script = (Script) classLoader.parseClass(text).newInstance();
for (int i = 0; i < count; i++) {
ConfigObject config = new ConfigSlurper().parse(script);
GroovySystem.getMetaClassRegistry().setMetaClass(script.getClass(),
null);
}

java.lang.NullPointerException
at
org.codehaus.groovy.runtime.metaclass.ConcurrentReaderHashMap.put(ConcurrentReaderHashMap.java:488)
at
org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.setMetaClass(MetaClassRegistryImpl.java:292)
at SlurperMemoryLeak.go(SlurperMemoryLeak.java:24)
at SlurperMemoryLeak.main(SlurperMemoryLeak.java:15)

org.codehaus.groovy.runtime.metaclass.ConcurrentReaderHashMap.put
*/
public Object put(Object key, Object value) {
if (value == null)
throw new NullPointerException();
--
View this message in context: http://www.nabble.com/ConfigSlurper---GroovyClassLoader-memory-leak-tp20115910p20117287.html

Sent from the groovy - user mailing list archive at Nabble.com.

[asperkins]

I used the removeMetaClass method instead. Looks like that might be working.
I'll keep my fingers crossed.
--
View this message in context: http://www.nabble.com/ConfigSlurper---GroovyClassLoader-memory-leak-tp20115910p20117527.html

[Graeme Rocher]

When its been modified by the user

Cheers

--
Graeme Rocher
Grails Project Lead
G2One, Inc. Chief Technology Officer
http://www.g2one.com

[Tom Nichols]

Is this a bug that the class can't be unloaded? Someone else recently
had the same problem. If it is in fact a bug, I think it should be
fixed for 1.6

-Tom

[Jochen Theodorou]

Tom Nichols schrieb:

> Is this a bug that the class can't be unloaded? Someone else recently
> had the same problem. If it is in fact a bug, I think it should be
> fixed for 1.6

that was a different issue. It had nothing to do with constant meta
classes. In case of ConfigSlurper we have a custom meta class and that
meta class is modified, making it into a constant meta class, that can't
be unloaded automatically.

[asperkins]

Ok, It looks like the removeMetaClass wasn't what fixed my problem.

This leaks...

GroovyClassLoader classLoader = new
GroovyClassLoader(ClassUtils.getDefaultClassLoader());

for (int i = 0; i < 10000; i++) {

Script script = (Script) classLoader.parseClass("foo =
bar").newInstance();
new ConfigSlurper().parse(script);
}

Change "foo = bar" to a File and NO leak.

GroovyClassLoader classLoader = new
GroovyClassLoader(ClassUtils.getDefaultClassLoader());

for (int i = 0; i < 10000; i++) {

Script script = (Script) classLoader.parseClass(new
File("./someFile.groovy")).newInstance();
new ConfigSlurper().parse(script);
}

Any idea why this is the case?
--
View this message in context: http://www.nabble.com/ConfigSlurper---GroovyClassLoader-memory-leak-tp20115910p21607195.html

Sent from the groovy - user mailing list archive at Nabble.com.

[John Prystash]

This thread may help provide some more insight into the situation:
http://www.nabble.com/MetaClassRegistryImpl-memory-retainment-and-1.5.7-td19979094.html#a20040179

---

From: asperkins <tony.p...@travelocity.com>
To: us...@groovy.codehaus.org
Sent: Thursday, January 22, 2009 10:57:44 AM
Subject: Re: [groovy-user] ConfigSlurper / GroovyClassLoader memory leak

[asperkins]

I noticed in 1.6 rc2 that the following code still runs out of perm gen. We
use the config slurper extensively and this is a rather bothersome issue.
I've had to resort to writing the string out to a file, then loading it via
the class loader, but even then I have to reuse filenames or the classes
still pile up and I run out of perm gen.

Any ideas on what to do if I have an application that is retrieving groovy
config objects via a rest call and parsing them?

for (int i = 0; i < 50000; i++) {
ConfigObject config = new ConfigSlurper().parse("foo = bar");
print " ${i}"
}

BTW, The removal of the metaclass didn't seem to help this issue.
This didn't work...

GroovyClassLoader classLoader = new GroovyClassLoader();

for (int i = 0; i < 50000; i++) {

Script script = (Script) classLoader.parseClass("foo = bar").newInstance();

ConfigObject config = new ConfigSlurper().parse(script);

print " ${i}"
GroovySystem.getMetaClassRegistry().removeMetaClass(script.getClass() );
}

--
View this message in context: http://www.nabble.com/ConfigSlurper---GroovyClassLoader-memory-leak-tp20115910p21708060.html

[asperkins]

I think I found a work-around.
If I call clearCache on the classLoader everytime, it seems to work. The
perm gen gets cleaned up when it hits the max.

GroovyClassLoader classLoader = new GroovyClassLoader();
for (int i = 0; i < 50000; i++) {
Script script = (Script) classLoader.parseClass("foo = bar").newInstance();
ConfigObject config = new ConfigSlurper().parse(script);
print " ${i}"
GroovySystem.getMetaClassRegistry().removeMetaClass(script.getClass() );

classLoader.clearCache();
}

asperkins wrote:
>
> I noticed in 1.6 rc2 that the following code still runs out of perm gen.
> We use the config slurper extensively and this is a rather bothersome
> issue. I've had to resort to writing the string out to a file, then
> loading it via the class loader, but even then I have to reuse filenames
> or the classes still pile up and I run out of perm gen.
>
> Any ideas on what to do if I have an application that is retrieving groovy
> config objects via a rest call and parsing them?
>
> for (int i = 0; i < 50000; i++) {
> ConfigObject config = new ConfigSlurper().parse("foo = bar");
> print " ${i}"
> }
>
> BTW, The removal of the metaclass didn't seem to help this issue.
> This didn't work...
> GroovyClassLoader classLoader = new GroovyClassLoader();
> for (int i = 0; i < 50000; i++) {
> Script script = (Script) classLoader.parseClass("foo =
> bar").newInstance();
> ConfigObject config = new ConfigSlurper().parse(script);
> print " ${i}"
> GroovySystem.getMetaClassRegistry().removeMetaClass(script.getClass() );
> }
>
>

--
View this message in context: http://www.nabble.com/ConfigSlurper---GroovyClassLoader-memory-leak-tp20115910p21709296.html

[Jochen Theodorou]

asperkins schrieb:

> I noticed in 1.6 rc2 that the following code still runs out of perm gen. We
> use the config slurper extensively and this is a rather bothersome issue.
> I've had to resort to writing the string out to a file, then loading it via
> the class loader, but even then I have to reuse filenames or the classes
> still pile up and I run out of perm gen.
>
> Any ideas on what to do if I have an application that is retrieving groovy
> config objects via a rest call and parsing them?
>
> for (int i = 0; i < 50000; i++) {
> ConfigObject config = new ConfigSlurper().parse("foo = bar");
> print " ${i}"
> }
>
> BTW, The removal of the metaclass didn't seem to help this issue.
> This didn't work...
> GroovyClassLoader classLoader = new GroovyClassLoader();
> for (int i = 0; i < 50000; i++) {
> Script script = (Script) classLoader.parseClass("foo = bar").newInstance();
> ConfigObject config = new ConfigSlurper().parse(script);
> print " ${i}"
> GroovySystem.getMetaClassRegistry().removeMetaClass(script.getClass() );
> }

Running out of permgen means that the
classes created through classLoader.parseClass("foo = bar") are not
collected. But a class cannot be collected if the class loader is still
alive.

Since you do not discard the class loader in your loop, there is no
chance the loader gets collected. Put the classLoader in the loop and
all should be fine

bye blackdrag

--
Jochen "blackdrag" Theodorou
The Groovy Project Tech Lead (http://groovy.codehaus.org)
http://blackdragsview.blogspot.com/

---------------------------------------------------------------------

[asperkins]

You are correct!

When I tested that I didn't allow the app to run long enough to see the perm
gen clean up in jprofiler.

Thanks!

Tony

--
View this message in context: http://www.nabble.com/ConfigSlurper---GroovyClassLoader-memory-leak-tp20115910p21710226.html

Sent from the groovy - user mailing list archive at Nabble.com.

[filirom1]

I have exactly the same issue:

I use a ConfigSlurper in order to send emails in a Java Application.

After few hundred emails, I got a PermGen Exception.

Here is the Java code I had before :

ConfigSlurper slurper = new ConfigSlurper();
slurper.setBinding(variables);
ConfigObject conf = slurper.parse(template);

And now, the updated code without permgen, thanks to this mailing list:

GroovyClassLoader classLoader = new GroovyClassLoader();

Script script = (Script) classLoader.parseClass(template).newInstance();
ConfigSlurper slurper = new ConfigSlurper();
slurper.setBinding(variable);
ConfigObject conf = slurper.parse(script);
GroovySystem.getMetaClassRegistry().removeMetaClass(script.getClass());
classLoader.clearCache();

Is it possible to write a caution in the ConfigSlurper documentation
http://groovy.codehaus.org/gapi/groovy/util/ConfigSlurper.html ?

It could save some times to other developers, that use the ConfigSlurper
inside Java code.

http://groovy.329449.n5.nabble.com/file/n3280248/jconsole_permgen_exception.png

Cheers

Romain
--
View this message in context: http://groovy.329449.n5.nabble.com/ConfigSlurper-GroovyClassLoader-memory-leak-tp364279p3280248.html

Sent from the groovy - user mailing list archive at Nabble.com.

---------------------------------------------------------------------