Thread locals on FieldReferenceLinker holding the GC and the world hostage

[Heikki Vesalainen]

Hi,

our project (running on Tomcat) has been running out of PermGen after
a few rounds of redeploying the app. On each undeploy/stop Tomcat
gives this warning:

SEVERE: The web application [/foobar] created a ThreadLocal with key
of type [null] (value [org.squeryl.internals.FieldReferenceLinker$$anon
$1@e8787a]) and a value of type
[org.squeryl.internals.FieldReferenceLinker.YieldInspection] (value
[org.squeryl.internals.FieldReferenceLinker$YieldInspection@105fa26])
but failed to remove it when the web application was stopped. This is
very likely to create a memory leak.

I started to debug this and found out that a lot of classes and
objects related to the use of Squeryl don't get garbage collected.

The root cause (as per GC root analysis) seems to be in the
ThreadLocals of FieldReferenceLinker and especially the
queryExpressionNode.

Is anybody working on this problem and does anybody have a
workaround?

Couldn't the stuff in the ThreadLocals be instead in the Session and
cleared at the end of each transaction? Anyway, something needs to be
done. Any ideas?

Heikki

[Heikki Vesalainen]

Is there a bug in
FieldReferenceLinker.YieldInspection.turnOffAndCollectOutExpressions?
Should it actually clear queryExpressionNode (that is set by turnOn)?

[Heikki Vesalainen]

Fixed that, but still no sigar.

--> java.lang.ThreadGroup@0x71862f60 (43 bytes) (field threads:)
--> [Ljava.lang.Thread;@0x71ae0ab0 (72 bytes) (Element 1 of
[Ljava.lang.Thread;@0x71ae0ab0:)
--> java.lang.ref.Finalizer$FinalizerThread@0x718630e0 (104 bytes)
(field threadLocals:)
--> java.lang.ThreadLocal$ThreadLocalMap@0x71948b78 (20 bytes) (field
table:)
--> [Ljava.lang.ThreadLocal$ThreadLocalMap$Entry;@0x71cbdc40 (136
bytes) (Element 15 of [Ljava.lang.ThreadLocal$ThreadLocalMap
$Entry;@0x71cbdc40:)
--> java.lang.ThreadLocal$ThreadLocalMap$Entry@0x71b34148 (28 bytes)
(field value:)
--> org.squeryl.internals.FieldReferenceLinker
$YieldInspection@0x71b34dc8 (25 bytes) (??:)
--> class org.squeryl.internals.FieldReferenceLinker$YieldInspection
(84 bytes) (??:)
--> org.apache.catalina.loader.WebappClassLoader@0x71af78e8 (164
bytes)

Still the _yieldInspectionTL is holding the WebappClassLoader (which
in turn references all the other classes loaded in the webapp) for
ransom.

When could this ThreadLocal be freed?

[goldmar]

Hello,

I've got exactly the same problem! I'm also using Squeryl with Lift
(via squeryl-record) and I have to restart Tomcat after every second
redeploy. (On the production server, locally I'm using JRebel.) I
actually wanted to post about this issue myself but you were faster :-
D

I'd also like to know if there is a workaround or if anybody is
working on this problem.

Regards,
Mark



On 5 Sep., 21:59, Heikki Vesalainen <heikkivesalai...@yahoo.com>
wrote:

[Maxime Lévesque]

I don't use tomcat, but after reading this :

 http://wiki.apache.org/tomcat/MemoryLeakProtection

my understanding is that before tomcat 7.0.6, threads pools are preserved upon redeploys,
while classe loaders are not, so for any use of TLs, one must clear them before
undeploying or else you get a classloaded leak.

Which version of Tomcat are you using ?

This here might be a solution :

>In this particular case, the leak is detected and a message is logged. Tomcat 6.0.24 to 6.0.26 modify internal structures of the JDK (ThreadLocalMap) to >remove the reference to the ThreadLocal instance, but this is unsafe (see #48895) so that it became optional and disabled by default from 6.0.27. Starting >with Tomcat 7.0.6, the threads of the pool are renewed so that the leak is safely fixed.


At the end of a transaction all TLs with FieldReferenceLinker could be cleared, but I wonder
if it's worth doing, if renewing the threads upon redeploy solves it.


ML

2011/9/5 Heikki Vesalainen <heikkive...@yahoo.com>

[Heikki Vesalainen]

> At the end of a transaction all TLs with FieldReferenceLinker could be
> cleared, but I wonder if it's worth doing, if renewing the threads upon
> redeploy solves it.

Installed tomcat 7.0.21, which now spits out:

06-Sep-2011 10:06:37 org.apache.catalina.loader.WebappClassLoader
checkThreadLocalMapForLeaks

SEVERE: The web application [/foobar] created a ThreadLocal with key

of type [scala.util.DynamicVariable$$anon$1] (value
[scala.util.DynamicVariable$$anon$1@94d313]) and a value of type
[org.scalatra.util.MultiMap] (value [Map()]) but failed to remove it
when the web application was stopped. Threads are going to be renewed
over time to try and avoid a probable memory leak.

Alas, there is no change. Even though I see tomcat eventually killing
all the http threads, the thread locals are still referenced through
the FinalizerThread.

So I really think this problem needs to be fixed.

Heikki

[Heikki Vesalainen]

It seems I got it fixed without doing wide spread changed. I'm going
to now test it a little and will report back with a patch if it's a
success.

[David Whittaker]

Heikki,

Since it's a Scalatra map I doubt Squeryl is at fault.  I'm guessing that your upgrade fixed your previous issue but uncovered a new issue with Scalatra itself.

-Dave

[Heikki Vesalainen]

David,

both Scalatra (M4) and Squeryl had the problem and I needed a fix to
both to get it working. Scalatra project had already fixed it for M1
so I had to only update.

I fixed Squeryl Session and FieldReferenceLinker myself and now
everything is working nicely. I'll push my changes to my fork of
Squeryl and make a pull request.

Heikki

[Heikki Vesalainen]

My fix for your humour or horror:

https://github.com/hvesalai/Squeryl/commit/f92a560e4abd6181e2edfd49739e306fec2f0d2a

[goldmar]

Can this be incorporated into Squeryl?

[Maxime Lévesque]

It just went into master :

  https://github.com/max-l/Squeryl/network

ML

2011/9/11 goldmar <bigan...@gmail.com>

[goldmar]

Great! :-)

On 11 Sep., 16:27, Maxime Lévesque <maxime.leves...@gmail.com> wrote:
> It just went into master :
>
>  https://github.com/max-l/Squeryl/network
>
> ML
>

> 2011/9/11 goldmar <bigande...@gmail.com>