Netty thread pool issue

[Bogdan SOLGA]

Hello, everyone!

I have recently found the Hikari connection pool, for which I would like to congratulate Brett for the excellent job :)

I have started to use it on a personal project, replacing the existing Tomcat connection pool. I have made the integration via Spring XML config, and I have also used the recommended pgjdbc-ng PostgreSQL JDBC driver.

Shortly, my config is:
<bean id=“dataSource" class="com.zaxxer.hikari.HikariDataSource" destroy-method="close">
        <constructor-arg ref="hikariConfig"/>
</bean>

<bean id="hikariConfig" class="com.zaxxer.hikari.HikariConfig">
    <property name="dataSourceClassName" value="com.impossibl.postgres.jdbc.PGDataSource"/>
    <property name="dataSourceProperties" >
        <props>
            <prop key="host, port, database, ..."/>
        </props>
    </property>
</bean>

When I stop Tomcat (8.0.15), I see a lot of errors with the following format.

04-Dec-2014 15:25:16.446 SEVERE [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [] created a ThreadLocal with key of type [io.netty.util.internal.FastThreadLocal$1] (value [io.netty.util.internal.FastThreadLocal$1@6e4aae7b]) and a value of type [io.netty.util.Recycler.Stack] (value [io.netty.util.Recycler$Stack@690545ce]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

As far as I can see, they are related to the Netty thread pool, which is used internally either by Hikari and / or pgjdbc-ng driver (I haven't verified which one is using it).

Can anyone point me to any misconfiguration issue I might have introduced?

Thank you!

Kind regards,

Bogdan

[Brett Wooldridge]

Thanks, Bogdan.  There is no misconfiguration on your part.  The issue is two fold.  Tomcat is extremely strict about checking ThreadLocal use.  HikariCP has had to bend over backward to avoid these leak warnings (see issue #39, #148).  Basically, in order to avoid these Tomcat warnings, a library cannot subclass ThreadLocal at all.  For some libraries, this is a tall order.  We also had to bend over backward with the addition of WeakReferences.

A basic (but technical) summary of the issue is this.  When Tomcat starts a WAR, all classes in that WAR are loaded by a dedicated ClassLoader.  In Java, classes are only garbage collected when the ClassLoader that loaded them is itself garbage collected.  When Tomcat un-deploys a WAR, it throws away the dedicated ClassLoader.  However...

Because threads are shared across WARs, if a class in the WAR puts associates a ThreadLocal subclass that is itself contained in the WAR ... or if a class even puts a class local to the WAR into a regular ThreadLocal ... then when the WAR is un-deployed, the ThreadLocals (which are tied to threads whose lifetime extend beyond the WAR) retain references to classes loaded by the dedicated ClassLoader.  This prevents the ClassLoader itself, and all of the classes that it loaded from being garbage collected.

ThreadLocals are completely isolated from other threads, and there is no way for Tomcat or even the library itself to clean them up.

I understand Tomcat's motivation for the warning.  And it did use to be a WARN.  Now it is classified as SEVERE.  I think this is a little bit disingenuous.  As noted in the message, " Threads are going to be renewed over time to try and avoid a probable memory leak."  This will prevent any leak.  So, in reality, whether a WAR leaks or not is ultimately in Tomcat's hands.  If they detect these thread locals, they could simply more aggressively schedule the retirement of existing threads and the creation of new ones.

TLDR; Tomcat is "correct", but the warning is ultimately BS and mostly scares users.

The "issue" ultimately lies with netty, but there is about a 0.5% chance of them changing what they are doing.

[Bogdan SOLGA]

Hello, Brett!

Thank you very much for the thorough explanation!

Besides (plainly) ignoring those warnings / errors... would there be any other recommendation on how to deal with them?

Thank you, and keep up the good work, Brett!

Kind regards,
Bogdan