[Tomcat & JDBI] Potential ThreadLocal memory leak

[Michael Seele]

Hi,

i'm using jdbi in a web application running in a tomcat and get the following error when the tomcat tries to undeploy the war:

12-Oct-2016 13:15:15.654 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@1fda589d]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@34e4ac92]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Seems like i'm doing something wrong or jdbi has a memory leak bug. Has somebody ever seen this error before and has a idea what the problem is?

What i find out is that the ThreadLocal object org.skife.jdbi.v2.BasicHandle.sqlObjectContext does never call remove(). Can this be a potential memory leak of jdbi?

Thanks,

Michael

[Matthew Hall]

Hi Michael,

That ThreadLocal is instance-bound, so something is holding a reference to that BasicHandle instance somewhere. The most common culprit I've seen is an on-demand with an unclosed statement attached to it.

Search your code for any on-demand SQL objects, through DBI.onDemand(Class). Look for methods on your SQL object types that return Iterables. Ensure that all callers of those method are either consuming the entire iterable, or are actively closing the iterator e.g. in a try-with-resources block.

Hope this helps. Let me know what you find.

-Matthew

--
You received this message because you are subscribed to the Google Groups "jDBI" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jdbi+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Seele]

Hi Matthew,

thanks for your fast reply.
I'm not using the onDemand method, i only use org.skife.jdbi.v2.IDBI.open(Class<SqlObjectType>) in try-with-resources block's. What i use in my DAO's is the org.skife.jdbi.v2.sqlobject.mixins.GetHandle interface to use the current handle to execute a method in another dao. here is a simple example:

public abstract class ExampleDAO implements Closeable, GetHandle {

@SqlQuery("SELECT e.id FROM example e")

public abstract int getExample();

@Transaction

public void example() {

int example = getExample();

getHandle().attach(Example2DAO.class).processExample(example);

}

}

public abstract class Example2DAO implements Closeable {

@SqlBatch("INSERT INTO example (example) VALUES (:example)")

public abstract void processExample(@Define("example") int example);

}

try (ExampleDAO dao = dbi.open(ExampleDAO.class)) {

dao.example();

}

Is there anything wrong at this code?

Thanks,

Michael

To unsubscribe from this group and stop receiving emails from it, send an email to jdbi+uns...@googlegroups.com.

[Matthew Hall]

Hi Michael,

I don't see anything in your code example that would contribute to a memory leak. I'm sorry I can't help you further--there's just not much information to go on here.

If you see the problem happen again, see if you can collect a heap dump from the running VM and pick through it in jhat. Once we know what's holding on to that `BasicHandle` reference, cleaning up the leak should be straightforward.

Regards,

-Matthew

To unsubscribe from this group and stop receiving emails from it, send an email to jdbi+unsubscribe@googlegroups.com.

[Michael Seele]

Hi Matthew,

i got the same effect again. Here is a link to the dump i've created with jmap (jmap -dump:file=dump.dat <ps_id>) on the running vm: https://drive.google.com/open?id=0B6e86GGLWDffTHR0SGxfLTNZM0k

I don't find the problem. Do you find time to look into it to find the problem? Whould be a big help.

Thanks,

Michael

[Michael Seele]

Here are the log messages i've got:

26-Oct-2016 11:47:03.983 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161014-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@32d50cbb]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@62aaf3c0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

26-Oct-2016 11:47:03.984 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161014-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@55124885]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@50495082]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

26-Oct-2016 11:47:03.984 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161014-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@4a633ee4]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@7f62e09b]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

26-Oct-2016 11:47:03.984 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161014-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@38f3c992]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@2224434e]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

26-Oct-2016 11:47:03.985 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161014-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@7163cdf0]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@19ed8d7f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

26-Oct-2016 11:47:03.985 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161014-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@3ff46889]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@3fca720a]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

26-Oct-2016 11:47:03.985 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161014-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@5029244e]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@25dadd41]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

26-Oct-2016 11:47:03.985 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161014-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@39764953]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@55c03655]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

26-Oct-2016 11:47:03.985 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161014-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@4cd273f7]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@3f118ff9]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

[Matthew Hall]

Sure thing Michael, I'll have a look.

To unsubscribe from this group and stop receiving emails from it, send an email to jdbi+unsubscribe@googlegroups.com.

[Matthew Hall]

Do you have JDBI living at the tomcat container level or at the application (WAR/EAR) level?

[Matthew Hall]

I have a guess of what's holding those objects in memory:

In the SqlObject class, we have a handful of static final ConcurrentHashMaps with Class keys. We should use a WeakHashMap instead, which should allow unloaded classes to be GC'd.

I've opened an issue on Github: https://github.com/jdbi/jdbi/issues/543

-Matthew

[Matthew Hall]

Opened pull request: https://github.com/jdbi/jdbi/pull/544

Michael, would you be able to build from that branch, and see if you can duplicate the class leak in your container with these changes?

-Matthew

[Michael Seele]

Hi,

is it possible that you provide a binary jar for me (e.g.
jdbi-2.77-class-leak.jar)? We don't use maven and it would be complex
for me to set everything up just for the test.
If you provide a jdbi jar with the patch, i'll test it for you.

>>> Do you have JDBI living at the tomcat container level or at the
>>> application (WAR/EAR) level?

We use the jdbi at application level (WAR) and we use Automatic
Application Deployment
(http://tomcat.apache.org/tomcat-7.0-doc/config/host.html#Automatic%20Application%20Deployment).

Thanks,
Michael

> You received this message because you are subscribed to a topic in the
> Google Groups "jDBI" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/jdbi/A-PJfuNswpM/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

[Matthew Hall]

Try this one. I had to rename it to .zip for Google Mail to let me attach it--you'll probably need to rename it back to .jar?

-Matthew

>>>>> an email to jdbi+unsubscribe@googlegroups.com.

>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>
>>
>
> --

> You received this message because you are subscribed to a topic in the
> Google Groups "jDBI" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/jdbi/A-PJfuNswpM/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

> jdbi+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "jDBI" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jdbi+unsubscribe@googlegroups.com.

[Michael Seele]

Hi Matthew,

thanks for the jar. I've tried it and it seems to work half ;)
I have less Errors now but there are still a few of them:

28-Oct-2016 06:44:31.856 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161028-03] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@55fc8354]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@52c1c3fb]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

28-Oct-2016 06:44:31.857 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161028-03] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@34bec78f]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@683c14f8]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

28-Oct-2016 06:44:31.857 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161028-03] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@4826b30]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@4934aacf]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

28-Oct-2016 06:44:31.857 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161028-03] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@bf48ed9]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@367d08b]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

28-Oct-2016 06:44:31.857 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161028-03] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@142bf03c]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@782542a4]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

28-Oct-2016 06:44:31.858 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161028-03] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@5788893f]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@6ad6e8cd]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

28-Oct-2016 06:44:31.858 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161028-03] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@21374fbb]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@7355095b]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

28-Oct-2016 06:44:31.858 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161028-03] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@6087a77f]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@3d4aab22]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

28-Oct-2016 06:44:31.858 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161028-03] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@261b61a7]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@7d5fbbee]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

28-Oct-2016 06:44:31.858 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161028-03] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@203e4765]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@48c5ebfb]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

28-Oct-2016 06:44:31.859 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161028-03] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@35aebd1d]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@352ee08d]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Here is a new dump where i used your jar: https://drive.google.com/open?id=0B6e86GGLWDffbDlIRjJTYjdqOGs

Please note: The dump is packed as .tgz to save space. You have to extract the file before you can use it.

Btw, should we countine to discuss the problem in your opened bug at github?

Thanks,

Michael

> You received this message because you are subscribed to a topic in the
> Google Groups "jDBI" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/jdbi/A-PJfuNswpM/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

[Michael Seele]

Here's another dump (tgz again): https://drive.google.com/file/d/0B6e86GGLWDffc2RmeXRqWEJqejQ/view?usp=sharing

And here are the messages for this dump:

02-Nov-2016 12:32:47.444 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key o

f type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@36430591]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@21699cd6]) but failed to remove it when the w

eb application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.444 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key o

f type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@6d612e9c]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@6f5829b2]) but failed to remove it when the w

eb application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.444 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key o

f type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@4199c645]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@2cfb7d8c]) but failed to remove it when the w

eb application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.445 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key o

f type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@480aa341]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@66b1164b]) but failed to remove it when the w

eb application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.445 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key o

f type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@1ef55885]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@378f0b1]) but failed to remove it when the we

b application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.445 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key o

f type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@6d95bddb]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@2ebaa3b0]) but failed to remove it when the w

eb application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.446 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key o

f type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@1abfe792]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@5e86610c]) but failed to remove it when the w

eb application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.446 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key o

f type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@1ebd6462]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@65e1f326]) but failed to remove it when the w

eb application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.446 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key o

f type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@870b7f]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@6de49462]) but failed to remove it when the web

 application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.446 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key o

f type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@180f73c8]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@721dcb2c]) but failed to remove it when the w

eb application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.447 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key o

f type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@4a79eae3]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@377304a]) but failed to remove it when the we

b application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.447 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key o

f type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@608582df]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@7c5d7148]) but failed to remove it when the w

eb application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.447 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key o

f type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@7976a2d2]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@43b938d6]) but failed to remove it when the w

eb application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.447 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key o

f type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@fe6d904]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@18a62e81]) but failed to remove it when the we

b application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.448 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key o

f type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@540a7f75]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@5eff92aa]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.448 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@67684e88]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@4e93a6c2]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.448 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@49e599b2]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@6b2d38e6]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.449 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@4e943705]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@1d622da3]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.449 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@782f0fb4]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@7a47e25f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.450 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@1987f62b]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@19d50800]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.450 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@5c45447f]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@6f820d5c]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.450 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@49928829]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@78b60c13]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.451 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@e26d9c6]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@7131cdb3]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.451 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@2cbfd07c]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@64a1976c]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.451 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@5cf18ad7]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@235288af]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.451 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@71bc9089]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@194e7325]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.452 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@4939f5e9]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@7e191b91]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.452 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@6bb4f403]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@189650b]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.452 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@7994e6e]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@31ea089f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.452 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@7db2f64d]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@375965cd]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.452 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@7afeb776]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@1b31e646]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.453 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@71db7bf3]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@d501f08]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.453 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@29a5cd3]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@160d9910]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.453 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@3c237cc1]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@31068c12]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.453 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@612a2225]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@1b95e209]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.453 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@3c7740ce]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@39942d4e]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.453 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@91f1b53]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@3dae25d3]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.454 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@3bd97398]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@23bfc7f0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.454 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@520b8557]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@1e47b278]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.454 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@6e72579b]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@332f8de2]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.533 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@3d0097ed]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@1e773647]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.533 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [null] (value [null]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@62c5649d]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.533 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@2dcaa5ba]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@144cf011]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.534 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@33ae29d1]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@7a8652b5]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

02-Nov-2016 12:32:47.534 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@299cf2a]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@71a66b10]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Thanks,

Michael

[Matthew Hall]

Hi Michael,

I'm prepping a JDBI presentation so I won't have spare to time look at this until Friday.

-Matthew

To unsubscribe from this group and stop receiving emails from it, send an email to jdbi+unsubscribe@googlegroups.com.

[Michael Seele]

Hi Mathew,

no rush, i only want to give you more data ;)

-Michael

[Steven Schlansker]

Hi Michael,

I have a strong suspicion this will fix your problem
(or at least get us on to diagnosing the next leak):

https://github.com/jdbi/jdbi/pull/554

I uploaded jdbi 2.78-SNAPSHOT with this patch.

Could you test?

[Michael Seele]

Hi Steve,

sure. Can you please provide me a binary jar including these patch (like Matthew did)?

Thanks,

Michael

> >>>>>>> The web applic...

[Steven Schlansker]

I uploaded it to Sonatype, I think this should work:
https://oss.sonatype.org/service/local/repositories/snapshots/content/org/jdbi/jdbi/2.78-SNAPSHOT/jdbi-2.78-20161109.183121-1.jar

[Michael Seele]

Hi Steve,

i've tested it and there are way less warnings now. But there are still a few:

14-Nov-2016 09:57:06.668 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161114-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@13dd8bc1]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@24ccf549]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

14-Nov-2016 09:57:06.669 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161114-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@4c06feb5]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@62ae0595]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

14-Nov-2016 09:57:06.669 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161114-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@1d520982]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@5e77d1dc]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

14-Nov-2016 09:57:06.669 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161114-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@257f7ff1]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@37ef4466]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

14-Nov-2016 09:57:06.670 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161114-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@3a7c5500]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@5a659aa2]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Here's a new dump for these warnings (tgz again): https://drive.google.com/file/d/0B6e86GGLWDffZDcxRjl3Y0Zhajg/view?usp=sharing

I've also investigated in my code and maybe found a problem in my code, too. This is how i create/store the IDBI References:

/**

 * Access point to get a database connection.

 * 

 * @author mseele

 */

public class DB {

private static final Logger LOGGER = LogManager.getLogger(DB.class.getName());

// we use ThreadLocal intentionally to avoid the creation of calendar 

// instances (expensive) for every date operation (e.g. fetch rows from db)

private static final ThreadLocal<Calendar> UTC = new ThreadLocal<Calendar>() {

@Override

protected Calendar initialValue() {

return Calendar.getInstance(TimeZone.getTimeZone("UTC")); //$NON-NLS-1$

}

};

/**

* @return a thread-safe instance of calendar in UTC time zone

*/

public static final Calendar UTC() {

return UTC.get();

}

private static Map<String, DBI> DBI = new HashMap<>();

/**

* Returns a database access point for the dmm database

* 

* @return a database access point for the dmm database

*/

public static final IDBI dmm() {

try {

return dbi(ContextValues.get(ContextValues.DB_USER));

} catch (InvalidConfigurationException e) {

LOGGER.error("Could not load configuration value: " + ContextValues.DB_USER, e); //$NON-NLS-1$

}

return null;

}

/**

* Returns a database access point for the edm database

* 

* @return a database access point for the edm database

*/

public static final IDBI edm() {

try {

return dbi(ContextValues.get(ContextValues.DB_EDM_USER));

} catch (InvalidConfigurationException e) {

LOGGER.error("Could not load configuration value: " + ContextValues.DB_USER, e); //$NON-NLS-1$

}

return null;

}

private static final synchronized IDBI dbi(String username) {

DBI dbi = DBI.get(username);

if (dbi == null) {

try {

String sid = ContextValues.get(ContextValues.DB_SID);

final CachingNetrc cachingNetrc = new CachingNetrc();

String hostname = Network.getLocalHostname();

String login = username + "@" + sid; //$NON-NLS-1$

final String password = cachingNetrc.getPassword(hostname, login);

if (password == null) {

throw new InvalidConfigurationException("No password found for host " + hostname + " and user " + login + "."); //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$

}

final DataSource dataSource = ContextValues.get(ContextValues.DB_DATA_SOURCE);

dbi = new DBI(new ConnectionFactory() {

@Override

public Connection openConnection() throws SQLException {

return dataSource.getConnection(username, password);

}

});

dbi.setSQLLog(new Log4J2Log());

dbi.registerArgumentFactory(new TimestampArgumentFactory());

dbi.registerColumnMapper(new TimestampColumnMapper());

DBI.put(username, dbi);

} catch (InvalidConfigurationException | IOException | MissingSystemPropertyException e) {

LOGGER.error("Error while initializing connection to database", e); //$NON-NLS-1$

}

}

return dbi;

}

private static class TimestampColumnMapper implements ResultColumnMapper<Timestamp> {

public TimestampColumnMapper() {

super();

}

@Override

public Timestamp mapColumn(ResultSet r, int columnNumber, StatementContext ctx) throws SQLException {

return r.getTimestamp(columnNumber, DB.UTC());

}

@Override

public Timestamp mapColumn(ResultSet r, String columnLabel, StatementContext ctx) throws SQLException {

return r.getTimestamp(columnLabel, DB.UTC());

}

}

private static class TimestampArgumentFactory implements ArgumentFactory<Timestamp> {

public TimestampArgumentFactory() {

super();

}

@Override

public boolean accepts(Class<?> expectedType, Object value, StatementContext ctx) {

return value instanceof Timestamp;

}

@Override

public Argument build(Class<?> expectedType, Timestamp value, StatementContext ctx) {

return new Argument() {

@Override

public void apply(int position, PreparedStatement statement, StatementContext context) throws SQLException {

if (value != null) {

statement.setTimestamp(position, value, DB.UTC());

} else {

statement.setNull(position, Types.TIMESTAMP);

}

}

@Override

public String toString() {

return String.valueOf(value);

}

};

}

}

private static class Log4J2Log extends FormattedLog {

private final Logger fLogger;

private Level fLevel;

public Log4J2Log() {

this.fLogger = LogManager.getLogger(DBI.class.getPackage().getName());

this.fLevel = Level.DEBUG;

}

@Override

protected boolean isEnabled() {

return this.fLogger.isEnabled(this.fLevel);

}

@Override

protected void log(String msg) {

this.fLogger.log(this.fLevel, msg);

}

@Override

public void logObtainHandle(long time, Handle h) {

// do not log

}

@Override

public void logReleaseHandle(Handle h) {

// do not log

}

}

}

I think i have to replace the DBI map with a WeakHashMap:

private static Map<String, DBI> DBI = new WeakHashMap<>();

Thanks,

Michael

> > >>>>>>> the web application was stopped. T...

[Steven Schlansker]

Okay, more progress :)

Indeed, your code has problems too. The idea of a WeakHashMap is a good start but I don't think it is
the best solution in this case -- web applications have a well defined lifecycle and you should tie
your DBI's scope to it.

For example, since you are reloading your web application, you could make your DB class
not use static fields and instead put it in the web application context. This means
when the application server closes the application context, the DB instance itself
will be GCed, no need to play around with WeakHashMap.

I am a little confused by the dump you gave me. Looking through the GC roots,
it looks like one is through SqlObject.factories, which in your dump is a ConcurrentHashMap
but was changed to a WeakHashMap about two weeks ago.

Are you sure this dump was produced against the latest patched jdbi library? And that the
application server had a fresh start with the patched jdbi?

Please also make sure you aren't leaking Handle instances; the other easy explanation for
the previous fix not being effective is that Handle.close() wasn't called for every Handle object.
Even missing a single one will cause this leak.

Just in case, can you retry with a rebuilt jar and see if you can still reproduce the problem?

I uploaded a fresh copy here:
https://oss.sonatype.org/content/repositories/snapshots/org/jdbi/jdbi/2.78-SNAPSHOT/jdbi-2.78-20161114.234245-2.jar

[Michael Seele]

Hi Steve,

Indeed, your code has problems too.  The idea of a WeakHashMap is a good start but I don't think it is
the best solution in this case -- web applications have a well defined lifecycle and you should tie
your DBI's scope to it.

For example, since you are reloading your web application, you could make your DB class
not use static fields and instead put it in the web application context.  This means
when the application server closes the application context, the DB instance itself
will be GCed, no need to play around with WeakHashMap.

Thank you for your input. Appreciate.

 

I am a little confused by the dump you gave me.  Looking through the GC roots,
it looks like one is through SqlObject.factories, which in your dump is a ConcurrentHashMap
but was changed to a WeakHashMap about two weeks ago.

Are you sure this dump was produced against the latest patched jdbi library? And that the
application server had a fresh start with the patched jdbi?

Yes i'm sure. This is how i simulate/create that dump (this time, too):

1) deploy the war with your jar (e.g. dmm##20161115-01.war)

2) go into the browser and play around with the application (to have database calls)

3) deploy the same war with a higher version parallel (e.g. dmm##20161115-02.war)

4) wait until the first war is undeployed

5) check the warnings in catalina.out

5) create the dump (e.g.  jmap -dump:file=dump.dat 2140)

 

Please also make sure you aren't leaking Handle instances; the other easy explanation for
the previous fix not being effective is that Handle.close() wasn't called for every Handle object.
Even missing a single one will cause this leak.

 

I've checked my code and every IDBI.open is in a try-finally clause.

Just in case, can you retry with a rebuilt jar and see if you can still reproduce the problem?

I uploaded a fresh copy here:
https://oss.sonatype.org/content/repositories/snapshots/org/jdbi/jdbi/2.78-SNAPSHOT/jdbi-2.78-20161114.234245-2.jar

Here are the warnings with this jar:

15-Nov-2016 06:51:24.956 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161115-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@117ebe2e]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@4ae2f63f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

15-Nov-2016 06:51:24.957 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161115-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@5fdcf0a8]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@4f81b75a]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

15-Nov-2016 06:51:24.958 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161115-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@4e0540d2]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@cfc052f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

15-Nov-2016 06:51:24.958 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161115-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@7010ab45]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@239f7e46]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

15-Nov-2016 06:51:24.958 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161115-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@4778b937]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@2b41198a]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

15-Nov-2016 06:51:24.958 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161115-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@5e7ec524]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@64bfde3]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

15-Nov-2016 06:51:24.959 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161115-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@4785d1c]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@27569950]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

15-Nov-2016 06:51:24.959 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161115-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@f293f7b]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@56546f98]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

15-Nov-2016 06:51:24.959 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161115-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] (value [org.skife.jdbi.v2.BasicHandle$1@100248f3]) and a value of type [org.skife.jdbi.v2.SqlObjectContext] (value [org.skife.jdbi.v2.SqlObjectContext@38ddfba]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

 This is the dump (tgz as always): https://drive.google.com/open?id=0B6e86GGLWDffamxnbENiWGxqczA

 

> > > 28-Oct-2016 06:44:31.857 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.ap...

[Steven Schlansker]

Ah, another idea! Are any of the Handle instances shared across threads at all? Currently the code assumes that the Handle is opened and then closed in the same thread. Any other uses could cause a leak like this.

[Michael Seele]

No, they are used always single threaded.

> You received this message because you are subscribed to a topic in the Google Groups "jDBI" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/jdbi/A-PJfuNswpM/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to jdbi+uns...@googlegroups.com.

[Steven Schlansker]

How big of a problem is this for you? Since Tomcat recycles the threads, these ThreadLocals
eventually do get GCed, and the only problem is the warning - yeah?

I'm considering trying to write a more thorough test for this and assert that jdbi
is safe to use in the presence of class unloading, but that'd target jdbi3 only.

Fixing this piece by piece is brittle and doing these remote diagnoses are time consuming!

I filed a followup issue: https://github.com/jdbi/jdbi/issues/571

[Michael Seele]

No, its not a big problem for us. I just want to help you improving jdbi...

Good idea to target this to jdbi3

>>>>>> 02-Nov-2016 12:32:47.453 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [dmm##20161102-01] created a ThreadLocal with key of type [org.skife.jdbi.v2.BasicHandle$1] ...

[Steven Schlansker]

Thank you for being persistent and caring to help us improve it :)

I'll try to spend more time on this as I can, and hopefully in the
reasonably near future we will be totally safe to unload in webapps.