Memory leaks when stopping tomcat

[denny gee]

Hi Team,

I am currently using titan 0.9.0-M2 and building the graph data based on cassandra & elasticsearch.

storage.backend=cassandra

storage.hostname=xxx.xxx....

storage.cassandra.replication-factor=2

index.search.backend=elasticsearch

index.search.hostname=xxx.xxx.xxx.xxx

schema.default=none

I have actually successfully implemented unit test, where I could insert vertices and edges, and querying the graph without problems.

For the unit test, I am using TitanGraph, and on @AfterTest, I invoked the graph.close() to shutdown the graph instance, which takes roughly about 1-2 seconds to complete.

Everything works fine so far, except when running this on Tomcat webapp.

On tomcat, I have a context aware method attached, so on shutdown, we would cleanup resources. This means ideally invoking the graph.close() method when tomcat shutdowns. 

In which I get tons of errors on memory leaks. A copy of the error is below in this message.

I tried to google around, and found some discussion on Kryo that describes a similar problem.

I set the property "graph.kryo-instance-cache" for TitanGraph to concurrent_hash_map, but on shutdown, the same memory leaks happened.

I had also tried to close the graph instance before shutting down tomcat, just to check the method is working. But on triggering tomcat shutdown, the same memory leaks still occurs.

Questions:

Is this an issue already known, and if so, could you help to point it to me so I could read up?

Regarding this, does Titan API has a way I can access the cassandra & elasticsearch client so I could maybe workout something on the threads?

Thanks for your time, and if any details are needed, please let me know.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [Reaper-0] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [New I/O worker #1] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [New I/O worker #2] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [New I/O worker #3] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [New I/O worker #4] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [New I/O worker #5] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [New I/O worker #6] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [New I/O worker #7] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [New I/O worker #8] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [pool-6-thread-1] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [Hashed wheel timer #1] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [pool-13-thread-1] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [pool-13-thread-2] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [RingDescribeAutoDiscovery] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [elasticsearch[Boobytrap][[timer]]] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [elasticsearch[Boobytrap][scheduler][T#1]] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [elasticsearch[Boobytrap][transport_client_worker][T#1]{New I/O worker #1}] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [elasticsearch[Boobytrap][transport_client_worker][T#2]{New I/O worker #2}] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [elasticsearch[Boobytrap][transport_client_worker][T#3]{New I/O worker #3}] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [elasticsearch[Boobytrap][transport_client_worker][T#4]{New I/O worker #4}] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [elasticsearch[Boobytrap][transport_client_worker][T#5]{New I/O worker #5}] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [elasticsearch[Boobytrap][transport_client_worker][T#6]{New I/O worker #6}] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [elasticsearch[Boobytrap][transport_client_worker][T#7]{New I/O worker #7}] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [elasticsearch[Boobytrap][transport_client_worker][T#8]{New I/O worker #8}] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [elasticsearch[Boobytrap][transport_client_boss][T#1]{New I/O boss #9}] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [elasticsearch[Boobytrap][management][T#1]] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [elasticsearch[Boobytrap][transport_client_timer][T#1]{Hashed wheel timer #1}] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [elasticsearch[Boobytrap][generic][T#1]] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [pool-28-thread-1] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [Thread-7] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [elasticsearch[Boobytrap][management][T#2]] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/testwebapp] appears to have started a thread named [elasticsearch[Boobytrap][management][T#3]] but has failed to stop it. This is very likely to create a memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks

SEVERE: The web application [/testwebapp] created a ThreadLocal with key of type [org.apache.commons.lang.builder.HashCodeBuilder$1] (value [org.apache.commons.lang.builder.HashCodeBuilder$1@5d8c11a1]) and a value of type [java.util.HashSet] (value [[]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks

SEVERE: The web application [/testwebapp] created a ThreadLocal with key of type [org.apache.cassandra.utils.FBUtilities$1] (value [org.apache.cassandra.utils.FBUtilities$1@2a8796a3]) and a value of type [java.security.MessageDigest.Delegate] (value [MD5 Message Digest from SUN, <initialized>

]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks

SEVERE: The web application [/testwebapp] created a ThreadLocal with key of type [org.elasticsearch.common.inject.InjectorImpl$1] (value [org.elasticsearch.common.inject.InjectorImpl$1@51233cee]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;@ed6903e]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks

SEVERE: The web application [/testwebapp] created a ThreadLocal with key of type [org.elasticsearch.common.util.concurrent.jsr166e.Striped64.ThreadHashCode] (value [org.elasticsearch.common.util.concurrent.jsr166e.Striped64$ThreadHashCode@336a895c]) and a value of type [org.elasticsearch.common.util.concurrent.jsr166e.Striped64.HashCode] (value [org.elasticsearch.common.util.concurrent.jsr166e.Striped64$HashCode@25ce231c]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks

SEVERE: The web application [/testwebapp] created a ThreadLocal with key of type [org.elasticsearch.common.util.concurrent.jsr166e.Striped64.ThreadHashCode] (value [org.elasticsearch.common.util.concurrent.jsr166e.Striped64$ThreadHashCode@336a895c]) and a value of type [org.elasticsearch.common.util.concurrent.jsr166e.Striped64.HashCode] (value [org.elasticsearch.common.util.concurrent.jsr166e.Striped64$HashCode@6dc2af07]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Sep 25, 2015 6:41:46 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks

SEVERE: The web application [/testwebapp] created a ThreadLocal with key of type [org.elasticsearch.common.util.concurrent.jsr166e.Striped64.ThreadHashCode] (value [org.elasticsearch.common.util.concurrent.jsr166e.Striped64$ThreadHashCode@336a895c]) and a value of type [org.elasticsearch.common.util.concurrent.jsr166e.Striped64.HashCode] (value [org.elasticsearch.common.util.concurrent.jsr166e.Striped64$HashCode@572f85e1]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

[Matthias Broecheler]

Titan's graph.close() method only cleans up connections and internally started clients. I am not very familiar with Tomcat - maybe you could take a look at Titan's unit tests to see how we do the initiation and clean up there?

--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/4b08dc4b-affb-4b47-b6fe-bfcbdafdb899%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.