metrics-meter-tick-thread-1,2 memory leak exception

[manish kumar]

I am running spring-mvc application. When i was closing Tomcat server, it shows

SEVERE: The web application [/myapp] appears to have started a thread named [metrics-meter-tick-thread-1] but has failed to stop it. This is very likely to create a memory leak.
SEVERE: The web application [/myapp] appears to have started a thread named [metrics-meter-tick-thread-2] but has failed to stop it. This is very likely to create a memory leak.

when i explored jars dependency i saw that there are two metrics-core jars:

metrics-core:2.2.0 (used by `datastax`)
metrics-core:3.0.1 (used by `Titan`)

For more details pls have look

http://stackoverflow.com/questions/23801992/memory-leak-metrics-meter-tick-thread-and-new-i-o-client-worker

I am using jdk1.7,  titan-0.4.4, cassandra-1.2.2 ,tomcat-7.0.34

[Dan LaRocque]

Hi,

Those Metrics versions have different package names and work mostly
independently. Metrics 2 is used by Cassandra (and HBase), while Titan
uses Metrics 3.

Can you paste your Titan/Rexster configuration? I want to see whether
you're using embeddedcassandra.

Titan creates Metrics 3 reporters (when configured to do so) during
startup, but it doesn't remove/kill them during shutdown. That's a
resource leak, albeit one that I don't encounter very often, and I'll
look at fixing it.

However, I don't think Titan's reporter leak is the cause of this
specific problem. The string you pasted appears in the source for
Metrics 2.2.0 but not in the source for 3.0.1.

thanks,
Dan

[manish kumar]

Here is my titan-cassandra-es.properties file content

storage.backend=cassandrathrift
storage.hostname=127.0.0.1
cache.db-cache = true
cache.db-cache-clean-wait = 20
cache.db-cache-time = 180000
cache.db-cache-size = 0.25

# storage.index.search.backend=elasticsearch
# storage.index.search.directory=../db/es
# storage.index.search.client-only=false
# storage.index.search.local-mode=true

storage.index.search.backend=elasticsearch
storage.index.search.hostname=127.0.0.1
storage.index.search.client-only=true

[manish kumar]

again I got the following warning while stopping tomcat

SEVERE: The web application [/myapp] created a ThreadLocal with key of type [com.thinkaurelius.titan.graphdb.database.serialize.kryo.KryoSerializer$2] (value [com.thinkaurelius.titan.graphdb.database.serialize.kryo.KryoSerializer$2@76a35a84]) and a value of type [com.esotericsoftware.kryo.Kryo] (value [com.esotericsoftware.kryo.Kryo@5e8bccf5]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.


Jun 03, 2014 3:38:07 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/myapp] created a ThreadLocal with key of type [com.thinkaurelius.titan.graphdb.blueprints.TitanBlueprintsGraph$1] (value [com.thinkaurelius.titan.graphdb.blueprints.TitanBlueprintsGraph$1@2c3d1951]) and a value of type [com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx] (value [standardtitantx[null]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.


Jun 03, 2014 3:38:07 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/myapp] created a ThreadLocal with key of type [com.thinkaurelius.titan.graphdb.database.serialize.kryo.KryoSerializer$2] (value [com.thinkaurelius.titan.graphdb.database.serialize.kryo.KryoSerializer$2@76a35a84]) and a value of type [com.esotericsoftware.kryo.Kryo] (value [com.esotericsoftware.kryo.Kryo@5f492b87]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.


Jun 03, 2014 3:38:07 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/myapp] created a ThreadLocal with key of type [com.thinkaurelius.titan.graphdb.blueprints.TitanBlueprintsGraph$1] (value [com.thinkaurelius.titan.graphdb.blueprints.TitanBlueprintsGraph$1@2c3d1951]) and a value of type [com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx] (value [standardtitantx[null]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

[Dan LaRocque]

Hi Manish,

These threadlocal leaks are useful but distinct from the
metrics-meter-tick-thread-* thread leak. Are you still seeing those, too?

thanks,
Dan

--
Dan LaRocque
http://www.thinkaurelius.com

[avismg222]

Hi Dan,

 Even i am seeing those error reported by Mr. Manish when i stop tomcat server.

Configuration given my Manish is same as mine. But using Titan-0.5.0

Aug 31, 2014 8:44:06 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks

SEVERE: The web application [/mywebapp] created a ThreadLocal with key of type [com.thinkaurelius.titan.graphdb.database.serialize.kryo.KryoSerializer$2] (value [com.thinkaurelius.titan.graphdb.database.serialize.kryo.KryoSerializer$2@6ce450d3]) and a value of type [com.esotericsoftware.kryo.Kryo] (value [com.esotericsoftware.kryo.Kryo@176847f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Aug 31, 2014 8:44:06 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks

SEVERE: The web application [/mywebapp] created a ThreadLocal with key of type [com.thinkaurelius.titan.graphdb.database.serialize.kryo.KryoSerializer$2] (value [com.thinkaurelius.titan.graphdb.database.serialize.kryo.KryoSerializer$2@bf4ae99]) and a value of type [com.esotericsoftware.kryo.Kryo] (value [com.esotericsoftware.kryo.Kryo@6cf2c4f0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Aug 31, 2014 8:44:06 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks

SEVERE: The web application [/mywebapp] created a ThreadLocal with key of type [com.thinkaurelius.titan.graphdb.database.serialize.kryo.KryoSerializer$2] (value [com.thinkaurelius.titan.graphdb.database.serialize.kryo.KryoSerializer$2@365fa071]) and a value of type [com.esotericsoftware.kryo.Kryo] (value [com.esotericsoftware.kryo.Kryo@401e769a]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Aug 31, 2014 8:44:06 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks

SEVERE: The web application [/mywebapp] created a ThreadLocal with key of type [com.thinkaurelius.titan.graphdb.database.serialize.kryo.KryoSerializer$2] (value [com.thinkaurelius.titan.graphdb.database.serialize.kryo.KryoSerializer$2@ffc9959]) and a value of type [com.esotericsoftware.kryo.Kryo] (value [com.esotericsoftware.kryo.Kryo@10b7aa42]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Aug 31, 2014 8:44:06 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks

SEVERE: The web application [/mywebapp] created a ThreadLocal with key of type [com.thinkaurelius.titan.graphdb.database.serialize.kryo.KryoSerializer$2] (value [com.thinkaurelius.titan.graphdb.database.serialize.kryo.KryoSerializer$2@a0adba1]) and a value of type [com.esotericsoftware.kryo.Kryo] (value [com.esotericsoftware.kryo.Kryo@4ed92c2d]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Aug 31, 2014 8:44:07 AM org.apache.catalina.startup.HostConfig deployWAR

Could you please help in resolve this.

Regards

Avinash

[C.C. Wechsler]

Dan,
Greetings.  We are running Titan 0.5.0 and HBase 0.98 with these settings:

# the following parameters are not reloaded automatically and require a manual bounce
storage.backend=hbase
storage.lock.wait-time=300
schema.default=none

cache.db-cache = true
cache.db-cache-clean-wait = 20
cache.db-cache-time = 180000

cache.db-cache-size = 0.5

We aren't changing any settings when we install a new war file.

We are using TomEE (based on Tomcat 7). 

We had some code of our own using Timers and we had to ensure that we were calling the cancel() method on our timers when our servlet context was destroyed.

We are also calling graph.shutdown() when we destroy the servlet context but we get these same exceptions referenced in this thread:

SEVERE: The web application [/aai] created a ThreadLocal with key of type [com.thinkaurelius.titan.graphdb.database.serialize.kryo.KryoSerializer$2] (value [com.thinkaurelius.titan.graphdb.database.serialize.kryo.KryoSerializer$2@1ffddba]) and a value of type [com.esotericsoftware.kryo.Kryo] (value [com.esotericsoftware.kryo.Kryo@1361ffc]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak

I didn't see anything in the 0.5.1 or 0.5.2 release notes indicating a fix.  Do you have any status on this?  Do you think it's in your code or is it possibly something we're doing?

Thanks.
CC

[Dan LaRocque]

Hi C.C.,

I think this is something Titan is doing wrong.  The ThreadLocal on com.thinkaurelius.titan.graphdb.database.serialize.kryo.KryoSerializer should probably be removed when Titan shuts down.

I created an issue to track and fix this: https://github.com/thinkaurelius/titan/issues/877

thanks,
Dan

--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/cd7109be-6fca-4d48-a79a-1783189a80ff%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.