Memory leak

André Quintino Kuhnen

unread,

Sep 12, 2014, 2:54:33 PM9/12/14

to kamon...@googlegroups.com

Hello.

I think i found a memory leak on kamon metrics.

I am using

akka 2.3.2

spray 1.3.1

scala 2.11.2

kamon 0.3.4

The applicatin heap begins to increase since the application is started, usually it takes around 80 hours to run of GC overhead limit exceed.

I decided to take a heap_dump and analyze it with Eclipse MemoryAnalyzer

So I just post some result and I hope i will be able to help the kamon community :)

Objects on heap (akka.dispatch.UnboundedMailbox$MessageQueue)

Class Name | Objects | Shallow Heap

-----------------------------------------------------------------------------------

| |

kamon.metric.instrument.CompactHdrSnapshot | 9,939,709 | 477,106,032

scala.collection.immutable.HashMap$HashMap1 | 9,939,709 | 318,070,688

kamon.metric.TraceMetrics$TraceMetricsSnapshot | 9,939,709 | 238,553,016

scala.Tuple2 | 9,939,709 | 238,553,016

long[] | 9,939,709 | 165,750,496

scala.collection.immutable.HashMap[] | 3,214,080 | 109,014,808

scala.collection.immutable.HashMap$HashTrieMap | 3,214,080 | 77,137,920

akka.dispatch.Envelope | 3,416 | 109,312

kamon.metric.Subscriptions$TickMetricSnapshot | 3,348 | 107,136

java.util.concurrent.ConcurrentLinkedQueue$Node | 3,417 | 82,008

kamon.trace.TraceContextAware$DefaultTraceContextAware| 3,416 | 81,984

akka.dispatch.Dispatcher$$anon$1 | 1 | 32

akka.dispatch.UnboundedMailbox$MessageQueue | 1 | 24

Total: 13 entries | 56,140,304 | 1,624,566,472

-----------------------------------------------------------------------------------

As you can see kamonMetric.instrumnet.CompactHdrSnapshot is the villain

the HashMap is part ot the CompactHdrSnaphost as the next data shows:

Class Name | Objects | Shallow Heap

---------------------------------------------------------------------------------------------

| |

kamon.metric.instrument.CompactHdrSnapshot | 9,946,670 | 477,440,160

'- kamon.metric.TraceMetrics$TraceMetricsSnapshot | 9,946,670 | 238,720,080

|- scala.collection.immutable.HashMap$HashMap1 | 9,950,819 | 318,426,208

| '- scala.collection.immutable.HashMap[] | 3,167,430 | 107,528,280

| |- scala.collection.immutable.HashMap$HashTrieMap | 3,167,430 | 76,018,320

| |- scala.collection.immutable.Iterable[][] | 1 | 40

| |- scala.collection.immutable.HashMap$HashTrieMap$$anon$1| 1 | 40

| '- Total: 3 entries | |

|- scala.Tuple2 | 9,943,708 | 238,648,992

'- Total: 2 entries | |

---------------------------------------------------------------------------------------------

And here is the histogram of the HEAP:

Class Name | Objects | Shallow Heap | Retained Heap

------------------------------------------------------------------------------------------------

| | |

kamon.metric.instrument.CompactHdrSnapshot | 9,946,670 | 477,440,160 | >= 643,320,320

scala.collection.immutable.HashMap$HashMap1 | 9,951,683 | 318,453,856 | >= 1,438,869,712

long[] | 9,949,843 | 276,317,376 | >= 276,317,376

kamon.metric.TraceMetrics$TraceMetricsSnapshot | 9,946,670 | 238,720,080 | >= 882,040,400

scala.Tuple2 | 9,944,828 | 238,675,872 | >= 238,677,216

scala.collection.immutable.HashMap[] | 3,217,918 | 109,145,064 | >= 1,625,164,912

scala.collection.immutable.HashMap$HashTrieMap | 3,217,917 | 77,230,008 | >= 1,625,245,512

java.lang.Long | 602,554 | 14,461,296 | >= 14,461,328

char[] | 122,145 | 13,524,344 | >= 13,524,344

java.util.concurrent.ConcurrentSkipListMap$Node | 559,098 | 13,418,352 | >= 40,207,272

java.lang.Double | 558,116 | 13,394,784 | >= 13,394,856

byte[] | 13,368 | 13,213,560 | >= 13,213,560

java.util.concurrent.ConcurrentSkipListMap$Index| 279,543 | 6,709,032 | >= 6,709,048

scala.concurrent.forkjoin.ForkJoinTask[] | 170 | 5,573,280 | >= 5,575,272

java.util.TreeMap$Entry | 74,375 | 2,975,000 | >= 7,775,264

java.lang.String | 120,616 | 2,894,784 | >= 14,453,488

java.lang.Object[] | 21,402 | 1,505,824 | >= 17,353,792

java.util.TreeMap | 22,152 | 1,063,296 | >= 7,846,832

java.util.concurrent.ConcurrentHashMap$HashEntry| 33,169 | 1,061,408 | >= 2,305,120

java.util.HashMap$Entry[] | 9,221 | 863,552 | >= 4,636,896

java.util.HashMap$Entry | 18,893 | 604,576 | >= 3,916,792

org.aspectj.weaver.bcel.BcelMethod | 3,933 | 597,816 | >= 1,493,824

java.util.LinkedHashMap$Entry | 13,889 | 555,560 | >= 6,466,824

org.aspectj.apache.bcel.classfile.ConstantUtf8 | 22,945 | 550,680 | >= 2,334,264

javax.management.openmbean.CompositeDataSupport | 22,000 | 528,000 | >= 7,573,880

Total: 25 of 15,394 entries; 15,369 more | 59,441,010 | 1,855,168,448 |

------------------------------------------------------------------------------------------------

If you need any more information please ,let me know .

By the way Kamon is awesome :)

André Kuhnen

Chaordic | Developer

www.chaordic.com.br

55 48 3232.3200

55 48 9659 0725

Av. Afonso Delambert Neto, 26

Florianópolis, SC • 88062-000

Ivan Topolnjak

unread,

Sep 12, 2014, 2:57:06 PM9/12/14

to kamon...@googlegroups.com

Hello Andre!

Quick question: are you using the traceName directive to properly assign names to all of your requests?

--
You received this message because you are subscribed to the Google Groups "kamon-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kamon-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

André Quintino Kuhnen

unread,

Sep 13, 2014, 11:12:57 AM9/13/14

to kamon...@googlegroups.com

Thanks for you quick Answer Ivan.... I will review the code.. and if the problem persists I will write back ok?

Ivan Topolnjak

unread,

Sep 15, 2014, 1:07:33 AM9/15/14

to kamon...@googlegroups.com

Nice! Have a look at this post too: https://groups.google.com/d/msg/kamon-user/wCpexpMm5ik/AXuajnKbCxoJ

Reply all

Reply to author

Forward