Object creation, memory usage, and stuck threads

[Peter Matthew Eichman]

Hello all,

After we thought we had resolved our issues with hanging transactions and stuck threads on our dev server by increasing our ActiveMQ disk storage allocation [1], we have discovered that the issue has returned on our staging server.

This time, as far as we can tell, it is not the ActiveMQ resources getting used up that are causing the hanging (the disk space and memory usage percentages never went over 25% and 50%, respectively). We also increased the size of the ModeShape ring buffer from the default 1024 to 2048 [2]. Still, the system was hanging after anywhere from 100 to 300 transactions, where each transaction was creating (roughly) between 10-50 objects.

After enabling JMX remote monitoring on Tomcat, we had much more successful loads, and did not have any hanging transactions. However, things got progressively slower and slower over the 3 hour load. The slowdown happened in spurts, where any given request might take several seconds to complete, rather than less than a second as usual, which I believe corresponded to the JVM going through a GC cycle.

I'm attaching screenshots from the JMX monitoring of our CPU and memory usage. At a little ways into the load, the JVM started running at 100% CPU and spending about 20% of its time in garbage collection. And still the used memory in the heap kept growing.

My questions are:

1. How does this performance compare to expectations for Fedora?

2. What might be causing the gradual increase in memory that is unable to be freed by GC?

3. Has anyone else encountered stuck threads that mysteriously vanish once JMX is enabled?

Thanks,

-Peter

Configuration:

Oracle JDK 1.8u65

Tomcat 7.0.70

Fedora 4.7.0

JVM memory management params:

-XX:+UseConcMarkSweepGC

-XX:+CMSClassUnloadingEnabled

-XX:ConcGCThreads=5

-XX:MaxGCPauseMillis=200

-XX:ParallelGCThreads=20

-XX:MaxMetaspaceSize=512M

-Xms1024m

-Xmx2048m

[1] https://groups.google.com/d/msg/fedora-tech/0CletEhKO7s/sA5Wq-IeBQAJ

[2] https://modeshape.wordpress.com/2014/04/25/using-a-ring-buffer-for-events-in-4-0/

--

Peter Eichman
Senior Software Developer
University of Maryland Libraries

peic...@umd.edu

[Peter Matthew Eichman]

Update on my investigation into this issue:

Examination of the heap dumps shows a problem area might be the org.fcrepo.event.serializer.JSONLDSerializer class. Its ObjectMapper looks like it is creating but then never releasing Jackson JSON serializer and deserializer objects. 1 of each is being created for each event. Each of those is about 1MB in size, and by the time of the last heap dump there were 228,300 of each of them, with a total memory usage of 456MB.

See attached screenshot taken from the Eclipse Memory Analyzer of the final heap dump.

-Peter

[Peter Matthew Eichman]

Ticket created for this issue: https://jira.duraspace.org/browse/FCREPO-2540