Fedora Commons 4 Error: (JcrRepository) Error during background garbage collection: Java heap space

[Anderson, Steven]

Hiya,

I have a Fedora 4 Commons instance up and it seems to be getting a "java.lang.OutOfMemoryError: Java heap space" error over time that causes it to crash. I'm running it under tomcat7 with a Ubuntu 14.04 box. I have three stack traces to share if anyone has any ideas. The first is the potential cause of the problem that is in the subject of this email. The other two are ones I saw during a time before Fedora 4 Commons crashed. Any thoughts on this would be appreciated (for example, -Xmx is set to 1024M but that may just need to be significantly higher). The exact version of Fedora 4 Commons being run is: fcrepo-webapp-plus-webac-audit-4.4.0.war. Thanks!

Error 1:
---------

ERROR 20:14:36.596 (JcrRepository) Error during background garbage collection: Java heap space

java.lang.OutOfMemoryError: Java heap space

java.lang.NullPointerException

at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3102)

at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2879)

at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1208)

at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1688)

at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1569)

at ch.qos.logback.classic.spi.PackagingDataCalculator.loadClass(PackagingDataCalculator.java:207)

at ch.qos.logback.classic.spi.PackagingDataCalculator.bestEffortLoadClass(PackagingDataCalculator.java:226)

at ch.qos.logback.classic.spi.PackagingDataCalculator.computeBySTEP(PackagingDataCalculator.java:138)

at ch.qos.logback.classic.spi.PackagingDataCalculator.populateUncommonFrames(PackagingDataCalculator.java:113)

at ch.qos.logback.classic.spi.PackagingDataCalculator.populateFrames(PackagingDataCalculator.java:105)

at ch.qos.logback.classic.spi.PackagingDataCalculator.calculate(PackagingDataCalculator.java:57)

at ch.qos.logback.classic.spi.ThrowableProxy.calculatePackagingData(ThrowableProxy.java:147)

at ch.qos.logback.classic.spi.LoggingEvent.<init>(LoggingEvent.java:124)

at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:440)

at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:396)

at ch.qos.logback.classic.Logger.error(Logger.java:559)



Error 2:
-----------

rg.infinispan.commons.CacheException: java.lang.OutOfMemoryError: Java heap space at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:341) at org.infinispan.CacheImpl.get(CacheImpl.java:377) at org.infinispan.CacheImpl.get(CacheImpl.java:369) at org.infinispan.schematic.internal.CacheSchematicDb.get(CacheSchematicDb.java:72) at org.modeshape.jcr.cache.document.LocalDocumentStore.get(LocalDocumentStore.java:71) at org.modeshape.jcr.cache.document.WorkspaceCache.documentFor(WorkspaceCache.java:180) at org.modeshape.jcr.cache.document.WorkspaceCache.documentFor(WorkspaceCache.java:199) at org.modeshape.jcr.cache.document.WorkspaceCache.getNode(WorkspaceCache.java:231) at org.modeshape.jcr.cache.document.AbstractSessionCache.getNode(AbstractSessionCache.java:240) at org.modeshape.jcr.cache.document.WritableSessionCache.getNode(WritableSessionCache.java:161) at org.modeshape.jcr.JcrSession.node(JcrSession.java:498) at org.modeshape.jcr.AbstractJcrNode$ChildNodeResolver.nodeFrom(AbstractJcrNode.java:3578) at org.modeshape.jcr.JcrChildNodeIterator.hasNext(JcrChildNodeIterator.java:104) at com.google.common.collect.Iterators$7.computeNext(Iterators.java:650) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:43) at com.google.common.collect.Iterators$5.hasNext(Iterators.java:547) at com.google.common.collect.Iterators$5.hasNext(Iterators.java:547) at com.google.common.collect.Iterators$5.hasNext(Iterators.java:547) at com.google.common.collect.Iterators$5.hasNext(Iterators.java:547) at com.google.common.collect.Iterators$5.hasNext(Iterators.java:547) at com.google.common.collect.Iterators$5.hasNext(Iterators.java:547) at com.google.common.collect.Iterators$5.hasNext(Iterators.java:547) at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:43) at com.google.common.collect.Iterators$5.hasNext(Iterators.java:547) at com.google.common.collect.ForwardingIterator.hasNext(ForwardingIterator.java:43) at com.google.common.collect.Iterators$5.hasNext(Iterators.java:547) at com.google.common.collect.ForwardingIterator.hasNext(ForwardingIterator.java:43) at com.google.common.collect.Iterators$5.hasNext(Iterators.java:547) at com.google.common.collect.Iterators$5.hasNext(Iterators.java:547) at com.google.common.collect.Iterators$5.hasNext(Iterators.java:547) at com.google.common.collect.Iterators$5.hasNext(Iterators.java:547) at com.google.common.collect.Iterators$5.hasNext(Iterators.java:547) at com.google.common.collect.Iterators$5.hasNext(Iterators.java:547) at com.google.common.collect.Iterators$5.hasNext(Iterators.java:547) at com.google.common.collect.ForwardingIterator.hasNext(ForwardingIterator.java:43) at com.google.common.collect.Iterators$5.hasNext(Iterators.java:547) at org.fcrepo.kernel.api.utils.iterators.RdfStream.asModel(RdfStream.java:303) at org.fcrepo.http.api.responses.StreamingBaseHtmlProvider.writeTo(StreamingBaseHtmlProvider.java:203) at org.fcrepo.http.api.responses.StreamingBaseHtmlProvider.writeTo(StreamingBaseHtmlProvider.java:85) at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265) at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250) at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106) at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85) at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1154) at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:621) at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:377) at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:367) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) at org.glassfish.jersey.internal.Errors.process(Errors.java:315) at org.glassfish.jersey.internal.Errors.process(Errors.java:297) at org.glassfish.jersey.internal.Errors.process(Errors.java:267) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:373) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:381) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:344) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:221) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:610) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.OutOfMemoryError: Java heap space at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:300) at java.lang.StringCoding.encode(StringCoding.java:344) at java.lang.String.getBytes(String.java:918) at java.io.UnixFileSystem.getLastModifiedTime(Native Method) at java.io.File.lastModified(File.java:943) at java.util.zip.ZipFile.(ZipFile.java:219) at java.util.zip.ZipFile.(ZipFile.java:149) at java.util.jar.JarFile.(JarFile.java:166) at java.util.jar.JarFile.(JarFile.java:130) at org.apache.catalina.loader.WebappClassLoader.openJARs(WebappClassLoader.java:2844) at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3097) at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2879) at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1208) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1688) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1569) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:126) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73) at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74) at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333) at org.infinispan.CacheImpl.get(CacheImpl.java:377) at org.infinispan.CacheImpl.get(CacheImpl.java:369) at org.infinispan.schematic.internal.CacheSchematicDb.get(CacheSchematicDb.java:72) at org.modeshape.jcr.cache.document.LocalDocumentStore.get(LocalDocumentStore.java:71) at org.modeshape.jcr.cache.document.WorkspaceCache.documentFor(WorkspaceCache.java:180) at org.modeshape.jcr.cache.document.WorkspaceCache.documentFor(WorkspaceCache.java:199) at org.modeshape.jcr.cache.document.WorkspaceCache.getNode(WorkspaceCache.java:231) at org.modeshape.jcr.cache.document.AbstractSessionCache.getNode(AbstractSessionCache.java:240) at org.modeshape.jcr.cache.document.WritableSessionCache.getNode(WritableSessionCache.java:161) at org.modeshape.jcr.JcrSession.node(JcrSession.java:498) at org.modeshape.jcr.AbstractJcrNode$ChildNodeResolver.nodeFrom(AbstractJcrNode.java:3578) at org.modeshape.jcr.JcrChildNodeIterator.hasNext(JcrChildNodeIterator.java:104)



Error 3 (only had saved the top line of this one):
--------------------------------------------------------------------

org.infinispan.commons.CacheException: java.lang.OutOfMemoryError: Java heap space Caused by: java.lang.OutOfMemoryError: Java heap space



Sincerely,
Steven Anderson
Web Services - Digital Library Repository Developer
617-859-2393
sand...@bpl.org

[Andrew Woods]

Hello Steven,

The comments in this ticket may be of interest:

https://jira.duraspace.org/browse/FCREPO-1294

Since every usage scenario is slightly different, defining a single set of JAVA_OPTS that is ideal for everyone is tricky. Please report back on what (if anything) you find to work, and we will pull that into the project administration documentation.

Thanks,

Andrew

--
You received this message because you are subscribed to the Google Groups "Fedora Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fedora-communi...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Stuart Kenny]

Hi, just to say we are having a similar problem as that described in the ticket. I had already increased the RAM on the machine to 8GB and set -Xmx6144m but I'm still seeing the OS killing the java process. Its possible that running on a VM is not helping so we may have to try and migrate to a physical server. Is there a reason for the heap space usage to just gradually keep increasing like this? Maybe people that are not seeing this behaviour have different modeshape/infinispan configurations that they could share?

Thanks,

Stuart.

[Stefano Cossu]

Stuart, Steven,

For what is worth, I managed to get rid of OOME on my staging server with 8Gb of RAM using these settings:
 -Xms512m -Xmx4g -XX:NewSize=256m -XX:MaxNewSize=1g -XX:MetaspaceSize=64m -XX:MaxMetaspaceSize=2g -XX:+DisableExplicitGC

On my production server, with 32Gb of RAM, I have:
-Xms512m -Xmx6144m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:MetaspaceSize=64m -XX:MaxMetaspaceSize=2048m -XX:+DisableExplicitGC

I am not suggesting that these numbers are best for you (or even for me) and I have only tested with large numbers of small resources so far.

Also, one of the documents [1] that Andrew mentions in the related JIRA ticket [1] points out an interesting detail:

"When allocating memory to a JVM, more memory does not always equate to a better experience. An increase in heap size will lead to longer GC times, which can make instances appear frozen when a GC occurs, or can also lead to this error."

Hope this helps,

Stefano

[1] https://confluence.atlassian.com/display/CONFKB/Confluence+crashes+due+to+java.lang.OutOfMemoryError+GC+overhead+limit+exceeded
[2] https://jira.duraspace.org/browse/FCREPO-1294

--

Stefano Cossu
Director of Application Services, Collections

The Art Institute of Chicago
116 S. Michigan Ave.
Chicago, IL 60603
312-499-4026

[Steven DiDomenico]

There could certainly be an issue in the Fedora/Modeshape/Infinispan code. Though I’m curious which Java virtual machine you’re using, and if switching to a different one might give different results. 

I should note we’re running all of our projects on Oracle Java with generally good results (versus OpenJDK which comes with Ubuntu and Red Hat by default) but we haven’t had an opportunity to fully test Fedora 4 yet.

Steve

--------

Steve DiDomenico — Repository and Digital Curation

Northwestern University Library — Digital Strategies
st...@northwestern.edu  847-491-4230  Evanston, IL

[Stuart Kenny]

Hi, modifying the JAVA_OPTS does seem to have helped. Adding -XX:MetaspaceSize and -XX:MaxMetaspaceSize was the only real change over what I was already using. With this a user was able to ingest 14,000 objects without errors.

We are using OpenJDK by the way.

Thanks,

Stuart.