severe error "[/fedora] appears to have started a thread ... but has failed to stop it"

[Ernie Gillis]

I have been getting a memory leak from fedora (see my catalina.out log below). I have already set my environment variable so JAVA_OPTS=-Xms1024m -Xmx1024m -XX:PermSize=32M -XX:MaxPermSize=128m 

I have a feeling it's linked to JMS, but I have it setup based on the documentaiont.

If anyone can give me assistance, it would be greatly appreciated! 

- catalina.out log -

SLF4J: Found binding in [jar:file:/usr/local/fedora/base/webapps/fedora/WEB-INF/lib/logback-classic-1.0.9.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]

Jun 19, 2014 7:12:49 PM org.apache.catalina.core.StandardContext start

SEVERE: Error listenerStart

Jun 19, 2014 7:12:49 PM org.apache.catalina.core.StandardContext start

SEVERE: Context [/fedora] startup failed due to previous errors

Jun 19, 2014 7:12:49 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/fedora] appears to have started a thread named [Thread-4] but has failed to stop it. This is very likely to create a memory leak.

Jun 19, 2014 7:12:49 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/fedora] appears to have started a thread named [JotmBatch] but has failed to stop it. This is very likely to create a memory leak.

Jun 19, 2014 7:12:49 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/fedora] appears to have started a thread named [JotmClock] but has failed to stop it. This is very likely to create a memory leak.

Jun 19, 2014 7:12:50 PM org.apache.catalina.startup.HostConfig deployDescriptor

INFO: Deploying configuration descriptor manager.xml

Jun 19, 2014 7:12:50 PM org.apache.catalina.startup.HostConfig deployDescriptor

INFO: Deploying configuration descriptor host-manager.xml

Jun 19, 2014 7:12:50 PM org.apache.catalina.startup.HostConfig deployWAR

INFO: Deploying web application archive fedoragsearch.war

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/local/fedora/base/lib/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/local/fedora/base/webapps/fedoragsearch/WEB-INF/lib/tika-app-1.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

Jun 19, 2014 7:12:54 PM org.apache.catalina.startup.HostConfig deployWAR

INFO: Deploying web application archive adore-djatoka.war

Jun 19, 2014 7:12:54 PM org.apache.catalina.startup.HostConfig deployDirectory

INFO: Deploying web application directory examples

Jun 19, 2014 7:12:54 PM org.apache.catalina.startup.HostConfig deployDirectory

INFO: Deploying web application directory docs

Jun 19, 2014 7:12:54 PM org.apache.catalina.startup.HostConfig deployDirectory

INFO: Deploying web application directory ROOT

Jun 19, 2014 7:12:54 PM org.apache.coyote.http11.Http11Protocol start

SEVERE: Error starting endpoint

java.net.BindException: Address already in use <null>:8080

        at org.apache.tomcat.util.net.JIoEndpoint.init(JIoEndpoint.java:549)

        at org.apache.tomcat.util.net.JIoEndpoint.start(JIoEndpoint.java:565)

        at org.apache.coyote.http11.Http11Protocol.start(Http11Protocol.java:203)

        at org.apache.catalina.connector.Connector.start(Connector.java:1122)

        at org.apache.catalina.core.StandardService.start(StandardService.java:540)

        at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)

        at org.apache.catalina.startup.Catalina.start(Catalina.java:595)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)

        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)

Caused by: java.net.BindException: Address already in use

        at java.net.PlainSocketImpl.socketBind(Native Method)

        at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)

        at java.net.ServerSocket.bind(ServerSocket.java:376)

        at java.net.ServerSocket.<init>(ServerSocket.java:237)

        at java.net.ServerSocket.<init>(ServerSocket.java:181)

        at org.apache.tomcat.util.net.DefaultServerSocketFactory.createSocket(DefaultServerSocketFactory.java:50)

        at org.apache.tomcat.util.net.JIoEndpoint.init(JIoEndpoint.java:538)

        ... 12 more

Jun 19, 2014 7:12:54 PM org.apache.catalina.core.StandardService start

SEVERE: Failed to start connector [Connector[HTTP/1.1-8080]]

LifecycleException:  service.getName(): "Catalina";  Protocol handler start failed: java.net.BindException: Address already in use <null>:8080

        at org.apache.catalina.connector.Connector.start(Connector.java:1129)

        at org.apache.catalina.core.StandardService.start(StandardService.java:540)

        at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)

        at org.apache.catalina.startup.Catalina.start(Catalina.java:595)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)

        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)

Jun 19, 2014 7:12:54 PM org.apache.jk.common.ChannelSocket init

INFO: Port busy 8009 java.net.BindException: Address already in use

Jun 19, 2014 7:12:54 PM org.apache.jk.common.ChannelSocket init

SEVERE: Can't find free port 8009 8009

Jun 19, 2014 7:12:54 PM org.apache.jk.server.JkMain start

INFO: Jk running ID=0 time=0/30  config=null

Jun 19, 2014 7:12:54 PM org.apache.coyote.http11.Http11Protocol start

SEVERE: Error starting endpoint

java.net.BindException: Address already in use <null>:8443

        at org.apache.tomcat.util.net.JIoEndpoint.init(JIoEndpoint.java:549)

        at org.apache.tomcat.util.net.JIoEndpoint.start(JIoEndpoint.java:565)

        at org.apache.coyote.http11.Http11Protocol.start(Http11Protocol.java:203)

        at org.apache.catalina.connector.Connector.start(Connector.java:1122)

        at org.apache.catalina.core.StandardService.start(StandardService.java:540)

        at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)

        at org.apache.catalina.startup.Catalina.start(Catalina.java:595)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)

        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)

Caused by: java.net.BindException: Address already in use

        at java.net.PlainSocketImpl.socketBind(Native Method)

        at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)

        at java.net.ServerSocket.bind(ServerSocket.java:376)

        at java.net.ServerSocket.<init>(ServerSocket.java:237)

        at java.net.ServerSocket.<init>(ServerSocket.java:181)

        at javax.net.ssl.SSLServerSocket.<init>(SSLServerSocket.java:136)

        at sun.security.ssl.SSLServerSocketImpl.<init>(SSLServerSocketImpl.java:107)

        at sun.security.ssl.SSLServerSocketFactoryImpl.createServerSocket(SSLServerSocketFactoryImpl.java:84)

        at org.apache.tomcat.util.net.jsse.JSSESocketFactory.createSocket(JSSESocketFactory.java:157)

        at org.apache.tomcat.util.net.JIoEndpoint.init(JIoEndpoint.java:538)

        ... 12 more

Jun 19, 2014 7:12:54 PM org.apache.catalina.core.StandardService start

SEVERE: Failed to start connector [Connector[HTTP/1.1-8443]]

LifecycleException:  service.getName(): "Catalina";  Protocol handler start failed: java.net.BindException: Address already in use <null>:8443

        at org.apache.catalina.connector.Connector.start(Connector.java:1129)

        at org.apache.catalina.core.StandardService.start(StandardService.java:540)

        at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)

        at org.apache.catalina.startup.Catalina.start(Catalina.java:595)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)

        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)

Jun 19, 2014 7:12:54 PM org.apache.catalina.startup.Catalina start

INFO: Server startup in 15787 ms

Jun 19, 2014 7:12:38 PM org.apache.coyote.http11.Http11Protocol pause

INFO: Pausing Coyote HTTP/1.1 on http-8080

[Andrew Woods]

Hello Ernie,

Per your log, it looks like a Tomcat service is already running on your machine:

java.net.BindException: Address already in use <null>:8080

and

java.net.BindException: Address already in use <null>:8443

You will likely want to shut all Tomcat services down, then restart.
Andrew

--
You received this message because you are subscribed to the Google Groups "Fedora Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fedora-communi...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ernie Gillis]

Thanka for the response Andrew! I had actually run the "shutdown.sh" script in the /usr/local/fedora folder (I'm running the packaged tomcat that is inside the Fedora installation). I even tried to run it several times and saw the output error that Catalina isn't running. Then I tried to run the startup.sh script, and continued to get the error I posted :-/
The only way to get it successfully started again was to fully restart the entire server.
The timing & reason for fedora to crash seems random. If I could figure out a pattern, I could force the crash om a clean log to help see what's happening :(

[Andrew Woods]

Hello Ernie,
Out of curiosity, what operating system are you on?
Andrew

[Ernie Gillis]

I'm on a VM running RHEL 6. It's only configured with 4GB of RAM (due to resource limitations on the VM).
I have the service currently running without a crash for 15 hrs now (yay!!). But I still see an error about trying to connect to JMS Messaging Service. Am watching log like a hawk now LOL

[Ernie Gillis]

So my server was running for a few days without issue. I was doing some collection object management, and then the system crashed.

I got the following error in my Fedora.log file:

ERROR 2014-06-23 17:12:54.157 [http-8443-7] (RISearchServlet) Unexpected error servicing API-A request

org.trippi.TrippiException: Query failed: GC overhead limit exceeded

at org.trippi.impl.mulgara.MulgaraSession.query(MulgaraSession.java:149) ~[trippi-mulgara-1.5.8.jar:na]

at org.trippi.impl.base.ConcurrentTriplestoreReader.findTuples(ConcurrentTriplestoreReader.java:80) ~[trippi-core-1.5.8.jar:na]

at org.fcrepo.server.resourceIndex.ResourceIndexImpl.findTuples(ResourceIndexImpl.java:280) ~[fcrepo-server-3.7.1.jar:na]

at org.fcrepo.server.resourceIndex.ResourceIndexModule.findTuples(ResourceIndexModule.java:313) ~[fcrepo-server-3.7.1.jar:na]

at org.trippi.server.TrippiServer.find(TrippiServer.java:119) ~[trippi-core-1.5.8.jar:na]

at org.fcrepo.server.access.RISearchServlet.doFind(RISearchServlet.java:356) [fcrepo-server-3.7.1.jar:na]

at org.fcrepo.server.access.RISearchServlet.doSearch(RISearchServlet.java:257) [fcrepo-server-3.7.1.jar:na]

at org.fcrepo.server.access.RISearchServlet.doGet(RISearchServlet.java:206) [fcrepo-server-3.7.1.jar:na]

at org.fcrepo.server.access.RISearchServlet.doGet(RISearchServlet.java:125) [fcrepo-server-3.7.1.jar:na]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) [servlet-api.jar:na]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) [servlet-api.jar:na]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) [catalina.jar:6.0.35]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:6.0.35]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:369) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]

at org.fcrepo.server.security.jaas.AuthFilterJAAS.doFilter(AuthFilterJAAS.java:329) [fcrepo-security-jaas-3.7.1.jar:na]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]

at org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:109) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]

at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:168) [spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]

at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) [spring-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]

at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) [spring-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) [catalina.jar:6.0.35]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) [catalina.jar:6.0.35]

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) [catalina.jar:6.0.35]

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) [catalina.jar:6.0.35]

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) [catalina.jar:6.0.35]

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [catalina.jar:6.0.35]

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [catalina.jar:6.0.35]

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) [catalina.jar:6.0.35]

at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) [tomcat-coyote.jar:6.0.35]

at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602) [tomcat-coyote.jar:6.0.35]

at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) [tomcat-coyote.jar:6.0.35]

at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]

Caused by: org.mulgara.query.QueryException: Query failed: GC overhead limit exceeded

at org.mulgara.resolver.DatabaseSession.execute(DatabaseSession.java:746) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.resolver.DatabaseSession.query(DatabaseSession.java:464) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.resolver.LocalJRDFDatabaseSession.query(LocalJRDFDatabaseSession.java:67) ~[mulgara-core-2.1.12.jar:na]

at org.trippi.impl.mulgara.MulgaraSession.query(MulgaraSession.java:147) ~[trippi-mulgara-1.5.8.jar:na]

... 32 common frames omitted

Caused by: org.mulgara.query.MulgaraTransactionException: Transaction rollback triggered

at org.mulgara.resolver.MulgaraInternalTransaction.implicitRollback(MulgaraInternalTransaction.java:516) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.resolver.MulgaraInternalTransaction.execute(MulgaraInternalTransaction.java:629) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.resolver.DatabaseSession.execute(DatabaseSession.java:743) ~[mulgara-core-2.1.12.jar:na]

... 35 common frames omitted

java.lang.OutOfMemoryError: GC overhead limit exceeded

at java.util.Arrays.copyOf(Arrays.java:2367) ~[na:1.7.0_51]

at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130) ~[na:1.7.0_51]

at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114) ~[na:1.7.0_51]

at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415) ~[na:1.7.0_51]

at java.lang.StringBuilder.append(StringBuilder.java:132) ~[na:1.7.0_51]

at org.mulgara.util.StackTrace.getStringTrace(StackTrace.java:95) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.util.StackTrace.<init>(StackTrace.java:60) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.store.xa.HybridTuples.clone(HybridTuples.java:283) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.store.tuples.Difference.clone(Difference.java:283) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.store.tuples.UnorderedProjection.<init>(UnorderedProjection.java:143) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.store.tuples.UnorderedProjection.clone(UnorderedProjection.java:317) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.store.xa.HybridTuples.clone(HybridTuples.java:284) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.store.tuples.DistinctTuples.clone(DistinctTuples.java:307) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.resolver.AppendAggregateTuples.clone(AppendAggregateTuples.java:214) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.resolver.GlobalizedAnswer.<init>(GlobalizedAnswer.java:112) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.resolver.SubqueryAnswer.<init>(SubqueryAnswer.java:110) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.resolver.DatabaseOperationContext.doQuery(DatabaseOperationContext.java:805) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.resolver.QueryOperation.execute(QueryOperation.java:136) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.resolver.MulgaraInternalTransaction.execute(MulgaraInternalTransaction.java:625) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.resolver.DatabaseSession.execute(DatabaseSession.java:743) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.resolver.DatabaseSession.query(DatabaseSession.java:464) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.resolver.LocalJRDFDatabaseSession.query(LocalJRDFDatabaseSession.java:67) ~[mulgara-core-2.1.12.jar:na]

at org.trippi.impl.mulgara.MulgaraSession.query(MulgaraSession.java:147) ~[trippi-mulgara-1.5.8.jar:na]

at org.trippi.impl.base.ConcurrentTriplestoreReader.findTuples(ConcurrentTriplestoreReader.java:80) ~[trippi-core-1.5.8.jar:na]

at org.fcrepo.server.resourceIndex.ResourceIndexImpl.findTuples(ResourceIndexImpl.java:280) ~[fcrepo-server-3.7.1.jar:na]

at org.fcrepo.server.resourceIndex.ResourceIndexModule.findTuples(ResourceIndexModule.java:313) ~[fcrepo-server-3.7.1.jar:na]

at org.trippi.server.TrippiServer.find(TrippiServer.java:119) ~[trippi-core-1.5.8.jar:na]

at org.fcrepo.server.access.RISearchServlet.doFind(RISearchServlet.java:356) [fcrepo-server-3.7.1.jar:na]

at org.fcrepo.server.access.RISearchServlet.doSearch(RISearchServlet.java:257) [fcrepo-server-3.7.1.jar:na]

at org.fcrepo.server.access.RISearchServlet.doGet(RISearchServlet.java:206) [fcrepo-server-3.7.1.jar:na]

at org.fcrepo.server.access.RISearchServlet.doGet(RISearchServlet.java:125) [fcrepo-server-3.7.1.jar:na]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) [servlet-api.jar:na]

[Ernie Gillis]

GAH! I'm driving myself crazy trying to sort this out :(

It crashed again... this time, no writes to the fedora.log, but I do have more writes to the catalina.out log.

Anyone? Bueller? 

-- catalina.out --

Jun 24, 2014 4:30:50 PM org.apache.catalina.startup.Catalina start

INFO: Server startup in 25142 ms

Exception in thread "Thread-7" java.lang.RuntimeException: Unable to start JMS Messaging Client, 5 attempts were made, each attempt resulted in a java.net.ConnectException. The messaging broker at tcp://localhost:61616 is not available

        at com.yourmediashelf.fedora.client.messaging.MessagingClient$JMSBrokerConnector.connect(MessagingClient.java:389)

        at com.yourmediashelf.fedora.client.messaging.MessagingClient$JMSBrokerConnector.run(MessagingClient.java:349)

Jun 25, 2014 4:34:28 PM org.apache.coyote.http11.Http11Processor process

SEVERE: Error processing request

java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "http-8443-5" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "timerFactory" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "http-8443-Acceptor-0" java.lang.OutOfMemoryError: GC overhead limit exceeded

Jun 25, 2014 4:34:56 PM org.apache.coyote.http11.Http11Processor process

SEVERE: Error processing request

java.lang.OutOfMemoryError: GC overhead limit exceeded

Jun 25, 2014 4:34:58 PM org.apache.coyote.http11.Http11Processor process

SEVERE: Error finishing response

java.lang.OutOfMemoryError: GC overhead limit exceeded

Jun 25, 2014 4:36:49 PM org.apache.coyote.http11.Http11Protocol pause

INFO: Pausing Coyote HTTP/1.1 on http-8080

Jun 25, 2014 4:36:51 PM org.apache.coyote.http11.Http11Protocol pause

INFO: Pausing Coyote HTTP/1.1 on http-8443

Jun 25, 2014 4:36:53 PM org.apache.catalina.core.StandardService stop

INFO: Stopping service Catalina

Jun 25, 2014 4:36:57 PM org.apache.catalina.core.StandardWrapper unload

INFO: Waiting for 1 instance(s) to be deallocated

Jun 25, 2014 4:36:58 PM org.apache.catalina.core.StandardWrapper unload

INFO: Waiting for 1 instance(s) to be deallocated

Jun 25, 2014 4:36:59 PM org.apache.catalina.core.StandardWrapper unload

INFO: Waiting for 1 instance(s) to be deallocated

Jun 25, 2014 4:37:03 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/fedora] appears to have started a thread named [JotmBatch] but has failed to stop it. This is very likely to create a memory leak.

Jun 25, 2014 4:37:03 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/fedora] appears to have started a thread named [JotmClock] but has failed to stop it. This is very likely to create a memory leak.

Jun 25, 2014 4:37:04 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/fedora] appears to have started a thread named [Write-lock Reaper] but has failed to stop it. This is very likely to create a memory leak.

Jun 25, 2014 4:37:04 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/fedora] appears to have started a thread named [http-8443-3] but has failed to stop it. This is very likely to create a memory leak.

Jun 25, 2014 4:37:04 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap

SEVERE: The web application [/fedora] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@603d164f]) and a value of type [org.apache.cxf.BusFactory.BusHolder] (value [org.apache.cxf.BusFactory$BusHolder@4a83cde5]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.

Jun 25, 2014 4:37:04 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap

SEVERE: The web application [/fedora] created a ThreadLocal with key of type [org.neilja.net.interruptiblermi.InterruptibleRMIServerSideSocket$1] (value [org.neilja.net.interruptiblermi.InterruptibleRMIServerSideSocket$1@13429b9b]) and a value of type [java.lang.Boolean] (value [false]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.

Jun 25, 2014 4:37:04 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap

SEVERE: The web application [/fedora] created a ThreadLocal with key of type [org.mulgara.store.tuples.SimpleTuplesFormat$1] (value [org.mulgara.store.tuples.SimpleTuplesFormat$1@79f5e6fd]) and a value of type [java.text.DecimalFormat] (value [java.text.DecimalFormat@674dc]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.

Jun 25, 2014 4:37:04 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap

SEVERE: The web application [/fedora] created a ThreadLocal with key of type [org.neilja.net.interruptiblermi.InterruptibleRMIServerSideSocket$1] (value [org.neilja.net.interruptiblermi.InterruptibleRMIServerSideSocket$1@13429b9b]) and a value of type [java.lang.Boolean] (value [false]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.

Jun 25, 2014 4:37:04 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap

SEVERE: The web application [/fedora] created a ThreadLocal with key of type [org.mulgara.store.tuples.SimpleTuplesFormat$1] (value [org.mulgara.store.tuples.SimpleTuplesFormat$1@79f5e6fd]) and a value of type [java.text.DecimalFormat] (value [java.text.DecimalFormat@674dc]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.

Jun 25, 2014 4:37:04 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap

SEVERE: The web application [/fedora] created a ThreadLocal with key of type [org.neilja.net.interruptiblermi.InterruptibleRMIServerSideSocket$1] (value [org.neilja.net.interruptiblermi.InterruptibleRMIServerSideSocket$1@13429b9b]) and a value of type [java.lang.Boolean] (value [false]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.

Jun 25, 2014 4:37:04 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap

SEVERE: The web application [/fedora] created a ThreadLocal with key of type [org.mulgara.store.tuples.SimpleTuplesFormat$1] (value [org.mulgara.store.tuples.SimpleTuplesFormat$1@79f5e6fd]) and a value of type [java.text.DecimalFormat] (value [java.text.DecimalFormat@674dc]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.

Jun 25, 2014 4:37:04 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap

SEVERE: The web application [/fedora] created a ThreadLocal with key of type [org.neilja.net.interruptiblermi.InterruptibleRMIServerSideSocket$1] (value [org.neilja.net.interruptiblermi.InterruptibleRMIServerSideSocket$1@13429b9b]) and a value of type [java.lang.Boolean] (value [false]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.

Jun 25, 2014 4:37:04 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap

SEVERE: The web application [/fedora] created a ThreadLocal with key of type [org.mulgara.store.tuples.SimpleTuplesFormat$1] (value [org.mulgara.store.tuples.SimpleTuplesFormat$1@79f5e6fd]) and a value of type [java.text.DecimalFormat] (value [java.text.DecimalFormat@674dc]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.

Jun 25, 2014 4:37:04 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap

SEVERE: The web application [/fedora] created a ThreadLocal with key of type [org.neilja.net.interruptiblermi.InterruptibleRMIServerSideSocket$1] (value [org.neilja.net.interruptiblermi.InterruptibleRMIServerSideSocket$1@13429b9b]) and a value of type [java.lang.Boolean] (value [false]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.

Jun 25, 2014 4:37:04 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap

SEVERE: The web application [/fedora] created a ThreadLocal with key of type [org.mulgara.store.tuples.SimpleTuplesFormat$1] (value [org.mulgara.store.tuples.SimpleTuplesFormat$1@79f5e6fd]) and a value of type [java.text.DecimalFormat] (value [java.text.DecimalFormat@674dc]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.

Jun 25, 2014 4:37:04 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap

SEVERE: The web application [/fedora] created a ThreadLocal with key of type [org.neilja.net.interruptiblermi.InterruptibleRMIServerSideSocket$1] (value [org.neilja.net.interruptiblermi.InterruptibleRMIServerSideSocket$1@13429b9b]) and a value of type [java.lang.Boolean] (value [false]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.

Jun 25, 2014 4:37:04 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap

SEVERE: The web application [/fedora] created a ThreadLocal with key of type [org.mulgara.store.tuples.SimpleTuplesFormat$1] (value [org.mulgara.store.tuples.SimpleTuplesFormat$1@79f5e6fd]) and a value of type [java.text.DecimalFormat] (value [java.text.DecimalFormat@674dc]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.

Jun 25, 2014 4:37:14 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap

SEVERE: The web application [/fedoragsearch] created a ThreadLocal with key of type [org.apache.axis.utils.XMLUtils.ThreadLocalDocumentBuilder] (value [org.apache.axis.utils.XMLUtils$ThreadLocalDocumentBuilder@34b93cb3]) and a value of type [org.apache.xerces.jaxp.DocumentBuilderImpl] (value [org.apache.xerces.jaxp.DocumentBuilderImpl@6d4026cf]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.

log4j:ERROR LogMananger.repositorySelector was null likely due to error in class reloading, using NOPLoggerRepository.

Jun 25, 2014 4:37:27 PM org.apache.coyote.http11.Http11Protocol destroy

INFO: Stopping Coyote HTTP/1.1 on http-8080

Jun 25, 2014 4:37:27 PM org.apache.coyote.http11.Http11Protocol destroy

INFO: Stopping Coyote HTTP/1.1 on http-8443

[Benjamin Armintor]

You either have too little memory assigned to the Java heap for your install, or you should use a different GC algorithm. Are you sure that the JAVA_OPTS you mentioned are the ones that Tomcat is using?

[Ernie Gillis]

It should be. I only have JAVA_OPTS for the one time:

$] echo $JAVA_OPTS

-Xms1024m -Xmx1024m -XX:PermSize=32M -XX:MaxPermSize=128m

Is there a place I can check to see what tomcat may be using (if it's not grabbing the env vars)?

[Benjamin Armintor]

If I remember correctly, modern versions of Tomcat list that out in the server status page of the bundled manager app.

[Andrew Woods]

Hello Ernie,

Have you given Fedora 4 a look? It should be much easier to use.

https://wiki.duraspace.org/display/FF/Quick+Start

Andrew

On Wed, Jun 25, 2014 at 5:00 PM, Ernie Gillis <egi...@berklee.edu> wrote:

Thanks Andrew!!

I always had a hunch it had something to do with JMS... but don't know how to debug it, since the documentation didn't come across as quite clear to me for it. I know it's a service, but I don't believe it's a client, or is it? And where would I test / check it :?

On Wed, Jun 25, 2014 at 4:51 PM, Andrew Woods <awo...@duraspace.org> wrote:

Hello Ernie,

There is not a lot of context here, but it looks like your JMS message broker is not running and you are running into Java garbage-collection limits. 

Someone on the list may have some insights if you post your JAVA_OPTS and/or CATALINA_OPTS as well as a description of your setup.

Regards,

Andrew

--

You received this message because you are subscribed to the Google Groups "Fedora Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fedora-communi...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Ernest Gillis
Manager of Learning Resources Web Development
Stan Getz Media Center and Library
Phone: 617-747-8533
E-Mail: egi...@berklee.edu

[Ernie Gillis]

I'm running fedora 4 on a test environment right now, but Islandora is my main discovery layer, so I have to still use 3.7.1

I did find the Tomcat server status page.

JVM has an output of:

Free memory: 321.48 MB Total memory: 363.00 MB Max memory: 853.50 MB


And that's the only memory output I can find in the manager web app. 

If I run the "find leaks" test, it all comes back clean.

I'm going trying to go through various tomcat files now to see if I can find a hardcoded JAVA_OPTS.

My tomcat is the bundled tomcat 6 that comes in fedora 3.7.1. I was going to use yum to install tomcat, but the IT folks didn't like me adding random repositories to the the RHEL OS

[Ernie Gillis]

Due to deadlines, I ended up having to go live. The server still crashes.

If anyone has diagnostic advice (tools, code to look at, etc, etc), it would be greatly appreciated.

[Timothy Cornwell]

Ernie,

It appears that the jvm does not have enough memory to do what you are asking of it. "... java.lang.OutOfMemoryError: GC overhead limit exceeded" is pretty clear and this (https://plumbr.eu/outofmemoryerror/gc-overhead-limit-exceeded) has a nice explanation.

What else is running on this system? If you can spare more memory for the JVM, then I'd expand Xms and Xmx to something bigger - 2g perhaps.

A "ps -ef | grep jav[a]" should return what the jvm is using as startup parmeters - it would be good to know, especially if you can't find all the java options that might be set. This may lead you to a different or better configured GC mechanism too.

Best of luck,
-T

[Ernie Gillis]

Thanks Timothy!

It took me a while to get X11 forwarding working and then connecting to the process (which is owned by root) to launch in the jconsole, but I've gotten to that point!

I'm going through all the threads to see what trace outputs there are, but, so far, nothing jumping out at me... or at least nothing jumping out that seems any different from what was written to the logs :(

According to the various memory displays, it says I have over a gig of memory avaialble.

In my debugging, I have found more of a pattern to the crashing. It appears  as though the crashing is linked to the fgsearch webapp. Or, at least how it couples with Islandora's Tuque. If I do object management directly in fedora, I can go for a much longer time without issue. As soon as I do work within Islandora (with Tuque and FgSearch), Fedora starts to have the memory leak and crash. I don't see anything in the log to say where / why this happens, yet, but this is what I have so far.

Thanks for the assistance!

[Ernie Gillis]

More updates!

I have tracked down some possible related issues (with the help of jconsole).

It seems as though catalina.sh is not recognizing environment variables that I set inside my RHEL "/etc/profile.d/" directory. I have a "fedora.sh" file there that sets things like JAVA_OPTS, JAVA_HOME, etc. These environment variables are available for all users (at least from all tests I've done), but when "catalina.sh" gets called from "startup.sh", none of these environment variables persist. It seems as though the tomcat scripts bundled aren't correclty grabbing the environment variables.

Example debug / test:

file:  /etc/profile.d/ernie.sh

content:

export ERNIE_TEST=hello

On the command line I can:

$echo $ERNIE_TEST

hello

Or

$ sudo -s

$ echo $ERNIE_TEST

hello

I have tried with various shell headers of "#!/etc/bash" or "#!/etc/sh" or nothing

in my catalina / tomcat "startup.sh" file, I have the following at the very end:

echo "testing"

echo $ERNIE_TEST

echo "end test"

When I start catalina / tomcat  with "./startup.sh", my output is:

testing

end test

A few things with this.... I have no idea if this has any relation to the memory leak I have. It doesn't seem that tomcat / catalina grabs any variables I export in my "profile.d". I'd like to have my environment variables there so I can have it global for all users. This makes me think other variables are not set properly, but I don't feel I should be editing the "startup.sh" directly. 

[Ernie Gillis]

Small update to my last post. 

I figured out a solution to the environment variable issue.

I have been trying to centralize and persist my environment variables, but have had issues with things not work.

I have my environment variables set in a custom "fedora.sh" file located at "/etc/profile.d/fedora.sh"

I run my fedora installation with:

CATALINA_HOME/bin $] sudo ./startup.sh

I thought that the "profile.d" script files would persist for all users. It turns out, when I do "sudo ....." the custom environment variables aren't persisting. I have to run it as "sudo -E ...." to persist the environment variables.

I guess that means my hope of centralizing and persisting things may be all for not.... we'll see.

Now to see how long the before the memory leak creeps up again :-/

[Scott Prater]

A simpler, more robust solution may be to create a
$CATALINA_HOME/bin/setenv.sh file, and put the environment variables you
want in there. $CATALINA_HOME/bin/catalina.sh as shipped with Tomcat
looks for setenv.sh in the bin directory, and if it exists, sources it
in. A little-known feature of Tomcat.

From catalina.sh:

# Ensure that any user defined CLASSPATH variables are not used on startup,
# but allow them to be specified in setenv.sh, in rare case when it is
needed.
CLASSPATH=

if [ -r "$CATALINA_BASE"/bin/setenv.sh ]; then
. "$CATALINA_BASE"/bin/setenv.sh
elif [ -r "$CATALINA_HOME"/bin/setenv.sh ]; then
. "$CATALINA_HOME"/bin/setenv.sh
fi

-- Scott

On 07/09/2014 01:01 PM, Ernie Gillis wrote:
> Small update to my last post.
>
> I figured out a solution to the environment variable issue.
> I have been trying to centralize and persist my environment variables,
> but have had issues with things not work.
>
> I have my environment variables set in a custom "fedora.sh" file located
> at "/etc/profile.d/fedora.sh"
>
> I run my fedora installation with:
> CATALINA_HOME/bin $] sudo ./startup.sh
>
>

> I *thought* that the "profile.d" script files would persist for all

> users. It turns out, when I do "sudo ....." the custom environment
> variables aren't persisting. I have to run it as "sudo -E ...." to
> persist the environment variables.
>
> I guess that means my hope of centralizing and persisting things may be
> all for not.... we'll see.
>
>
> Now to see how long the before the memory leak creeps up again :-/
>

> --
> You received this message because you are subscribed to the Google
> Groups "Fedora Community" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to fedora-communi...@googlegroups.com

> <mailto:fedora-communi...@googlegroups.com>.

> For more options, visit https://groups.google.com/d/optout.

--
Scott Prater
Shared Development Group
General Library System
University of Wisconsin - Madison
pra...@wisc.edu
5-5415

[Ernie Gillis]

I'm back and forth between here and the Islandora group.

Out of curiosity... would running Java 8 with Fedora 3.7.1 be the root of my issue? I didn't go back to Java 6 or 7 in order to make sure I was up to date with it, and have any security fixes in place.

Also, I decided to enable the JMS in my fedora.fcfg just to see what would happen. This is the catalina.out log content:

INFO: Server startup in 12844 ms

Exception in thread "pool-2-thread-3" java.lang.OutOfMemoryError: GC overhead limit exceeded

        at java.lang.String.toCharArray(String.java:2884)

        at java.util.zip.ZipCoder.getBytes(ZipCoder.java:78)

        at java.util.zip.ZipFile.getEntry(ZipFile.java:311)

        at java.util.jar.JarFile.getEntry(JarFile.java:240)

        at java.util.jar.JarFile.getJarEntry(JarFile.java:223)

        at org.apache.catalina.loader.WebappClassLoader.findResources(WebappClassLoader.java:1315)

        at java.lang.ClassLoader.getResources(ClassLoader.java:1139)

        at java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:348)

        at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)

        at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)

        at javax.xml.parsers.FactoryFinder$1.run(FactoryFinder.java:293)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.xml.parsers.FactoryFinder.findServiceProvider(FactoryFinder.java:289)

        at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:267)

        at javax.xml.parsers.SAXParserFactory.newInstance(SAXParserFactory.java:127)

        at info.aduna.xml.XMLReaderFactory.createXMLReader(XMLReaderFactory.java:66)

        at org.openrdf.rio.rdfxml.RDFXMLParser.parse(RDFXMLParser.java:256)

        at org.openrdf.rio.rdfxml.RDFXMLParser.parse(RDFXMLParser.java:209)

        at org.trippi.io.RIOTripleIterator.run(RIOTripleIterator.java:163)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

Jul 17, 2014 3:07:47 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run

SEVERE: Socket accept failed

java.lang.OutOfMemoryError: GC overhead limit exceeded

Jul 17, 2014 3:07:52 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run

SEVERE: Socket accept failed

java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "ActiveMQ Data File Writer" java.lang.OutOfMemoryError: GC overhead limit exceeded

Still no idea how to remedy this :-\

[Ernie Gillis]

To answer my own question... it does not seem as though Java SE 8 was the root of the problem. I rolled back to Java SE 7 (couldn't find a Java SE 6 download), and still had it crash

INFO: Server startup in 22677 ms

Jul 17, 2014 4:26:39 PM org.apache.jasper.runtime.JspFactoryImpl internalGetPageContext

SEVERE: Exception initializing page context

java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "ActiveMQ Journal Checkpoint Worker" java.lang.OutOfMemoryError: GC overhead limit exceeded

Jul 17, 2014 4:26:59 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run

SEVERE: Socket accept failed

java.lang.OutOfMemoryError: GC overhead limit exceeded

Jul 17, 2014 4:27:11 PM org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor processChildren

SEVERE: Exception invoking periodic operation: 

java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "http-8443-Acceptor-0" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: GC overhead limit exceeded

Jul 17, 2014 4:27:41 PM org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor processChildren

SEVERE: Exception invoking periodic operation: 

[Timothy Cornwell]

Ernie,

 

Did you ever increase your (JAVA_OPTS) to anything above  Xmx=1024?  I believe that bumping this up may improve your up-time.

 

I understand that you are running on a 4G system, and that memory may be tight, but unless you are currently utilizing swap space heavily, I believe that increasing the Xmx (and Xms) in your JAVA_OPTS for fedora to something more than 1024 will probably improve your up-ness.  I’d at least try half again as much: -Xmx=1536m –Xms=1536m and see what happens.  You may end up utilizing more swap space, and performance may suffer, but overall, you should be able to keep the jvm from crashing with the OutOfMemoryError.  Your jvm is running out of memory.  You didn’t mention what else is running on this system, so it is hard to tell how much memory you have available.  In my experience, on a 4G Red Hat 6 system running only 1 jvm, and perhaps apache httpd, you should comfortably be able to run a jvm with at least -Xmx=2G and perhaps more.

 

Also:  I would not characterize this as a memory leak unless you have code you know to be misbehaving, or that can be counted on to increase memory usage over time and not free it up.  Running out of memory can also happen when there just isn’t enough of it to do what is called for.  Happens to me all the time.

 

Hth,

-T

 

Timothy Cornwell

Application System Management

Commercial Applications Group

CIT Information Systems

 

 

From: fedora-c...@googlegroups.com [mailto:fedora-c...@googlegroups.com] On Behalf Of Ernie Gillis
Sent: Thursday, July 17, 2014 5:11 PM
To: fedora-c...@googlegroups.com
Cc: egi...@berklee.edu
Subject: Re: [fedora-community] severe error "[/fedora] appears to have started a thread ... but has failed to stop it"

 

To answer my own question... it does not seem as though Java SE 8 was the root of the problem. I rolled back to Java SE 7 (couldn't find a Java SE 6 download), and still had it crash

--

You received this message because you are subscribed to the Google Groups "Fedora Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fedora-communi...@googlegroups.com.

[Ernie Gillis]

I found out how to do a jmap heap dump, and have attached screen shots of three different bits of info that jumped out at me: "Leak suspects", "Duplicate classes", and "Unreachable Objects". I may be grasping at straws, but it is info I didn't have before.

Also... I had responded via email to TImothy's post, but didn't do reply all (sorry for the double thread, Timothy)....

Many thanks Timothy!
I am so new to all things java that I don't know what base resources are required. I will up to 1.5G on the ram and see what happens :)

On the plus's side, this VM I am on has no GUI (saving cycles / ram there), and the daemons are postgres (for fedora), mysql (for Drupal), apache2 (httpd), and tomcat. There are the system services aside from those, and selinux is in permissive mode.

I have binaries for the djatoka, imagemagick, etc. And LUNs attached for the different data storage needs (like fedora).

Thanks again!! The more I read the more I know! :)

[Ernie Gillis]

I was able to recreate the problem without getting to a crash point (but leaving just 30-ish MB left of memory for the JVM). I'm attaching a screenshot of the Tomcat Server Status page. You can also note the various tabs of my browser trying to load. I'm loading each of those tabs as my pressure test to see if / when the server crashed.

It still points to something called "mulgara" in the catalina WebappClassLoader (both of which I have very little knowledge of what they are / do).

I've also attached some jconsole screenshots to show stats / settings in that same timeframe (though they were a little earlier, so the memory is used up quite as much).

[Ernie Gillis]

Quick update, is kind of a non update, but more to just keep it documented.

My workload shifted me away from this. It still happens, but we aren't officially live, yet, so it isn't as urgent (at the moment).

My next test steps are going to be:
- see if I can replicate the memory issue purely through the ri interface (without using Islandora or tuque)

If it still happens, I will know it's more fedora than tuque. If it doesn't crash, I will know it's more tuque.

Onwards and upwards!

[Ernie Gillis]

I did some more testing, and I did find out that the "RISearch" interface had far better garbage collection than through the Islandora Tuque (no idea why). I also found some more JAVA_OPTS values that appears to have greatly reduced the chance of crashing.

So... Now... My JAVA_OPTS includes:

-Xms1524m -Xmx1524m -XX:NewSize=256m -XX:MaxNewSize=512m -XX:PermSize=256m -XX:MaxPermSize=512m -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+UseConcMarkSweepGC

I still can invoke a crash, but the garbage collection gets trigger more frequently. Also, setting my Max Size values (MaxNewSize & MaxPermSize) to 512 seems to have allow all the libraries to load properly (which I now know is what triggers the Heap Memory error). This is hopefully a solution, but time will tell :)

Thanks to all who gave assistance! :)

[Ernie Gillis]

OK. I had a false positive. My server still ended up crashing, but I think I did get a positive positive! Yay!!

I went back to the drawing board and reviewed everything I had done for server setup. When I did the first install of fedora commons with the embedded tomcat, I had startup conflict issues because of the SLF4J logger ".jar" files. Tomcat's core "lib" directory, and every webapp for fedora usage appears to have an SLF4J file that creates a lot of "warning" havoc in the log files. For Fedora, this conflict actually causes failure to start the webapp.

In my original install, I decided to keep the tomcat "lib" directory SLF4J file, because I didn't know what would happen with the core tomcat server. I moved my SLF4J file out of "webapps/fedora/WEB-INF/lib" and put it in "webapps/fedora/WEB-INF" with a ".bak" suffix. All things ran smoothly.

When my recent round of tests, I decided to put the "webapps/fedora" SLF4J file back, and move the "tomcat/lib" SLF4J file into "tomcat" with a ".bak" suffix.

If you've read this far, I think the the tomcat SLF4J is a lower version, but the version numbers in the filenames are a little odd to follow. In the tomcat "lib" folder, it is called "slf4j-log4j12-1.6.6.jar", but in the "webapps/fedora" "lib" directory, it's called "log4j-over-slf4j-1.7.2.jar". Again, having both files in place caused a non-start for the fedora service. Removing one allowed the fedora service to start properly. The big hint I had was when I tried doing a rebuild for the first time... without the "-1.7.2.jar" file, the rebuild failed.

I'm still not certain if "-1.7.2.jar" is newer than the "-1.6.6.jar" because the "-.1.6.6.jar" file has a "12" in the filename, and the format of how the file is named is different (with the "-1.7.2.jar" having the whole "-over-" in it).

Either way... fingers crossed to see how well this stays up. So far, much improved garbage collection on the whole.

[Benjamin Armintor]

"When I did the first install of fedora commons with the embedded tomcat, I had startup conflict issues because of the SLF4J logger ".jar" files. Tomcat's core "lib" directory, and every webapp for fedora usage appears to have an SLF4J file that creates a lot of "warning" havoc in the log files. For Fedora, this conflict actually causes failure to start the webapp."

I don't follow this. If the embedded Tomcat had a conflict with the slf4j library that caused the webapp not to start, none of the integration tests would run. Do you mean that you were installing Fedora into an existing Tomcat?

--

[Ernie Gillis]

Hi Armintor,

My process was from the beginning, was:

- download the fcrepo jar file

- install the fcrepo and use the embedded tomcat server

- start the tomcat server

   + at this point, the fedora.log file showed me that fedora would not start due to a slf4j logger conflict

   + tomcat would start, and I could see the manager page, but the fedora webapp would not start

   + in researching, I found that the "/usr/local/fedora/tomcat/webapps/fedora/WEB-INF/lib/log4j-over-slf4j-1.7.2.jar" was causing a conflict with "/usr/local/fedora/tomcat/lib/slf4j-log4j12-1.6.6.jar"

- I ran "mv /usr/local/fedora/tomcat/webapps/fedora/WEB-INF/lib/log4j-over-slf4j-1.7.2.jar /usr/local/fedora/tomcat/webapps/fedora/WEB-INF/log4j-over-slf4j-1.7.2.jar.bak"

- i restarted the tomcat service (the one that's embedded... i never installed a separate tomcat server)

- fedora webapp started properly

If I remember right, there wasn't really any instructions in the wiki pages about this slf4j conflict, but I did find the issue on an slf4j documentation page. 

Since I had the memory leak issues, I went back (and still not unconfirmed of which slf4j is the more recent one), and ran: 

     mv /usr/local/fedora/tomcat/webapps/fedora/WEB-INF/log4j-over-slf4j-1.7.2.jar.bak /usr/local/fedora/tomcat/webapps/fedora/WEB-INF/lib/log4j-over-slf4j-1.7.2.jar

     mv /usr/local/fedora/tomcat/lib/slf4j-log4j12-1.6.6.jar /usr/local/fedora/tomcat/lib/slf4j-log4j12-1.6.6.jar.bak

So far, the memory leaks seem to be little or nonexistent, and garbage collection seems to be acting normally. 

[Benjamin Armintor]

Can you post a link to the installer you downloaded? If this is true, it's a distribution error that needs to be fixed.

[Benjamin Armintor]

I say this because the tomcat zip distributed hasn't changed since v3.6.1, and I don't see that slf4j-log4j12-1.6.6.jar anywhere in the unpacked Tomcat in the integration tests.

[Ernie Gillis]

Hey armintor!

As an FYI, my first thread of this issue relating to the slf4j conflict was here:

     https://groups.google.com/forum/#!searchin/fedora-community/slf4j/fedora-community/uEUmEzQcM9Y/BPDtUgwxNQMJ

I did have a few issues to start (i.e. java versions and such), so I wonder if that could have created erroneous files as I had? It's not clear. My install was a rather long time ago, but I had used both "fcrepo-installer-3.7.0.jar" and "fcrepo-installer-3.7.1.jar" installers as ways to test. All of my downloads were from the duraspace wiki for Fedora Commons. I  It could have all been problems from the early java version issues I had (plus I'm also on RHEL with SELinux, which compounded many debugging issues for me). I have also had issues that my "/usr/local/fedora/tomcat" folder started out as "/usr/local/fedora/base" but on recent resintalls (with the same ".jar" files) I don't get the "base" folder name anymore.

[Benjamin Armintor]

Just to confirm: I just downloaded and installed 3.7.0 and 3.7.1, and neither embedded tomcat lib directory includes a file called *slf4j*.jar

Here's a listing of all such jars across those two installs:

./fcrepo-3.7.0/client/lib/jcl-over-slf4j-1.7.2.jar

./fcrepo-3.7.0/client/lib/log4j-over-slf4j-1.7.2.jar

./fcrepo-3.7.0/client/lib/slf4j-api-1.7.2.jar

./fcrepo-3.7.0/tomcat/webapps/fedora/WEB-INF/lib/jcl-over-slf4j-1.7.2.jar

./fcrepo-3.7.0/tomcat/webapps/fedora/WEB-INF/lib/jul-to-slf4j-1.7.2.jar

./fcrepo-3.7.0/tomcat/webapps/fedora/WEB-INF/lib/log4j-over-slf4j-1.7.2.jar

./fcrepo-3.7.0/tomcat/webapps/fedora/WEB-INF/lib/slf4j-api-1.7.2.jar

./fcrepo-3.7.0/tomcat/webapps/fop/WEB-INF/lib/jcl-over-slf4j-1.7.2.jar

./fcrepo-3.7.0/tomcat/webapps/fop/WEB-INF/lib/slf4j-api-1.7.2.jar

./fcrepo-3.7.0/tomcat/webapps/imagemanip/WEB-INF/lib/jcl-over-slf4j-1.7.2.jar

./fcrepo-3.7.0/tomcat/webapps/imagemanip/WEB-INF/lib/slf4j-api-1.7.2.jar

./fcrepo-3.7.0/tomcat/webapps/saxon/WEB-INF/lib/jcl-over-slf4j-1.7.2.jar

./fcrepo-3.7.0/tomcat/webapps/saxon/WEB-INF/lib/slf4j-api-1.7.2.jar

./fcrepo-3.7.1/client/lib/jcl-over-slf4j-1.7.2.jar

./fcrepo-3.7.1/client/lib/log4j-over-slf4j-1.7.2.jar

./fcrepo-3.7.1/client/lib/slf4j-api-1.7.2.jar

./fcrepo-3.7.1/tomcat/webapps/fedora/WEB-INF/lib/jcl-over-slf4j-1.7.2.jar

./fcrepo-3.7.1/tomcat/webapps/fedora/WEB-INF/lib/jul-to-slf4j-1.7.2.jar

./fcrepo-3.7.1/tomcat/webapps/fedora/WEB-INF/lib/log4j-over-slf4j-1.7.2.jar

./fcrepo-3.7.1/tomcat/webapps/fedora/WEB-INF/lib/slf4j-api-1.7.2.jar

./fcrepo-3.7.1/tomcat/webapps/fop/WEB-INF/lib/jcl-over-slf4j-1.7.2.jar

./fcrepo-3.7.1/tomcat/webapps/fop/WEB-INF/lib/slf4j-api-1.7.2.jar

./fcrepo-3.7.1/tomcat/webapps/imagemanip/WEB-INF/lib/jcl-over-slf4j-1.7.2.jar

./fcrepo-3.7.1/tomcat/webapps/imagemanip/WEB-INF/lib/slf4j-api-1.7.2.jar

./fcrepo-3.7.1/tomcat/webapps/saxon/WEB-INF/lib/jcl-over-slf4j-1.7.2.jar

./fcrepo-3.7.1/tomcat/webapps/saxon/WEB-INF/lib/slf4j-api-1.7.2.jar

- Ben 

--

[Ernie Gillis]

That makes sense to me, though I can't really explain the why.

I had originally downloaded the 3.7.0 and 3.7.1 installers back in February or March. I've been constantly running into these random issues (like making the default tomcat folder "base" and not "tomcat", or the memory leaks). I have run the installers on different servers (RHEL and CentOS only, but on different environments of public, dev, VM, hardware, etc). For some reason, this week was the first time I did the install and there wasn't the *slf4j*.jar file in the tomcat/lib after the install, AND the "base" folder was properly called "tomcat". 

I haven't re-downloaded the installers, I never installed anything outside the purpose of Fedora Commons packages - it is a "unitasking" server, but I do have gsearch, adore-djatoka for added support for fedora. I don't know that I changed anything major (aside from SE Linux configuration), so I went back to my more production installations and update the various library content folders to match the fresh install.

Is all rather a mystery to me at this point