Hello Everyone,
I recently installed the DataVerse solution on a virtual machine running Red Hat Enterprise Linux Server release 7.1 (Maipo). Because this is my first installation of DVN I followed the indications of the install guide that I found on the Dataverse.org website. All in all this went rathter smooth and the website is up and running which yuo can see here "
http://sohda.cumulus.vub.ac.be/".
The problem I have is that the site will be up for some time, normally like 6 to 8 hours after which it stops responding to url requests.
I have looked into the logs and found this as only indication:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[2015-09-21T12:41:17.577+0200] [glassfish 4.1] [WARN] [] [org.jboss.weld.Conversation] [tid: _ThreadID=26 _ThreadName=http-listener-1(4)] [timeMillis: 1442832077577] [levelValue: 900] [[
WELD-000315: Failed to acquire conversation lock in 1,000 ms for Transient conversation]]
[2015-09-21T12:50:37.275+0200] [glassfish 4.1] [WARN] [] [org.jboss.weld.Conversation] [tid: _ThreadID=23 _ThreadName=http-listener-1(1)] [timeMillis: 1442832637275] [levelValue: 900] [[
WELD-000315: Failed to acquire conversation lock in 1,000 ms for Transient conversation]]
[2015-09-21T17:33:26.458+0200] [glassfish 4.1] [SEVERE] [AS-WEB-CORE-00114] [javax.enterprise.web.core] [tid: _ThreadID=98 _ThreadName=ContainerBackgroundProcessor[StandardEngine[glassfish-web].StandardHost[server].StandardContext[]]] [timeMillis: 1442849606458] [levelValue: 1000] [[
Exception invoking periodic operation:
java.lang.OutOfMemoryError: Java heap space
at org.apache.catalina.session.ManagerBase.findSessions(ManagerBase.java:918)
at org.apache.catalina.session.StandardManager.processExpires(StandardManager.java:1050)
at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:6340)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1823)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1812)
at java.lang.Thread.run(Thread.java:745)
]]
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
When I get this problem the only thing I can do to restart the website is to stop the domain (/usr/local/glassfish4/bin/asadmin stop-domain) and then afterwards restarting it. This will keep it going again for a couple of hours. Somtimes I have to realy force a kill of the JDK process with a -9 signal in order to shut the domain down. Still, I always get the site up and running again.
Now, in the beginning I attributed only 2GB of memory to -Xmx and I thought that increasing this to 5G would solve the issue. Still, this isn't the case. I still keep getting the same behavior.
I am looking for clues here in the dark and I really would appreciate any help or clues you give me. Maybe someone also experienced this same issue?
Thanks in advance for any suggestion regarding my issue.
Xavier
PS: The give you an idea on how this site is configured I am pasting here the startup procedure from the same log file as above.
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[2015-09-22T08:05:21.386+0200] [] [INFO] [NCLS-GFLAUNCHER-00005] [javax.enterprise.launcher] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1442901921386] [levelValue: 800] [[
JVM invocation command line:
/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85-2.6.1.2.el7_1.x86_64/bin/java
-cp
/usr/local/glassfish4/glassfish/modules/glassfish.jar
-XX:+UnlockDiagnosticVMOptions
-XX:PermSize=256m
-XX:MaxPermSize=512m
-XX:NewRatio=2
-Xmx5864m
-client
-javaagent:/usr/local/glassfish4/glassfish/lib/monitor/flashlight-agent.jar
-Ddataverse.files.directory=/usr/local/glassfish4/glassfish/domains/domain1/files
-Dfelix.fileinstall.disableConfigSave=false
-Djavax.net.ssl.keyStore=/usr/local/glassfish4/glassfish/domains/domain1/config/keystore.jks
-Ddoi.password=apitest
-Djava.awt.headless=true
-Dcom.ctc.wstx.returnNullForDefaultNamespace=true
-Dfelix.fileinstall.poll=5000
-Djava.endorsed.dirs=/usr/local/glassfish4/glassfish/modules/endorsed:/usr/local/glassfish4/glassfish/lib/endorsed
-Ddoi.username=apitest
-Dfelix.fileinstall.bundles.startTransient=true
-Djavax.net.ssl.trustStore=/usr/local/glassfish4/glassfish/domains/domain1/config/cacerts.jks
-Djavax.xml.accessExternalSchema=all
-Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as
-Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl
-Djava.security.auth.login.config=/usr/local/glassfish4/glassfish/domains/domain1/config/login.conf
-DANTLR_USE_DIRECT_CLASS_LOADING=true
-Dgosh.args=--nointeractive
-Ddataverse.rserve.port=6311
-Dosgi.shell.telnet.maxconn=1
-Djdbc.drivers=org.apache.derby.jdbc.ClientDriver
-Dfelix.fileinstall.dir=/usr/local/glassfish4/glassfish/modules/autostart/
-Dosgi.shell.telnet.port=6666
-Djava.security.policy=/usr/local/glassfish4/glassfish/domains/domain1/config/server.policy
-Dfelix.fileinstall.log.level=2
-Dcom.sun.aas.instanceRoot=/usr/local/glassfish4/glassfish/domains/domain1
-Ddataverse.rserve.user=rserve
-Ddataverse.rserve.host=localhost
-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
-Dosgi.shell.telnet.ip=127.0.0.1
-Dcom.sun.aas.installRoot=/usr/local/glassfish4/glassfish
-Djava.ext.dirs=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85-2.6.1.2.el7_1.x86_64/lib/ext:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85-2.6.1.2.el7_1.x86_64/jre/lib/ext:/usr/local/glassfish4/glassfish/domains/domain1/lib/ext
-Ddataverse.auth.password-reset-timeout-in-minutes=60
-Dfelix.fileinstall.bundles.new.start=true
-Dorg.glassfish.additionalOSGiBundlesToStart=org.apache.felix.shell,org.apache.felix.gogo.runtime,org.apache.felix.gogo.shell,org.apache.felix.gogo.command,org.apache.felix.shell.remote,org.apache.felix.fileinstall
-Ddataverse.rserve.password=<LEFT BLANK ;)>
-Djdk.corba.allowOutputStreamSubclass=true
-Djava.library.path=/usr/local/glassfish4/glassfish/lib:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
com.sun.enterprise.glassfish.bootstrap.ASMain
-domainname
domain1
-asadmin-args
--host,,,localhost,,,--port,,,4848,,,--secure=false,,,--terse=false,,,--echo=false,,,--interactive=true,,,start-domain,,,--verbose=false,,,--watchdog=false,,,--debug=false,,,--domaindir,,,/usr/local/glassfish4/glassfish/domains,,,domain1
-instancename
server
-verbose
false
-debug
false
-asadmin-classpath
/usr/local/glassfish4/glassfish/lib/client/appserver-cli.jar
-asadmin-classname
com.sun.enterprise.admin.cli.AdminMain
-upgrade
false
-type
DAS
-domaindir
/usr/local/glassfish4/glassfish/domains/domain1
-read-stdin
true]]
[2015-09-22T08:05:28.153+0200] [glassfish 4.1] [INFO] [NCLS-LOGGING-00009] [javax.enterprise.logging] [tid: _ThreadID=15 _ThreadName=RunLevelControllerThread-1442901927958] [timeMillis: 1442901928153] [levelValue: 800] [[
Running GlassFish Version: GlassFish Server Open Source Edition 4.1 (build 13)]]
[2015-09-22T08:05:28.157+0200] [glassfish 4.1] [INFO] [NCLS-LOGGING-00010] [javax.enterprise.logging] [tid: _ThreadID=15 _ThreadName=RunLevelControllerThread-1442901927958] [timeMillis: 1442901928157] [levelValue: 800] [[
Server log file is using Formatter class: com.sun.enterprise.server.logging.ODLLogFormatter]]
[2015-09-22T08:05:28.882+0200] [glassfish 4.1] [INFO] [NCLS-SECURITY-01115] [javax.enterprise.system.core.security] [tid: _ThreadID=14 _ThreadName=RunLevelControllerThread-1442901927955] [timeMillis: 1442901928882] [levelValue: 800] [[
Realm [admin-realm] of classtype [com.sun.enterprise.security.auth.realm.file.FileRealm] successfully created.]]
[2015-09-22T08:05:28.928+0200] [glassfish 4.1] [INFO] [] [org.glassfish.ha.store.spi.BackingStoreFactoryRegistry] [tid: _ThreadID=15 _ThreadName=RunLevelControllerThread-1442901927958] [timeMillis: 1442901928928] [levelValue: 800] [[
Registered org.glassfish.ha.store.adapter.cache.ShoalBackingStoreProxy for persistence-type = replicated in BackingStoreFactoryRegistry]]
[2015-09-22T08:05:28.998+0200] [glassfish 4.1] [INFO] [NCLS-SECURITY-01115] [javax.enterprise.system.core.security] [tid: _ThreadID=14 _ThreadName=RunLevelControllerThread-1442901927955] [timeMillis: 1442901928998] [levelValue: 800] [[
[2015-09-22T08:05:29.036+0200] [glassfish 4.1] [INFO] [NCLS-SECURITY-01115] [javax.enterprise.system.core.security] [tid: _ThreadID=14 _ThreadName=RunLevelControllerThread-1442901927955] [timeMillis: 1442901929036] [levelValue: 800] [[
Realm [certificate] of classtype [com.sun.enterprise.security.auth.realm.certificate.CertificateRealm] successfully created.]]
[2015-09-22T08:05:29.267+0200] [glassfish 4.1] [INFO] [SEC-SVCS-00100] [javax.enterprise.security.services] [tid: _ThreadID=14 _ThreadName=RunLevelControllerThread-1442901927955] [timeMillis: 1442901929267] [levelValue: 800] [[
Authorization Service has successfully initialized.]]
[2015-09-22T08:05:29.279+0200] [glassfish 4.1] [INFO] [NCLS-CORE-00087] [javax.enterprise.system.core] [tid: _ThreadID=17 _ThreadName=RunLevelControllerThread-1442901927967] [timeMillis: 1442901929279] [levelValue: 800] [[
Grizzly Framework 2.3.15 started in: 167ms - bound to [/0.0.0.0:8080]]]
[2015-09-22T08:05:29.430+0200] [glassfish 4.1] [INFO] [NCLS-CORE-00087] [javax.enterprise.system.core] [tid: _ThreadID=17 _ThreadName=RunLevelControllerThread-1442901927967] [timeMillis: 1442901929430] [levelValue: 800] [[
Grizzly Framework 2.3.15 started in: 2ms - bound to [/0.0.0.0:8181]]]
[2015-09-22T08:05:29.457+0200] [glassfish 4.1] [INFO] [NCLS-CORE-00087] [javax.enterprise.system.core] [tid: _ThreadID=17 _ThreadName=RunLevelControllerThread-1442901927967] [timeMillis: 1442901929457] [levelValue: 800] [[
Grizzly Framework 2.3.15 started in: 3ms - bound to [/0.0.0.0:4848]]]
[2015-09-22T08:05:29.588+0200] [glassfish 4.1] [INFO] [NCLS-CORE-00087] [javax.enterprise.system.core] [tid: _ThreadID=14 _ThreadName=RunLevelControllerThread-1442901927955] [timeMillis: 1442901929588] [levelValue: 800] [[
Grizzly Framework 2.3.15 started in: 1ms - bound to [/0.0.0.0:3700]]]
[2015-09-22T08:05:30.065+0200] [glassfish 4.1] [INFO] [AS-WEB-GLUE-00198] [javax.enterprise.web] [tid: _ThreadID=15 _ThreadName=RunLevelControllerThread-1442901927958] [timeMillis: 1442901930065] [levelValue: 800] [[
Created HTTP listener http-listener-1 on host/port
0.0.0.0:8080]]
[2015-09-22T08:05:30.074+0200] [glassfish 4.1] [INFO] [AS-WEB-GLUE-00198] [javax.enterprise.web] [tid: _ThreadID=15 _ThreadName=RunLevelControllerThread-1442901927958] [timeMillis: 1442901930074] [levelValue: 800] [[
Created HTTP listener http-listener-2 on host/port
0.0.0.0:8181]]
[2015-09-22T08:05:30.079+0200] [glassfish 4.1] [INFO] [AS-WEB-GLUE-00198] [javax.enterprise.web] [tid: _ThreadID=15 _ThreadName=RunLevelControllerThread-1442901927958] [timeMillis: 1442901930079] [levelValue: 800] [[
Created HTTP listener admin-listener on host/port
0.0.0.0:4848]]
[2015-09-22T08:05:30.082+0200] [glassfish 4.1] [INFO] [AS-WEB-GLUE-00198] [javax.enterprise.web] [tid: _ThreadID=15 _ThreadName=RunLevelControllerThread-1442901927958] [timeMillis: 1442901930082] [levelValue: 800] [[
Created HTTP listener jk-connector on host/port
0.0.0.0:8009]]
[2015-09-22T08:05:30.106+0200] [glassfish 4.1] [INFO] [AS-WEB-GLUE-00200] [javax.enterprise.web] [tid: _ThreadID=15 _ThreadName=RunLevelControllerThread-1442901927958] [timeMillis: 1442901930106] [levelValue: 800] [[
Created virtual server server]]
[2015-09-22T08:05:30.109+0200] [glassfish 4.1] [INFO] [AS-WEB-GLUE-00200] [javax.enterprise.web] [tid: _ThreadID=15 _ThreadName=RunLevelControllerThread-1442901927958] [timeMillis: 1442901930109] [levelValue: 800] [[
Created virtual server __asadmin]]
[2015-09-22T08:05:30.449+0200] [glassfish 4.1] [INFO] [AS-WEB-CORE-00306] [javax.enterprise.web.core] [tid: _ThreadID=15 _ThreadName=RunLevelControllerThread-1442901927958] [timeMillis: 1442901930449] [levelValue: 800] [[
Setting JAAS app name glassfish-web]]
[2015-09-22T08:05:30.449+0200] [glassfish 4.1] [INFO] [AS-WEB-GLUE-00201] [javax.enterprise.web] [tid: _ThreadID=15 _ThreadName=RunLevelControllerThread-1442901927958] [timeMillis: 1442901930449] [levelValue: 800] [[
Virtual server server loaded default web module ]]
[2015-09-22T08:05:30.818+0200] [glassfish 4.1] [INFO] [AS-CORE-JAVAEE-0002] [
javax.enterprise.system.core.ee] [tid: _ThreadID=15 _ThreadName=RunLevelControllerThread-1442901927958] [timeMillis: 1442901930818] [levelValue: 800] [[
Done with starting web container.]]
[2015-09-22T08:05:32.722+0200] [glassfish 4.1] [INFO] [] [javax.enterprise.system.tools.deployment.common] [tid: _ThreadID=16 _ThreadName=RunLevelControllerThread-1442901927967] [timeMillis: 1442901932722] [levelValue: 800] [[
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------