Zenoss 4.2.5 - ZEP error - java.lang.OutOfMemoryError: GC overhead limit exceeded

[Satay Epic]

Hello,

While trying to access the Event console UI,  I get a message "Error occurred". Looking at the ZEP log, I see this error

java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded. 

I suppose this would require to bump up  ZEP JVM heap size. Currently I don't see JVM heap size defined anywhere nor I see -Xmx argument  with the ZEP process.  So I don't know /sure about the current/default JVM heap size.

Wondering where one can define the heap size?. So far I found jetty.ini having reference of it. Should I define it there or somewhere else?

Searching online I came across a reference of "DEFAULT_ZEP_JVM_ARGS" variable defined in the "zenoss" .profile.  But I don''t know if it applies to 4.2.5 Core.

Before this issue surfaced, I've had issue where event console UI was very slow and I was planning to have ZEP re-create the indexes after I remove them. But didn't have opportunity to do so.  It should help to speed up the ZEP I think right ?

Please advise,

Thanks

[Jane Curry]

I think you will find Zenoss Jetty stuff (for 4.2.5) under /opt/zenoss/etc/zeneventserver/jetty - there's a start.config and a jetty.ini.  Not yet found any definitive examples to help you though.

There is one reference in the forum archive here - https://www.skills-1st.co.uk/techzen-forums-archive/forums/3/10/11.html but the links don't go anywhere directly useful these days.

Cheers,

Jane

[Satay Epic]

Thanks Jane.

I found the variable reference in the Wiki page ( now in the Web archive ). So setting up the -Xmx attribute is the way to increase the JVM heap size. 

Since I'm not sure if increasing the heap size will help sanely, I think clearing / closing the old outstanding events via api might be required.

By the way, were you able to or got a chance to save the Wiki web archive ?  Seems Wes Davis sent you a tarball of original Wiki earlier as well.

Thanks

[Jane Curry]

Sounds like you are making a little progress - do report back if you make -Xmx parameters work.

On the wiki - yup - Wes did send us the tarball but it needs a lot of cleaning up.  It is still a WIP but there's been a lot going on recently.  It is still on the list though.

Cheers,

Jane

[Satay Epic]

Ok the "Xmx" parameter worked with both ways i.e. the env variable and as defined in the jetty.ini file.  The only difference is that Jetty.ini option creates a child java process with "Xmx" parameter besides usual jetty parameters probabably because "-exec" option in there. 

[CODE]

 ps -aefl | grep zeneventserver
0 S zenoss    2967     1  0  80   0 - 909063 futex_ 22:29 pts/2   00:00:02 java -Djetty.host=localhost -server -XX:+HeapDumpOnOutOfMemoryError -DZENOSS_COMMAND=zeneventserver -DZENHOME=/usr/local/zenoss -Djetty.home=/usr/local/zenoss -Djetty.logs=/usr/local/zenoss/log -Dlogback.configurationFile=/usr/local/zenoss/etc/zeneventserver/logback.xml -DZENOSS_DAEMON=y -jar /usr/local/zenoss/lib/jetty-start-7.5.3.v20111011.jar --config=/usr/local/zenoss/etc/zeneventserver/jetty/start.config --ini=/usr/local/zenoss/etc/zeneventserver/jetty/jetty.ini --pre=etc/zeneventserver/jetty/jetty-logging.xml
0 S zenoss    2986  2967  8  80   0 - 1450637 futex_ 22:29 pts/2  00:00:42 java -Xmx3000m -Djetty.home=/usr/local/zenoss -cp /usr/local/zenoss/lib/jetty-xml-7.5.3.v20111011.jar:/usr/local/zenoss/lib/servlet-api-2.5.jar:/usr/local/zenoss/lib/jetty-http-7.5.3.v20111011.jar:/usr/local/zenoss/lib/jetty-continuation-7.5.3.v20111011.jar:/usr/local/zenoss/lib/jetty-server-7.5.3.v20111011.jar:/usr/local/zenoss/lib/jetty-security-7.5.3.v20111011.jar:/usr/local/zenoss/lib/jetty-servlet-7.5.3.v20111011.jar:/usr/local/zenoss/lib/jetty-webapp-7.5.3.v20111011.jar:/usr/local/zenoss/lib/jetty-deploy-7.5.3.v20111011.jar:/usr/local/zenoss/lib/jetty-util-7.5.3.v20111011.jar:/usr/local/zenoss/lib/jetty-io-7.5.3.v20111011.jar org.eclipse.jetty.xml.XmlConfiguration /tmp/start3640366378706439536.properties /usr/local/zenoss/etc/zeneventserver/jetty/jetty-logging.xml /usr/local/zenoss/etc/zeneventserver/jetty/jetty.xml /usr/local/zenoss/etc/zeneventserver/jetty/jetty-deploy.xml /usr/local/zenoss/etc/zeneventserver/jetty/jetty-webapps.xml

ps -aefl | grep zeneventserver
0 S zenoss     960     1 90  80   0 - 1448838 futex_ 22:08 pts/2  00:00:33 java -server -Xmx3000m -XX:+HeapDumpOnOutOfMemoryError -DZENOSS_COMMAND=zeneventserver -DZENHOME=/usr/local/zenoss -Djetty.home=/usr/local/zenoss -Djetty.logs=/usr/local/zenoss/log -Dlogback.configurationFile=/usr/local/zenoss/etc/zeneventserver/logback.xml -DZENOSS_DAEMON=y -jar /usr/local/zenoss/lib/jetty-start-7.5.3.v20111011.jar --config=/usr/local/zenoss/etc/zeneventserver/jetty/start.config --ini=/usr/local/zenoss/etc/zeneventserver/jetty/jetty.ini --pre=etc/zeneventserver/jetty/jetty-logging.xml

[/CODE]