Islandora 7.3: java.lang.OutOfMemoryError: PermGen space

[A L]

Hi,

I have searched in this forum and various other places to see if there was any solution that could fix this problem I have with Fedora Commons 3.5 with Islandora 7.3. I got the following exception (catalina.out from $FEDORA_HOME/tomcat/logs):

Exception in thread "ActiveMQ Session Task-2" java.lang.OutOfMemoryError: PermGen space
              at java.lang.ClassLoader.defineClass1(Native Method)
              at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
              ....

I tried increasing the permgen space by setting the following in catalina.sh.

export JAVA_OPTS="-Xms1024m -Xmx10246m -XX:NewSize=256m -XX:MaxNewSize=356m -XX:PermSize=256m -XX:MaxPermSize=356m"

The error did disappear for a while but it showed up again. I think there is a something more critical to it and possibly a memory leak in Fedora Commons 3.5. Any suggestions please?

thanks for any help!

[Ernie Gillis]

So I don't create too many threads... I thought I had posted in here in the Islandora group about my very similar issue, but I can't find it.

In any case, I also have a memory leak issue, that *seems* to be related to JMS (which I know is related to ActiveMQ). So I'm not posting all kinds of lines, you can check out the fedora group post I made [1].

At this point, I have found ways to kill the process without completely restarting the box. Early in the debugging, "sudo kill [PID]" didn't fully kill the JVM, so I had to restart the box. If I did "sudo -s" to do a shell change to root within my session (if I'm getting the flow right), I am able to successfully kill the java PID and restart fedora with the bundled tomcat / catalina.

[1] https://groups.google.com/forum/#!topic/fedora-community/2P_s4QaUh24

[A L]

Hi Ernie!

So you mean when you made sure you killed the process correctly and then restarted tomcat that solved your out of memory issue?

Laxmi

[Ernie Gillis]

I changed a few things. When the memory leak occurs, I wasn't able to successfully kill the process with just "sudo kill [PID]". It forced me to have to do a full restart of the whole box, which is obviously far from ideal.  When I came across the "sudo -s" command to do a shell switch an (I think) spoof as root (as opposed to running "sudo kill [PID}), and *then* did just a "kill [PID]" (since I'm virtually root), the tomcat process died properly. At this point, I didn't have to do a full box restart.

Since I've also updated how my environment variables are set, it *may* solve some other issues. So, that's why I'm waiting to see if the memory leak happens again. I do need to try and trigger a lot of REST traffic through Islandora, since that is what creates the memory issues than if I'm just in fedora's web admin interface. And by a lot of REST traffic, I mean, less than 20 or 25 clicks... or could be number of clicks per second (again... hard to trace / track)

--
For more information about using this group, please read our Listserv Guidelines: http://islandora.ca/content/welcome-islandora-listserv
---
You received this message because you are subscribed to a topic in the Google Groups "islandora" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/islandora/YjU696EikYU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to islandora+...@googlegroups.com.
Visit this group at http://groups.google.com/group/islandora.
For more options, visit https://groups.google.com/d/optout.

--
Ernest Gillis
Manager of Learning Resources Web Development
Stan Getz Media Center and Library
Phone: 617-747-8533
E-Mail: egi...@berklee.edu

[A L]

Hi Ernie!

I figured out a fix. I am not sure if it would really last long but I am pretty sure the default perm gen space that is specified with Tomcat is too small. So I created a file named "setenv.sh" with the following line and then I added that in $CATALINA_HOME/bin/ folder and restarted tomcat. It seems to be smooth so far

export JAVA_OPTS="-Xms1024m -Xmx10246m -XX:NewSize=256m -XX:MaxNewSize=356m -XX:PermSize=256m -XX:MaxPermSize=356m"

Let me know if you have questions.

[Ernie Gillis]

Thanks A L!

I had most of this, but didn't have "NewSize" or "MaxNewSize". Added it in... fingers crossed :)

[Ernie Gillis]

Aaannd... nope :( Still crashes. 

It seems to be based on the number of transactions between Islandora and Fedora. So, if I open 10 (completely random choice) collections in their own windows. While they're loading, it could crash Fedora.

[A L]

Sorry to hear that, Ernie! Did you make sure tomcat was restarted properly when you last made some modification to the JAVA_OPTS? Also, have you tried the fix I told you to put everything in setenv.sh file?

[Ernie Gillis]

I did make separate setenv.sh, and did a full box restart to do a clear of any resources that may have needed flushing.

Sent from mobile device. Please ignore brevity & typos.

--

[Ernie Gillis]

New "symptom"... Don't think I noticed this before, but I was moreso trying to kill processes when stuff crashes.

When I do a "$CATALINA_BASE/bin/shutdown.sh", I get the following java error:

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=256m; support was removed in 8.0

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=356m; support was removed in 8.0

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=256m; support was removed in 8.0

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=356m; support was removed in 8.0

If I do a "java -version", I get..

]# java -version

java version "1.8.0_05"

Java(TM) SE Runtime Environment (build 1.8.0_05-b13)

Java HotSpot(TM) 64-Bit Server VM (build 25.5-b02, mixed mode)

I found out that Oracle changed the memory management for Java 8. "Perm" memory ("Permanent Generation") structure to now be called "Metaspace" memory for Java 8 [1]. So I looked up the proper syntax on how to reference "Metaspace" in JAVA_OPTS, and found it literally replaces the word "Perm" with the word "Metaspace" [2]. But it didn't help the crashing.

A while back, I found a tutorial about tomcat OutOfMemoryError [3], but never posted. I will check in the jar files (which may be fgsearch or some jms thing, based on how I've interpreted logs and symptoms). It does say that the "PermSize" (or in my case "MetaspaceSize") are really temporary fixes and you'll eventually get the OutOfMemoryError again. I originally didn't want to be moving things around outside the expected install mentioned in the Islandora or Fedora tutorials. Will give that a shot this week and see what happens.

[1] : http://javaeesupportpatterns.blogspot.com/2013/02/java-8-from-permgen-to-metaspace.html

[2] : https://plumbr.eu/outofmemoryerror/metaspace

[2] : http://javarevisited.blogspot.com/2012/01/tomcat-javalangoutofmemoryerror-permgen.html

[A L]

Ernie,

Is there any reason you are using Java 8? I have Oracle Java 7. I am not sure if Java 8 is recommended for Fedora 3.7.1. I don't remember seeing it anywhere. That could reason why it just ignores your new permgen values.

[Ernie Gillis]

Java has had a lot of news lately about security flaws, so I wanted to go with what would be considered the most buttoned up one. If I should roll it back, I can, but am trying to keep the box secure since it will be public facing.

[Peter Murray]

It is useful to keep in mind that the public internet does not need to access the Tomcat server directly.  All accesses to it are proxied through Apache (in the case of Djatoka) or through Drupal/Islandora (datastreams, SOLR).  So one can firewall public internet access to the Tomcat server.

Peter

On Jul 16, 2014, at 11:48 AM, Ernie Gillis <egi...@berklee.edu> wrote:

Java has had a lot of news lately about security flaws, so I wanted to go with what would be considered the most buttoned up one. If I should roll it back, I can, but am trying to keep the box secure since it will be public facing.

--
For more information about using this group, please read our Listserv Guidelines: http://islandora.ca/content/welcome-islandora-listserv

--
Peter Murray
Assistant Director, Technology Services Development
LYRASIS
Peter....@lyrasis.org
+1 678-235-2955
800.999.8558 x2955

[Ernie Gillis]

Yup! Quite aware of that, but we also do security stress tests on things, and java does still leave security holes on it (even with proper firewalls in place). I'm just preferring to be more safe than sorry.

Would the garbage collection memory leak issue be linked to my version of Java, though?

For the actual maintenance / tests I'm looking to do, I also have questions for anyone who may know.

In the blog post I linked to previously [1], it mentions possible issues to look at the "JDBC Drivers" and the "Logging Framework". I know that "log4j" has its idiosyncrasies, so I'm kind of leading more toward that (especially since the crashing seems so linked to the traffic done via REST with Tuque & FedoraGSearch). 

My questions are, though:

- If it is a JDBC issue...and I were to transfer the "driver's jar into tomcat lib instead of bundling it on web application's war file" (as mentioned in the blog post), where would I look for, say, FedoraGSearch's use of driver jars? "webapps/fedoragsearch/WEB-INF/lib"? And which would I move? All of them?

- If it were a log4j issue.... I expect I could (as I ask above) just move all jar files from "webapps/fedoragsearch/WEB-INF/lib"... or I could just move any "log4" files to "[CATALINA_HOME]/lib". Does this sound accurate?

I could be just grasping at straws, but it's both helping me to learn and helping me to fix the problem :)

[1] : http://javarevisited.blogspot.com/2012/01/tomcat-javalangoutofmemoryerror-permgen.html

[A L]

Hi Ernie!

I remember seeing that blogspot and I am pretty sure I tried those two suggestions. When I moved the mysql jar (as that is what I am using for fedora), Fedora freaked out when I restarted it. So, that was not helpful at all. I tried the second one as well "log4j" which didn't make any difference so I rolled it back to how it was.

But I didn't see the issue of outofmemory permgen since the time I posted you that information on how I set some variables in setenv.sh. I am not sure if I will see it in future but things are okay now for me. I am using Oracle Java 7 though.

I think it is recommended for you to check with Fedora Community about using Java 8 for Fedora 3.7.1. In the past, when I was working with one of older version of Fedora and tried to play with Java 7, it didn't like it so I HAD to use Java 6 for some reason. So, please check with Fedora Community about using Java 8 for 3.7.1.

Good luck!

[Ernie Gillis]

I confirmed that Java SE 8 was not the root of the problem. I rolled back to Java SE 7, but it still crashes

INFO: Server startup in 22677 ms

Jul 17, 2014 4:26:39 PM org.apache.jasper.runtime.JspFactoryImpl internalGetPageContext

SEVERE: Exception initializing page context

java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "ActiveMQ Journal Checkpoint Worker" java.lang.OutOfMemoryError: GC overhead limit exceeded

Jul 17, 2014 4:26:59 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run

SEVERE: Socket accept failed

java.lang.OutOfMemoryError: GC overhead limit exceeded

Jul 17, 2014 4:27:11 PM org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor processChildren

SEVERE: Exception invoking periodic operation: 

java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "http-8443-Acceptor-0" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: GC overhead limit exceeded

Jul 17, 2014 4:27:41 PM org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor processChildren

SEVERE: Exception invoking periodic operation: 

java.lang.OutOfMemoryError: GC overhead limit exceeded

[A L]

Ernie,

Have you made sure you updated path variables correctly when you rolled back?

[A L]

Any progress?

[Ernie Gillis]

I posted more info in the fedora group since it is definitely a fedora crash, and gave more tomcat / fedora info and screenshots:

https://groups.google.com/d/msg/fedora-community/2P_s4QaUh24/2-O0jQosBNsJ

But... no successful keeping the server alive yet. It is also QUITE slow for me (compared to other sites I've used that are Islandora / Fedora), which is possibly related to the whole thing :(

[Ernie Gillis]

Quick update, is kind of a non update, but more to just keep it documented.

My workload shifted me away from this. It still happens, but we aren't officially live, yet, so it isn't as urgent (at the moment).

My next test steps are going to be:
- see if I can replicate the memory issue purely through the ri interface (without using Islandora or tuque)

If it still happens, I will know it's more fedora than tuque. If it doesn't crash, I will know it's more tuque.

Onwards and upwards!

[Ernie Gillis]

I did some more testing, and I did find out that the "RISearch" interface had far better garbage collection than through the Islandora Tuque (no idea why). I also found some more JAVA_OPTS values that appears to have greatly reduced the chance of crashing.

So... Now... My JAVA_OPTS includes:

-Xms1524m -Xmx1524m -XX:NewSize=256m -XX:MaxNewSize=512m -XX:PermSize=256m -XX:MaxPermSize=512m -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+UseConcMarkSweepGC

I still can invoke a crash, but the garbage collection gets trigger more frequently. Also, setting my Max Size values (MaxNewSize & MaxPermSize) to 512 seems to have allow all the libraries to load properly (which I now know is what triggers the Heap Memory error). This is hopefully a solution, but time will tell :)

Thanks to all who gave assistance! :)

[A L]

I am glad you figured out a solution! Sorry, I couldn't help much. Thanks for sharing your fixes.

[Ernie Gillis]

Had a false positive, and still had crash :(

BUT! I think I finally really fixed it! I posted the possible solution in the fedora group: https://groups.google.com/d/msg/fedora-community/2P_s4QaUh24/1dt4vNuYSCAJ

Here's to hoping it says up! :)