Out of memory problem
John Smith
We have some java applications deployed on tomcat.
From time to time the whole system freezes up.
It seems that there is not enough space for JVM. The maximum is set to
896MB.
Instead of just increasing the size I'd like to know whats causing the
problem.
I'm trying to use "-XX:-PrintClassHistogram" parameter but the log file
is not generated.
I put this to tomcat5 config file:
JAVA_OPTS="-Xms512m -Xmx896m -XX:MaxPermSize=256m
-XX:HeapDumpPath=/tmp/java_pid<pid>.hprof
-XX:-HeapDumpOnOutOfMemoryError -XX:-PrintClassHistogram"
I tried stoping the tomcat afterwords or causing the OutOfMemory
problem, but the file is not created.
I'm I doing wrong something?
Any suggestions?
Marcin Konopka
[...]
> JAVA_OPTS="-Xms512m -Xmx896m -XX:MaxPermSize=256m
> -XX:HeapDumpPath=/tmp/java_pid<pid>.hprof
-XX:HeapDumpPath=/tmp
> -XX:-HeapDumpOnOutOfMemoryError -XX:-PrintClassHistogram"
-XX:+HeapDumpOnOutOfMemoryError -XX:+PrintClassHistogram
You can also set these options on running jvm with jinfo;
similarly You can see class histogram or generate heap dump
with jmap.
Best regards
MK
John Smith
> On 2010-09-02, John Smith<jsm...@yahoo.com> wrote:
> [...]
>> JAVA_OPTS="-Xms512m -Xmx896m -XX:MaxPermSize=256m
>> -XX:HeapDumpPath=/tmp/java_pid<pid>.hprof
>
> -XX:HeapDumpPath=/tmp
>
>> -XX:-HeapDumpOnOutOfMemoryError -XX:-PrintClassHistogram"
>
> -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintClassHistogram
>
Thank you very much. I still have a couple of questions.
When does the above generate a file? Upon OutOfMemory error or on
process termination?
> You can also set these options on running jvm with jinfo;
> similarly You can see class histogram or generate heap dump
> with jmap.
>
> Best regards
> MK
I used jmap like jmap -histo -F pid and get something like this:
num #instances #bytes Class description
--------------------------------------------------------------------------
1: 183266 22430576 * ConstMethodKlass
2: 183266 14666160 * MethodKlass
3: 115336 10325256 char[]
4: 19461 10191176 * ConstantPoolKlass
5: 19461 8299040 * InstanceKlassKlass
This is just after server restart, so I suppose I should just wait some
time and maybe if there is a leak it will show up at the begining?
Any more recommendations?
Lew
>> -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintClassHistogram
John Smith wrote:
> Thank you very much. I still have a couple of questions.
> When does the above generate a file? Upon OutOfMemory error [sic] or on
> process termination?
It's called "HeapDumpOn..."?
The process does terminate, of course.
--
Lew
John Smith
Ok, but I'm interested about PrintClassHistogram.
I'm not sure it this works with the same trigger (OnOutOfMemoryError)
and if the destination file/directory is the samo?
Lew
>>> Thank you very much. I still have a couple of questions.
>>> When does the above generate a file? Upon OutOfMemory error [sic] or on
>>> process termination?
Lew wrote:
>> It's called "HeapDumpOn..."?
>>
>> The process does terminate, of course.
John Smith wrote:
> Ok, but I'm interested about PrintClassHistogram.
> I'm not sure it this works with the same trigger (OnOutOfMemoryError)
> and if the destination file/directory is the samo?
<http://blogs.sun.com/watt/resource/jvm-options-list.html>
"-XX:+PrintClassHistogram ... Prints the all the java [sic] heap objects,
their instance count [sic] and total space they occupy in the heap. The only
downside is that you need to issue a SIGQUIT (see -Xsqnopause) [sic] which
will leave the app running but will dump all of this data to stdout. Very
useful to assist in identifying memory problems [sic] for example on a
production platform where an [sic] CPU intensive [sic] profiler cannot be used."
--
Lew
John Smith
I tried before with kill -QUIT pid but it does not generate anything.
Marcin Konopka
>> <http://blogs.sun.com/watt/resource/jvm-options-list.html>
>> "-XX:+PrintClassHistogram ... Prints the all the java [sic] heap
>> objects, their instance count [sic] and total space they occupy in the
>> heap. The only downside is that you need to issue a SIGQUIT (see
>> -Xsqnopause) [sic] which will leave the app running but will dump all of
>> this data to stdout. Very useful to assist in identifying memory
>> problems [sic] for example on a production platform where an [sic] CPU
>> intensive [sic] profiler cannot be used."
>>
>
> I tried before with kill -QUIT pid but it does not generate anything.
Dumps of stacktraces (and class histograms in Your case) are printed to
stdout so, with default tomcat's configuration, You'll find them in
catalina.out.
MK
Marcin Konopka
>
> num #instances #bytes Class description
> --------------------------------------------------------------------------
> 1: 183266 22430576 * ConstMethodKlass
> 2: 183266 14666160 * MethodKlass
> 3: 115336 10325256 char[]
> 4: 19461 10191176 * ConstantPoolKlass
> 5: 19461 8299040 * InstanceKlassKlass
>
>
> This is just after server restart, so I suppose I should just wait some
> time and maybe if there is a leak it will show up at the begining?
These are internal jvm objects used, for stuff like class'es bytecode,
string values etc. Yes - You can wait some time and see which classes
grows most. However You should be focused on Your application
classes (if You suspect that it leaks memory), rather than jvm's
internal objects.
> Any more recommendations?
You wrote that You're experiencing freezes - but did You get
OutOfMemoryError? If You have freeze, but no OOM then You can try other
garbage collectors (ex. Throughput Collector - -XX:+UseParallelGC).
It may be also the case, that Your problem lies in PermGen, not the
heap. To check this You can use jstat. Start jvm with
-XX:MaxPermSize the same as -XX:PermSize and -Xms same as -Xmx and
run jstat -gcutil with some interval. Start using application, and when
freeze occurs, take a look on jstat output. If "P" column is about 100%
then PermGen is Your bottleneck. If it's "O" column then it's heap.
regards
MK
John Smith
I think the PermGen is not the problem, because there is no error in the
Catalina.out. The funny thing is there is no OutOfMemory error at all. I
figured it out by checking the system and proces memory.
John Smith
So I should look only at Catalina.out? There is no easy way to redirect
these 2 to some other output?
Lew
>>>> <http://blogs.sun.com/watt/resource/jvm-options-list.html>
>>>> "-XX:+PrintClassHistogram ... Prints the all the java [sic] heap
>>>> objects, their instance count [sic] and total space they occupy in the
>>>> heap. The only downside is that you need to issue a SIGQUIT (see
>>>> -Xsqnopause) [sic] which will leave the app running but will dump
>>>> all of
>>>> this data to stdout. Very useful to assist in identifying memory
>>>> problems [sic] for example on a production platform where an [sic] CPU
>>>> intensive [sic] profiler cannot be used."
but did write:
>>> I tried before with kill -QUIT pid but it does not generate anything.
Marcin Konopka wrote:
>> Dumps of stacktraces (and class histograms in Your case) are printed to
>> stdout so, with default tomcat's configuration, You'll find them in
>> catalina.out.
John Smith wrote:
> So I should look only at Catalina.out?
No, you should look at catalina.out. You might find the log files (pattern
like "catalina.2010-08-01.log") helpful.
> There is no easy way to redirect these 2 to some other output?
I don't think there is, but catalina.out is an easy way itself.
--
Lew
John Smith
I'm struggling with this jmap not working on the application server.
I tried 2 different java versions and I get this exception:
jmap -histo -F 802
Attaching to process ID 802, please wait...
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at sun.tools.jmap.JMap.runTool(JMap.java:196)
at sun.tools.jmap.JMap.main(JMap.java:128)
Caused by: sun.jvm.hotspot.runtime.VMVersionMismatchException: Supported
versions are 1.6.0-b09. Target VM is 17.0-b16
at sun.jvm.hotspot.runtime.VM.checkVMVersion(VM.java:217)
at sun.jvm.hotspot.runtime.VM.<init>(VM.java:280)
at sun.jvm.hotspot.runtime.VM.initialize(VM.java:333)
at
sun.jvm.hotspot.bugspot.BugSpotAgent.setupVM(BugSpotAgent.java:594)
at sun.jvm.hotspot.bugspot.BugSpotAgent.go(BugSpotAgent.java:494)
at
sun.jvm.hotspot.bugspot.BugSpotAgent.attach(BugSpotAgent.java:332)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:163)
at
sun.jvm.hotspot.tools.ObjectHistogram.main(ObjectHistogram.java:56)
... 6 more
2. Another thing is if I manage to find a problematic class, how do I
know where is the leak? Do I have to manually search it or can I use
some other tool?
Luuk
> Caused by: sun.jvm.hotspot.runtime.VMVersionMismatchException: Supported
> versions are 1.6.0-b09. Target VM is 17.0-b16
wrong version......?
--
Luuk
John Smith
It seems strange.
What version is supported than??
Machine which works:
java -version
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Java HotSpot(TM) Server VM (build 16.3-b01, mixed mode)
Machine which doesn't work:
java -version
java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
Java HotSpot(TM) Server VM (build 17.0-b16, mixed mode)
The version before that was 1.6.0.18 I think.
John Smith
Sorry for the last post. It seems that my jre version is different than
jdk which has this jmap utility.
John Smith
I experienced the freeze again, and I captured manually the logs with jmap.
It seems the memory allocation is ok. Java uses around 500MB.
Minimum set is 512MB, maximum 1024MB, and MaxPermSize is set to 512MB.
This options (-XX:+HeapDumpOnOutOfMemoryError -XX:+PrintClassHistogram)
did not work. Probably because OutOfMemory did not happen.
Anyway I changed the minimum memory to 1024MB also, and set
-XX:+UseParallelGC.
I read about older java versions and garbage collector freezes if there
is a big difference between minimum and maximum amount of memory set.
Are there any other ideas?
Kevin McMurtrie
John Smith <jsm...@yahoo.com> wrote:
Try different versions of the JVM. As far as I know, all versions of
Sun Java 1.5 have GC bugs that may cause long stalls, JVM crashes, and
logically impossible NPEs. (I've seen 1.5 HotSpot unroll loops
incorrectly too.) Recent builds of Sun Java 1.6 seem to work much more
reliably.
Also check that you don't have any kernel bugs related to the high
resolution system clock. If GC timestamps go to 0.00, you have the bug
and GC will auto-tune itself to oblivion.
--
I won't see Google Groups replies because I must filter them as spam
John Smith
I tried the latest version of java 1.6.0_21, but it's still the same.
Can you give me more information what to do exactly with the GC?
John Smith
Now I have set JAVA_OPTS to this:
JAVA_OPTS="-Xms1024m -Xmx1024m -XX:MaxPermSize=256m -verbose:gc
-XX:+PrintGCTimeStamps -XX:+PrintGCDetails
-Xloggc:/var/log/tomcat5/gc_logs/gc.log"
I suppose this is enough?
Kevin McMurtrie
John Smith <jsm...@yahoo.com> wrote:
Look up the options for verbose GC modes. You'll see exactly which
phase is stalling.
Arne Vajhøj
> Try different versions of the JVM. As far as I know, all versions of
> Sun Java 1.5 have GC bugs that may cause long stalls, JVM crashes, and
> logically impossible NPEs.
Do you have a link to the bug reports?
Arne