Frequent FULL gc's and then heap crash, why?

[kwia...@gmail.com]

Hello,

I am running weblogic with jdk 1.7_0_25. We had been running an application for YEARS on jdk 1.6_0_xx without issue. When we migrated to jdk 1.7 we started having issues with GC.... Server works fine for several days, I see normal gc's, but then all of a sudden it goes in a continuous FULL GC cycle and will not stop... ideas??

416355.858: [GC [PSYoungGen: 620307K->57440K(620736K)] 1991153K->1428285K(2018880K), 0.0900130 secs] [Times: user=0.36 sys=0.00, real=0.11 secs]
416392.757: [GC [PSYoungGen: 605024K->62587K(610176K)] 1975869K->1433432K(2008320K), 0.0565270 secs] [Times: user=0.41 sys=0.00, real=0.05 secs]
416463.418: [GC [PSYoungGen: 610171K->51415K(620672K)] 1981016K->1430601K(2018816K), 0.0516100 secs] [Times: user=0.37 sys=0.00, real=0.05 secs]
416736.929: [GC [PSYoungGen: 592535K->50944K(619456K)] 1971721K->1432341K(2017600K), 0.0512740 secs] [Times: user=0.36 sys=0.00, real=0.05 secs]
416865.440: [GC [PSYoungGen: 592064K->51736K(623232K)] 1973461K->1435101K(2021376K), 0.0505060 secs] [Times: user=0.33 sys=0.01, real=0.05 secs]
417129.884: [GC [PSYoungGen: 598104K->51949K(622144K)] 1981469K->1436658K(2020288K), 0.0504380 secs] [Times: user=0.33 sys=0.00, real=0.05 secs]
417322.683: [GC [PSYoungGen: 598317K->54181K(624576K)] 1983026K->1440818K(2022720K), 0.0506910 secs] [Times: user=0.36 sys=0.00, real=0.05 secs]
417597.562: [GC [PSYoungGen: 604069K->52297K(624320K)] 1990706K->1444571K(2022464K), 0.0752950 secs] [Times: user=0.35 sys=0.00, real=0.07 secs]
417597.638: [Full GC [PSYoungGen: 52297K->1022K(624320K)] [ParOldGen: 1392273K->1398143K(1398144K)] 1444571K->1399166K(2022464K) [PSPermGen: 162096K->161853K(199808K)], 2.3327670 secs] [Times: user=15.54 sys=0.03, real=2.33 secs]
417743.769: [Full GC [PSYoungGen: 550910K->11213K(624320K)] [ParOldGen: 1398143K->1396146K(1398144K)] 1949054K->1407360K(2022464K) [PSPermGen: 161853K->161853K(194496K)], 2.7090850 secs] [Times: user=18.77 sys=0.06, real=2.71 secs]
418023.140: [Full GC [PSYoungGen: 549888K->9270K(624320K)] [ParOldGen: 1396146K->1397934K(1398144K)] 1946034K->1407205K(2022464K) [PSPermGen: 161853K->161853K(189248K)], 2.8707310 secs] [Times: user=19.94 sys=0.08, real=2.87 secs]
418342.476: [Full GC [PSYoungGen: 549888K->2929K(624320K)] [ParOldGen: 1397934K->1398143K(1398144K)] 1947822K->1401072K(2022464K) [PSPermGen: 161894K->161835K(184960K)], 2.7683850 secs] [Times: user=18.73 sys=0.05, real=2.80 secs]
.
.
.

Exception in thread "DoSManager"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DoSManager"
Exception in thread "Thread-13" java.lang.OutOfMemoryError: Java heap space

<Dec 22, 2013 5:42:07 AM EST> <Error> <Socket> <BEA-000405> <Uncaught Throwable in processSockets
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
<Dec 22, 2013 5:42:57 AM EST> <Error> <Server> <BEA-002608> <The ListenThread failed because of an error: java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
at weblogic.utils.io.Chunk.<init>(Chunk.java:348)
at weblogic.utils.io.Chunk.getChunk(Chunk.java:183)
at weblogic.socket.AbstractMuxableSocket.<init>(AbstractMuxableSocket.java:117)
at weblogic.socket.MuxableSocketDiscriminator.<init>(MuxableSocketDiscriminator.java:51)
at weblogic.server.channels.DynamicListenThread.registerSocket(DynamicListenThread.java:389)
Truncated. see log file for complete stacktrace
>

[kwia...@gmail.com]

Here is the last GC log entries:

606667.298: [Full GC [PSYoungGen: 549888K->540087K(624320K)] [ParOldGen: 1398140K->1398140K(1398144K)] 1948028K->1938228K(2022464K) [PSPermGen: 160980K->160980K(161280K)], 1.8969770 secs] [Times: user=12.32 sys=0.00, real=1.90 secs]
606710.698: [Full GC [PSYoungGen: 549888K->540087K(624320K)] [ParOldGen: 1398140K->1398140K(1398144K)] 1948028K->1938228K(2022464K) [PSPermGen: 160980K->160980K(161280K)], 1.8543470 secs] [Times: user=12.22 sys=0.01, real=1.86 secs]
606754.098: [Full GC [PSYoungGen: 549888K->540087K(624320K)] [ParOldGen: 1398140K->1398140K(1398144K)] 1948028K->1938228K(2022464K) [PSPermGen: 160980K->160980K(161280K)], 1.8512890 secs] [Times: user=12.26 sys=0.00, real=1.86 secs]

[markspace]

On 12/23/2013 5:41 AM, kwia...@gmail.com wrote:
> Hello,
>
> I am running weblogic with jdk 1.7_0_25. We had been running an
> application for YEARS on jdk 1.6_0_xx without issue. When we migrated
> to jdk 1.7 we started having issues with GC.... Server works fine for
> several days, I see normal gc's, but then all of a sudden it goes in
> a continuous FULL GC cycle and will not stop... ideas??

I'm no expert at reading GC logs, but I'll give it a shot:

>417597.638:

> [Full GC [PSYoungGen: 52297K->1022K(624320K)]

609 MB total

> [ParOldGen: 1392273K->1398143K(1398144K)] 1444571K->1399166K(2022464K)

1975 MB total

> [PSPermGen: 162096K->161853K(199808K)], 2.3327670 secs]

195 MB total

> [Times: user=15.54 sys=0.03, real=2.33 secs]

I think these number look a bit low for a "real" app. Are you running
64 or 32 bit Java? Can you verify that the new JVM is getting the
correct memory parameters when your app starts? Can you spit those
parameters out here on this list so we can see them? Are there any
other GC or memory parameters the VM starts with that we should know about?

Otherwise it looks normal. That is, you should expect to see the GC
thrash itself just before you run out of memory, and that's what's
happening. The problem is the OOM, not the thrashing.

[Roedy Green]

On Mon, 23 Dec 2013 05:41:32 -0800 (PST), kwia...@gmail.com wrote,
quoted or indirectly quoted someone who said :

>I am running weblogic with jdk 1.7_0_25. We had been running an application=
> for YEARS on jdk 1.6_0_xx without issue. When we migrated to jdk 1.7 we st=
>arted having issues with GC.... Server works fine for several days, I see n=
>ormal gc's, but then all of a sudden it goes in a continuous FULL GC cycle =

>and will not stop... ideas??

There are java.exe command line settings for the size of the virtual
address space. If it is too small, you fill it almost to the brim, and
it will go nuts running out of space over and over having to redo the
GC.

See http://mindprod.com/jgloss/javaexe.html

--
Roedy Green Canadian Mind Products http://mindprod.com
Unlike many machines, computers require no water once they are
manufactured.

[John B. Matthews]

On 12/28/13, 5:27 AM, Roedy Green wrote:
> On Mon, 23 Dec 2013 05:41:32 -0800 (PST), kwia...@gmail.com wrote,
> quoted or indirectly quoted someone who said :
>
>> I am running weblogic with jdk 1.7_0_25. We had been running an application=
>> for YEARS on jdk 1.6_0_xx without issue. When we migrated to jdk 1.7 we st=
>> arted having issues with GC.... Server works fine for several days, I see n=
>> ormal gc's, but then all of a sudden it goes in a continuous FULL GC cycle =
>> and will not stop... ideas??
>
> There are java.exe command line settings for the size of the virtual
> address space. If it is too small, you fill it almost to the brim, and
> it will go nuts running out of space over and over having to redo the
> GC.
>
> See http://mindprod.com/jgloss/javaexe.html

Some related examples are examined here:

<http://stackoverflow.com/a/15307253/230513>

--
John B. Matthews
trashgod at gmail dot com
<http://sites.google.com/site/drjohnbmatthews>