Gerrit Hangs with Java Heap Exhausted

[Rich Schmitt]

I'm hoping someone can give me some pointers with what I should look at to resolve this.

 

About once a day gerrit hangs.  The gerrit process is consuming close to 100% of the cpu.

 

I've tried to do a gerrit show-queue but that hangs as well.

 

Looking through the error log, I see a lot of java heap exhausted messages.  I'm assuming this is the cause.   I can restart gerrit and things recover.

 

I am using java openjdk 1.6

 

Does anyone have any experience with this?  Any pointers?

 

Thanks,

Rich

[Matthias Sohn]

create some thread dumps to see which threads are hanging where,

use jps to find your java process and then use jstack to create the

thread dumps

--

Matthias 

[Ted]

may seem like a rhetorical item but... have you checked your heap size?

> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups
> "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to repo-discuss...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>


--
Ted.

[Rich Schmitt]

Gerrit hung once again so I was able to obtain the suggested jstack trace.  I've attached the trace as a file.  Does this look familiar to anyone?  I'll appreciate any suggestions.

Thanks,

Rich

[Lundh, Gustaf]

Please use a tool like visualvm and verify that you are not running out of heap. Just so we can be 100% sure this is not the case. Also take a look at the GC-graph. How much is the GC working when Gerrit starts to slow down/hangs?

 

Are you using the default GarbageCollector or have you tweaked the java launch settings?

 

What “java -xmx” value are you running with?

 

Would you mind adding the java flag “-XX:+PrintCommandLineFlags” to your Gerrit launch script (or JAVA_OPTS if that’s what you are using) and post the output?

 

It really sounds like you are running out if heap and the GC is working like crazy….

 

/Gustaf

--

[Rich Schmitt]

This has happened again.

I've been able to capture output from jmap (below).  It does appear that gerrit is running out of heap, but it also seems that we have a 1GB heap which I hope would be plenty.  This info probably doesn't do anything more then confirm that it's a heap problem.

I start gerrit by using the gerrit.sh script so I'm not adding any special arguments to the command.  I will try some of the other suggestions (visualvm, various gc debug options, different vm).   Any new suggestions, let me know.

Rich

Heap Configuration:

   MinHeapFreeRatio = 40

   MaxHeapFreeRatio = 70

   MaxHeapSize      = 1002438656 (956.0MB)

   NewSize          = 1310720 (1.25MB)

   MaxNewSize       = 17592186044415 MB

   OldSize          = 5439488 (5.1875MB)

   NewRatio         = 2

   SurvivorRatio    = 8

   PermSize         = 21757952 (20.75MB)

   MaxPermSize      = 174063616 (166.0MB)

Heap Usage:

PS Young Generation

Eden Space:

   capacity = 134479872 (128.25MB)

   used     = 134478064 (128.24827575683594MB)

   free     = 1808 (0.0017242431640625MB)

   99.99865556088572% used

From Space:

   capacity = 65601536 (62.5625MB)

   used     = 0 (0.0MB)

   free     = 65601536 (62.5625MB)

   0.0% used

To Space:

   capacity = 98959360 (94.375MB)

   used     = 0 (0.0MB)

   free     = 98959360 (94.375MB)

   0.0% used

PS Old Generation

   capacity = 668336128 (637.375MB)

   used     = 668336048 (637.3749237060547MB)

   free     = 80 (7.62939453125E-5MB)

   99.99998802997524% used

PS Perm Generation

   capacity = 45350912 (43.25MB)

   used     = 44791256 (42.716270446777344MB)

   free     = 559656 (0.5337295532226562MB)

   98.765943229543% used

On Friday, March 8, 2013 6:45:03 AM UTC-6, Rich Schmitt wrote:

[Lundh, Gustaf]

Hi!

 

1GB Max Heap seems really low (if you are not running a _very_ small installation).

 

Please try double (or triple) it by setting container.heapLimit in gerrit.config.

 

Best regards

Gustaf

 

From: repo-d...@googlegroups.com [mailto:repo-d...@googlegroups.com] On Behalf Of Rich Schmitt
Sent: den 13 mars 2013 13:21
To: repo-d...@googlegroups.com
Subject: Re: Gerrit Hangs with Java Heap Exhausted

 

 

This has happened again.

--

[Rich Schmitt]

I just wanted to update folks that tripling the heap to 3G solved the problem.    Sounds like a big heap but it works.  I appreciate your help.  Thank You.

Rich

On Friday, March 8, 2013 6:45:03 AM UTC-6, Rich Schmitt wrote:

[Ian Kumlien]

We actually use a 100gb heap, takes some tweaking to get it to perform but.... :-)

--

[Colby Ranger]

Which version are you running with the 100GB heap? How do you have you garbage collector configured?

-Colby

[Martin Fick]

On Monday, March 25, 2013 10:40:01 am Colby Ranger wrote:
> Which version are you running with the 100GB heap? How do
you have you
> garbage collector configured?

Also, how much RAM does your system have. I would advise
leaving at least 25-50% more just for OS/FS caching,

-Martin

discuss+u...@googlegroups.com
> >> More info at http://groups.google.com/group/repo-
discuss?hl=en

> >>
> >> ---
> >> You received this message because you are subscribed to
the Google
> >> Groups "Repo and Gerrit Discussion" group.
> >> To unsubscribe from this group and stop receiving emails
from it, send
> >> an email to repo-discuss...@googlegroups.com.
> >> For more options, visit
https://groups.google.com/groups/opt_out.
> >>
> > --
> >
> > --
> > To unsubscribe, email repo-

discuss+u...@googlegroups.com
> > More info at http://groups.google.com/group/repo-
discuss?hl=en

> >
> > ---
> > You received this message because you are subscribed to
the Google Groups
> > "Repo and Gerrit Discussion" group.
> > To unsubscribe from this group and stop receiving emails
from it, send an
> > email to repo-discuss...@googlegroups.com.
> > For more options, visit
https://groups.google.com/groups/opt_out.

--

The Qualcomm Innovation Center, Inc. is a member of Code
Aurora Forum, hosted by The Linux Foundation

[Ian Kumlien]

On Mon, Mar 25, 2013 at 5:40 PM, Colby Ranger <cra...@google.com> wrote:

Which version are you running with the 100GB heap? How do you have you garbage collector configured?

This is 2.4.2 running with a 1.7 jdk from oracle

It's currently running with:

-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:+CMSClassUnloadingEnabled -XX:+ParallelRefProcEnabled -XX:+ExplicitGCInvokesConcurrentAndUnloadsClasses

 

I also have:

-XX:+PrintGCDetails -XX:+PrintGCTimeStamps

So i can debug GC timings.. 

Other options i use:

-XX:LargePageSizeInBytes=4m -XX:MaxPermSize=256M

[Ian Kumlien]

On Mon, Mar 25, 2013 at 6:04 PM, Martin Fick <mf...@codeaurora.org> wrote:

On Monday, March 25, 2013 10:40:01 am Colby Ranger wrote:
> Which version are you running with the 100GB heap? How do
you have you
> garbage collector configured?

Also, how much RAM does your system have.  I would advise
leaving at least 25-50% more just for OS/FS caching,

Well, the point with having 100 gb heap is that most data should be in the cache

but yes, it's a valid point. Especially since most file-systems are bitmap based... 

(and memory management overhead)

The machine has 142 gb of memory:

free -g

             total       used       free     shared    buffers     cached

Mem:           141        131         10          0          0         52

-/+ buffers/cache:         78         63

Swap:            9          0          9

And with transparent hugepages it doesn't have the same memory overhead:

grep AnonHugePages /proc/meminfo 

AnonHugePages:  77154304 kB

(gerrit has 73 gb memory in res atm)

from gerrit:

Tasks:   23  total =    1 running +      0 ready +   22 sleeping

Mem:  99.85g total =  24.39g used +  48.24g free +  27.22g buffers

      99.85g max

        3620 open files,       24 cpus available,      259 threads

--
--

To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en