How to increase Gerrit heap size?

[Eric Tsai]

When I do "git clone ssh://username@gerrit:29418/git_test" to clone
git_test which is a 300MB repository from Gerrit, I'll get:
Initialized empty Git repository in /root/git_test/.git/
fatal: internal server error
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed
fetch-pack from 'ssh://username@gerrit:29418/git_test' failed.

On the Gerrit server I'll see:
2009-04-24 15:52:06,293::ERROR:
com.google.gerrit.server.ssh.AbstractCommand - sshd error (account
1000000): git-upload-pack /test_kernel_8k: fatal: internal server
error
java.lang.OutOfMemoryError: Java heap space

How do I increase the heap space?

[Eric Tsai]

I add -Xmx512M to JAVA_OPTIONS in gerrit2-jetty script then the
problem is fixed.

[Mike]

I've added -Xmx512M to my JAVA_OPTIONS, but I still get the errors:

"java.lang.OutOfMemoryError: Java heap space"

and

"Exception in thread "pool-2-thread-3" java.lang.OutOfMemoryError: GC
overhead limit exceeded"

Any other solutions out there?

Thanks,
Mike

[Shawn Pearce]

2009/6/8 Mike <msw...@gmail.com>

I've added  -Xmx512M to my JAVA_OPTIONS, but I still get the errors:

"java.lang.OutOfMemoryError: Java heap space"

and

"Exception in thread "pool-2-thread-3" java.lang.OutOfMemoryError: GC
overhead limit exceeded"

Over the weekend we found a bug where a particular type of tree can cause Gerrit to go into an infinite loop while executing a merge, allocating more and more memory during each iteration, until it just runs out of heap space.  Its possible this is the same issue.  I submitted a patch upstream to JGit, but Robin hasn't picked it up yet.

Check your submit queue:

  SELECT change_id FROM changes WHERE status = 's';

If there are changes pending in the submit queue, unsubmit them:

  UPDATE changes SET status = 'n' WHERE status = 's';

rebase those changes onto current tip of tree, upload replace, and submit.  But, if the project is using cherry pick submits, you'll need to temporarily disable that and use fast forward only, and edit the commit message yourself, because its the 3-way merge inside Gerrit that blows up the memory usage.

I'm sort of holding 2.0.14 until JGit is patched for this.

[Mike]

reviewdb=> SELECT change_id FROM changes WHERE status = 's';
change_id
-----------
(0 rows)

reviewdb=>

Any other possibilities?


On Jun 8, 10:48 am, Shawn Pearce <s...@google.com> wrote:
> 2009/6/8 Mike <mswe...@gmail.com>

[Shawn Pearce]

On Mon, Jun 8, 2009 at 11:04, Mike <msw...@gmail.com> wrote:

Any other possibilities?

*sigh* 

Unfortunately OOME doesn't come with a stack trace, so its hard to say what the JVM was up to when it died.  

Thread name "pool-2-thread-3" sounds like it might be one of the worker threads, like for the remote replication.  Do you have a huge project being replicated to another slave, and the slave is maybe really far behind?

Are you sure -Xmx512m actually boosted your heap?  Somewhere in Java 5/6 Sun changed the JVM to make a guess on memory based on how much physical memory is installed in the host.  It could be that the initial guess was actually 512m, or around there, and explicitly setting it to that isn't making a significant change.

[Mike]

Nope, not sure. How can I tell?

Top shows this (I have no idea how to fixed-width style text in Google
Groups :/ ):
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12244 gerrit 20 0 917m 570m 9m S 1 14.5 16:34.72 java

Would it help if I attached my error log? If so, is there a way to
attach documents in Google Groups or should I just post it in a
message?

Thanks for the help,
Mike


On Jun 8, 11:12 am, Shawn Pearce <s...@google.com> wrote:

[Shawn Pearce]

On Mon, Jun 8, 2009 at 11:19, Mike <msw...@gmail.com> wrote:

Nope, not sure.  How can I tell?

Top shows this (I have no idea how to fixed-width style text in Google
Groups :/ ):
 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
12244 gerrit    20   0  917m 570m   9m S    1 14.5  16:34.72 java

Eh, that does seem like ~512m heap.  Maybe "gerrit show-caches" is more detailed, e.g.:

$ ssh -p 29418 review.source.android.com gerrit show-caches

...

JVM Heap:

  max         : 880.00 mb

  inuse       : 353.24 mb

  mem%        :  50%

also maybe look at your replication queue:

$ ssh -p 29418 review.source.android.com gerrit show-queue

 S  Start         Task

--------------------------------------------------------------

--------------------------------------------------------------

  0 tasks

 

Would it help if I attached my error log?  If so, is there a way to
attach documents in Google Groups or should I just post it in a
message?

Probably better to just email it to me privately, no need for the whole group to receive a copy of an error log that may or may not show something useful to troubleshoot.

[Mike]

show-caches:

JVM Heap:
max : 455.13 mb
inuse : 384.42 mb
mem% : 85%

show-queue

S Start Task
--------------------------------------------------------------
--------------------------------------------------------------
0 tasks

That max is definitely less than 512. Maybe it's saving memory for
other Java processes? Should I increase mine then since yours is at
800? Something like -Xmx800m?


On Jun 8, 11:22 am, Shawn Pearce <s...@google.com> wrote:

[Shawn Pearce]

On Mon, Jun 8, 2009 at 11:39, Mike <msw...@gmail.com> wrote:

show-caches:

JVM Heap:
 max         : 455.13 mb
 inuse       : 384.42 mb
 mem%        :  85%

... 

That max is definitely less than 512.  Maybe it's saving memory for
other Java processes?   Should I increase mine then since yours is at
800?  Something like -Xmx800m?

Yea, sounds like you should go higher, if you have the physical memory for it.  r.s.a.c runs with around 880 (as you saw), but it also doesn't have to process really big uploads from users.  So the ~880m is generally sufficient.

[Mike]

So far so good at 880...

Thanks,
Mike

On Jun 8, 11:50 am, Shawn Pearce <s...@google.com> wrote: