java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space

Guy Levkowitz

unread,

May 28, 2025, 9:53:55 AM5/28/25

to Repo and Gerrit Discussion

Hey

We have gerrit 3.10.1 below definitions :

3500 - projects (repos)

javaHome = /usr/lib/jvm/java-21-openjdk-21.0.6.0.7-1.0.1.el8.x86_64

heapLimit = 96g

CPU(s): 48

Red Hat Enterprise Linux release 8.10

free -h
total used free shared buff/cache available
Mem: 161Gi 45Gi 61Gi 1.3Gi 54Gi 113Gi
Swap: 23Gi 2.0Mi 23Gi

And we get lots of : "Java heap space" that cause our gerrit to get stuck and needed to restart the service - - what can be done ?

[2025-05-28T15:46:14.188+03:00] [sshd-SshDaemon[41c02b8c](port=22)-nio2-thread-2] WARN org.apache.sshd.server.session.ServerSessionImpl : exceptionCaught(ServerSessionImpl[null@/192.168.120.34:54098])[state=Opened] IOException: Broken pipe
[2025-05-28T15:46:31.615+03:00] [SSH git-receive-pack /repo (XXXX)] WARN com.google.gerrit.server.git.MultiProgressMonitor : MultiProgressMonitor worker did not call end() before returning
[2025-05-28T15:46:31.615+03:00] [SSH git-receive-pack /repo (XXXX)] ERROR com.google.gerrit.server.git.receive.AsyncReceiveCommits : error while processing push
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
at com.google.gerrit.server.git.receive.AsyncReceiveCommits.preReceive(AsyncReceiveCommits.java:407)
at com.google.gerrit.server.git.receive.AsyncReceiveCommits.lambda$asHook$0(AsyncReceiveCommits.java:351)
at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:2287)
at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:2200)
at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:98)
at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:109)
at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:74)
at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:492)
at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:703)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.search.comparators.LongComparator.<init>(LongComparator.java:37)
at org.apache.lucene.search.SortField.getComparator(SortField.java:529)
at org.apache.lucene.search.FieldValueHitQueue.<init>(FieldValueHitQueue.java:137)
at org.apache.lucene.search.FieldValueHitQueue$MultiComparatorsFieldValueHitQueue.<init>(FieldValueHitQueue.java:98)
at org.apache.lucene.search.FieldValueHitQueue.create(FieldValueHitQueue.java:161)
at org.apache.lucene.search.TopFieldCollector.create(TopFieldCollector.java:454)
at org.apache.lucene.search.IndexSearcher$2.newCollector(IndexSearcher.java:652)
at org.apache.lucene.search.IndexSearcher$2.newCollector(IndexSearcher.java:639)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:684)
at org.apache.lucene.search.IndexSearcher.searchAfter(IndexSearcher.java:667)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:583)
at com.google.gerrit.lucene.LuceneChangeIndex$QuerySource.doRead(LuceneChangeIndex.java:433)
at com.google.gerrit.lucene.LuceneChangeIndex$QuerySource$1.call(LuceneChangeIndex.java:357)
at com.google.gerrit.lucene.LuceneChangeIndex$QuerySource$1.call(LuceneChangeIndex.java:354)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:703)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.runWith(Thread.java:1596)
... 1 more

thanks

Guy

Luca Milanesio

unread,

May 29, 2025, 5:41:20 PM5/29/25

to Repo and Gerrit Discussion, Luca Milanesio

On 28 May 2025, at 15:53, Guy Levkowitz <sil...@gmail.com> wrote:

Hey

We have gerrit 3.10.1 below definitions :
3500 - projects (repos)

javaHome = /usr/lib/jvm/java-21-openjdk-21.0.6.0.7-1.0.1.el8.x86_64
heapLimit = 96g
CPU(s): 48
Red Hat Enterprise Linux release 8.10
free -h
total used free shared buff/cache available
Mem: 161Gi 45Gi 61Gi 1.3Gi 54Gi 113Gi
Swap: 23Gi 2.0Mi 23Gi

And we get lots of : "Java heap space" that cause our gerrit to get stuck and needed to restart the service - - what can be done ?
[2025-05-28T15:46:14.188+03:00] [sshd-SshDaemon[41c02b8c](port=22)-nio2-thread-2] WARN org.apache.sshd.server.session.ServerSessionImpl : exceptionCaught(ServerSessionImpl[null@/192.168.120.34:54098])[state=Opened] IOException: Broken pipe
[2025-05-28T15:46:31.615+03:00] [SSH git-receive-pack /repo (XXXX)] WARN com.google.gerrit.server.git.MultiProgressMonitor : MultiProgressMonitor worker did not call end() before returning
[2025-05-28T15:46:31.615+03:00] [SSH git-receive-pack /repo (XXXX)] ERROR com.google.gerrit.server.git.receive.AsyncReceiveCommits : error while processing push
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space

I doubt that a single push managed to use 96GBytes of RAM.

You should analyse the JVM heap utilisation from when the Gerrit instance started until the *very first* OOM exception in the logs.

Also, check in which classes the heap is used and understand if you have some caches misconfiguration.

HTH

Luca.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/d85bac2b-64ed-4d6c-b080-fbeed7978aacn%40googlegroups.com.

Matthias Sohn

unread,

May 30, 2025, 8:20:44 AM5/30/25

to Luca Milanesio, Repo and Gerrit Discussion

On Thu, May 29, 2025 at 11:41 PM Luca Milanesio <luca.mi...@gmail.com> wrote:

On 28 May 2025, at 15:53, Guy Levkowitz <sil...@gmail.com> wrote:

Hey

We have gerrit 3.10.1 below definitions :
3500 - projects (repos)

javaHome = /usr/lib/jvm/java-21-openjdk-21.0.6.0.7-1.0.1.el8.x86_64
heapLimit = 96g
CPU(s): 48
Red Hat Enterprise Linux release 8.10
free -h
total used free shared buff/cache available
Mem: 161Gi 45Gi 61Gi 1.3Gi 54Gi 113Gi
Swap: 23Gi 2.0Mi 23Gi

And we get lots of : "Java heap space" that cause our gerrit to get stuck and needed to restart the service - - what can be done ?
[2025-05-28T15:46:14.188+03:00] [sshd-SshDaemon[41c02b8c](port=22)-nio2-thread-2] WARN org.apache.sshd.server.session.ServerSessionImpl : exceptionCaught(ServerSessionImpl[null@/192.168.120.34:54098])[state=Opened] IOException: Broken pipe
[2025-05-28T15:46:31.615+03:00] [SSH git-receive-pack /repo (XXXX)] WARN com.google.gerrit.server.git.MultiProgressMonitor : MultiProgressMonitor worker did not call end() before returning
[2025-05-28T15:46:31.615+03:00] [SSH git-receive-pack /repo (XXXX)] ERROR com.google.gerrit.server.git.receive.AsyncReceiveCommits : error while processing push
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space

I doubt that a single push managed to use 96GBytes of RAM.
You should analyse the JVM heap utilisation from when the Gerrit instance started until the *very first* OOM exception in the logs.

The request logs track how much memory each request has allocated,

see https://gerrit-review.googlesource.com/Documentation/logs.html

To avoid overloading a gerrit server look at the thread pool sizes and relate them to the hardware gerrit is running on.

Use a monitoring solution, e.g. https://gerrit.googlesource.com/gerrit-monitoring/+/refs/heads/master to find potential issues.

As a rule of thumb you need one core per concurrent git operation. The maximum number of threads serving concurrent

git requests is controlled by sshd.threads.

Check how much CPU% you spend on Java gc. Enable Java gc logs to get more details about potential Java gc issues.

If you have a lot of load, serve git upload-pack (fetch, clone) requests from a separate JVM (co-located Gerrit replica).

We made good experience doing that and use Java 21 with generational ZGC for Gerrit primaries and ParallelGC for replicas.

To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/721DA241-3904-4390-9FE8-6C81CB61DD45%40gmail.com.

Reply all

Reply to author

Forward