java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space

85 views
Skip to first unread message

Guy Levkowitz

unread,
Jun 16, 2025, 2:21:31 AMJun 16
to Repo and Gerrit Discussion
     Hey
we get stuck of our gerrit service and i have run the 
/bin/jstack <PID>  before it get stuck to a log file nd we can see in the log those Lockeds what can it be -->   
     - locked <0x00007f02cd32ca28> (a java.lang.Object)
        - locked <0x00007f02cd32caa0> (a sun.nio.ch.ChannelOutputStream)
        - locked <0x00007f02cc002bf8> (a java.lang.ref.NativeReferenceQueue$Lock)
        - locked <0x00007f728201de18> (a com.google.gitiles.GitilesFilter)
        - locked <0x00007f728201de18> (a com.google.gitiles.GitilesFilter)
        - locked <0x00007f728201de18> (a com.google.gitiles.GitilesFilter)
        - locked <0x00007f726702dd20> (a com.google.inject.servlet.ManagedFilterPipeline)
        - locked <0x00007f7269f68d70> (a com.google.gerrit.httpd.plugins.HttpPluginServlet)
        - locked <0x00007f7269d34600> (a com.google.inject.servlet.ManagedFilterPipeline)
        - locked <0x00007f726623e8e0> (a org.eclipse.jetty.servlet.FilterHolder)
        - locked <0x00007f726623d710> (a java.lang.Object)
        - locked <0x00007f726623d4c0> (a java.lang.Object)
        - locked <0x00007f726623d3a0> (a java.lang.Object)
        - locked <0x00007f8a62550970> (a java.lang.ref.NativeReferenceQueue$Lock)
        - locked <0x00007f7269b70a40> (a java.util.ArrayList)
        - locked <0x00007f7269b818a0> (a java.util.ArrayList)
        - locked <0x00007f7266130370> (a java.util.ArrayList)
        - locked <0x00007f7266141800> (a java.util.ArrayList)
        - locked <0x00007f7266131d28> (a java.util.TaskQueue)
        - locked <0x00007f7266197ed0> (a org.h2.store.WriterThread)
        - locked <0x00007f72a5619a78> (a org.h2.engine.Database)
        - locked <0x00007f72a5800578> (a org.h2.engine.Session)
        - locked <0x00007f7268d65990> (a com.google.gerrit.server.cache.h2.H2CacheImpl$SqlStore)
        - locked <0x00007f7266186c48> (a org.h2.store.WriterThread)
        - locked <0x00007f72661cbc80> (a java.util.ArrayList)
        - locked <0x00007f72670c5460> (a org.h2.store.WriterThread)
        - locked <0x00007f7266fcdf60> (a org.h2.store.WriterThread)
        - locked <0x00007f7267037f50> (a org.h2.store.WriterThread)
        - locked <0x00007f726712f450> (a org.h2.store.WriterThread)
        - locked <0x00007f72671529a0> (a org.h2.store.WriterThread)
        - locked <0x00007f72821787b0> (a java.util.concurrent.ConcurrentHashMap$ReservationNode)
        - locked <0x00007f728215ce90> (a java.util.ArrayList)
        - locked <0x00007f72a57c1110> (a org.h2.store.WriterThread)
        - locked <0x00007f72940086d0> (a java.lang.ClassNotFoundException)
        - locked <0x00007f7294008490> (a java.lang.Object)
        - locked <0x00007f72940060c8> (a java.lang.Object)
        - locked <0x00007f72a397f288> (a java.lang.Object)
        - locked <0x00007f72a0ee74b8> (a org.apache.sshd.server.session.ServerUserAuthService)
        - locked <0x00007f72a397f298> (a java.lang.Object)
        - locked <0x00007f72a006af40> (a java.lang.Object)
        - locked <0x00007f72a1b0ca38> (a org.apache.sshd.server.session.ServerUserAuthService)
        - locked <0x00007f72a006af50> (a java.lang.Object)
        - locked <0x00007f728216cc28> (a java.lang.Object)
        - locked <0x00007f72a82ce9e0> (a org.apache.sshd.server.session.ServerUserAuthService)
        - locked <0x00007f728216cc38> (a java.lang.Object)
        - locked <0x00007f18bcf4b428> (a com.google.gerrit.server.plugins.PluginLoader)
        - locked <0x00007f18bcf4b428> (a com.google.gerrit.server.plugins.PluginLoader)
        - locked <0x00007f30b64f10b0> (a java.lang.ref.NativeReferenceQueue$Lock)
        - locked <0x00007f18be073268> (a java.util.ArrayList)
        - locked <0x00007f18be04cc80> (a java.util.ArrayList)
        - locked <0x00007f18be099850> (a java.util.ArrayList)
        - locked <0x00007f18be0bfe38> (a java.util.ArrayList)
        - locked <0x00007f18be17fd18> (a java.util.TaskQueue)
        - locked <0x00007f18be04d378> (a org.h2.store.WriterThread)
        - locked <0x00007f18ea2f57f0> (a org.h2.store.FileLock)
        - locked <0x00007f18ea2f46d8> (a org.h2.engine.Database)
        - locked <0x00007f18be000000> (a org.h2.engine.Engine)
        - locked <0x00007f18bcd75428> (a com.google.gerrit.server.cache.h2.H2CacheImpl$SqlStore)
        - locked <0x00007f18be0c0268> (a java.util.ArrayList)
        - locked <0x00007f18ed0f8378> (a org.h2.store.WriterThread)
        - locked <0x00007f18e06d72f0> (a org.h2.store.WriterThread)



BR

Guy

Luca Milanesio

unread,
Jun 16, 2025, 3:21:24 AMJun 16
to Repo and Gerrit Discussion, Luca Milanesio
Hi Guy,

Did you look at the stack trace of the locked threads? Which is the resource they are waiting for and who’s holding that resource?

HTH

Luca.



BR

Guy


--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/f763dbf1-a563-4755-b84d-36ee7d27e3cdn%40googlegroups.com.

Guy Levkowitz

unread,
Jun 16, 2025, 7:09:24 AMJun 16
to Repo and Gerrit Discussion

Unfortunately I'm not an expert in analyzing the  jstack  command -  the log is attached maybe you can help 

ב-יום שני, 16 ביוני 2025 בשעה 10:21:24 UTC+3, Luca Milanesio כתב/ה:
pid_check_gerrit.log

Guy Levkowitz

unread,
Jun 16, 2025, 7:21:15 AMJun 16
to Repo and Gerrit Discussion

checking ChatGPT this output log of jstat it said:
Your Gerrit instance is stuck due to H2 database engine lock contention. The main thread and others are blocked waiting for the H2 engine lock, likely due to slow file IO or too many concurrent DB connections. This is a classic embedded DB concurrency issue.

the question is what can be done ?

ב-יום שני, 16 ביוני 2025 בשעה 14:09:24 UTC+3, Guy Levkowitz כתב/ה:

Luca Milanesio

unread,
Jun 16, 2025, 7:45:24 AMJun 16
to Repo and Gerrit Discussion, Luca Milanesio

On 16 Jun 2025, at 12:21, Guy Levkowitz <sil...@gmail.com> wrote:


checking ChatGPT this output log of jstat it said:
Your Gerrit instance is stuck due to H2 database engine lock contention. The main thread and others are blocked waiting for the H2 engine lock, likely due to slow file IO or too many concurrent DB connections. This is a classic embedded DB concurrency issue.

the question is what can be done ?

Did you ask ChatGPT that question?
I do not know how ChatGPT came to the conclusion that there are too many concurrent DB connections, therefore I can’t tell.


ב-יום שני, 16 ביוני 2025 בשעה 14:09:24 UTC+3, Guy Levkowitz כתב/ה:

Unfortunately I'm not an expert in analyzing the  jstack  command -  the log is attached maybe you can help 

From what I see in your main thread, you’re stuck loading the Gitiles plugin:

"main" #1 [1495222] prio=5 os_prio=0 cpu=10862.38ms elapsed=15.58s tid=0x00007f8a8002b4b0 nid=1495222 runnable  [0x00007f8a87342000]
   java.lang.Thread.State: RUNNABLE
at java.lang.invoke.MethodTypeForm.cachedLambdaForm(java...@21.0.6/MethodTypeForm.java:132)
at java.lang.invoke.DirectMethodHandle.preparedLambdaForm(java...@21.0.6/DirectMethodHandle.java:229)
at java.lang.invoke.DirectMethodHandle.preparedLambdaForm(java...@21.0.6/DirectMethodHandle.java:216)
at java.lang.invoke.DirectMethodHandle.preparedLambdaForm(java...@21.0.6/DirectMethodHandle.java:225)
at java.lang.invoke.DirectMethodHandle.make(java...@21.0.6/DirectMethodHandle.java:106)
at java.lang.invoke.MethodHandles$Lookup.getDirectMethodCommon(java...@21.0.6/MethodHandles.java:4109)
at java.lang.invoke.MethodHandles$Lookup.getDirectMethodNoSecurityManager(java...@21.0.6/MethodHandles.java:4065)
at java.lang.invoke.MethodHandles$Lookup.getDirectMethodForConstant(java...@21.0.6/MethodHandles.java:4314)
at java.lang.invoke.MethodHandles$Lookup.linkMethodHandleConstant(java...@21.0.6/MethodHandles.java:4262)
at java.lang.invoke.MethodHandleNatives.linkMethodHandleConstant(java...@21.0.6/MethodHandleNatives.java:627)
at com.google.gitiles.Linkifier.<clinit>(Linkifier.java:46)
at com.google.gitiles.GitilesFilter.linkifier(GitilesFilter.java:302)
- locked <0x00007f728201de18> (a com.google.gitiles.GitilesFilter)
at com.google.gitiles.GitilesFilter.getDefaultHandler(GitilesFilter.java:258)
- locked <0x00007f728201de18> (a com.google.gitiles.GitilesFilter)
at com.google.gitiles.GitilesFilter.init(GitilesFilter.java:218)
- locked <0x00007f728201de18> (a com.google.gitiles.GitilesFilter)
at com.google.gitiles.GitilesServlet.init(GitilesServlet.java:109)

Have you tried starting it with ‘gerrit.sh run’ and see the output?

HTH

Luca.

Guy Levkowitz

unread,
Jun 16, 2025, 8:23:59 AMJun 16
to Repo and Gerrit Discussion
some are locked via  GitilesFilter, but other on something else , 
I have not start the  gerrit.sh run - I can give it a try , but the stuck it take sometime 1 day sometime 2 - it is not persistent - today we have it twice , and after I restart the service all goes well and work fine till our server get stuck again 


 
Line   21: - locked <0x00007f02cd32ca28> (a java.lang.Object)
Line   24: - locked <0x00007f02cd32caa0> (a sun.nio.ch.ChannelOutputStream)
Line   46: - locked <0x00007f02cc002bf8> (a java.lang.ref.NativeReferenceQueue$Lock)
Line  237: - locked <0x00007f728201de18> (a com.google.gitiles.GitilesFilter)
Line  239: - locked <0x00007f728201de18> (a com.google.gitiles.GitilesFilter)
Line  241: - locked <0x00007f728201de18> (a com.google.gitiles.GitilesFilter)
Line  246: - locked <0x00007f726702dd20> (a com.google.inject.servlet.ManagedFilterPipeline)
Line  251: - locked <0x00007f7269f68d70> (a com.google.gerrit.httpd.plugins.HttpPluginServlet)
Line  255: - locked <0x00007f7269d34600> (a com.google.inject.servlet.ManagedFilterPipeline)
Line  258: - locked <0x00007f726623e8e0> (a org.eclipse.jetty.servlet.FilterHolder)
Line  269: - locked <0x00007f726623d710> (a java.lang.Object)
Line  274: - locked <0x00007f726623d4c0> (a java.lang.Object)
Line  281: - locked <0x00007f726623d3a0> (a java.lang.Object)
Line  313: - locked <0x00007f8a62550970> (a java.lang.ref.NativeReferenceQueue$Lock)
Line  373: - locked <0x00007f7269b70a40> (a java.util.ArrayList)
Line  384: - locked <0x00007f7269b818a0> (a java.util.ArrayList)
Line  395: - locked <0x00007f7266130370> (a java.util.ArrayList)
Line  406: - locked <0x00007f7266141800> (a java.util.ArrayList)
Line  587: - locked <0x00007f7266131d28> (a java.util.TaskQueue)
Line  604: - locked <0x00007f7266197ed0> (a org.h2.store.WriterThread)
Line  617: - locked <0x00007f72a5619a78> (a org.h2.engine.Database)
Line  619: - locked <0x00007f72a5800578> (a org.h2.engine.Session)
Line  622: - locked <0x00007f7268d65990> (a com.google.gerrit.server.cache.h2.H2CacheImpl$SqlStore)
Line  659: - locked <0x00007f7266186c48> (a org.h2.store.WriterThread)
Line  670: - locked <0x00007f72661cbc80> (a java.util.ArrayList)
Line  720: - locked <0x00007f72670c5460> (a org.h2.store.WriterThread)
Line  758: - locked <0x00007f7266fcdf60> (a org.h2.store.WriterThread)
Line  790: - locked <0x00007f7267037f50> (a org.h2.store.WriterThread)
Line  808: - locked <0x00007f726712f450> (a org.h2.store.WriterThread)
Line  867: - locked <0x00007f72671529a0> (a org.h2.store.WriterThread)
Line  972: - locked <0x00007f72821787b0> (a java.util.concurrent.ConcurrentHashMap$ReservationNode)
Line 1004: - locked <0x00007f728215ce90> (a java.util.ArrayList)
Line 1516: - locked <0x00007f72a57c1110> (a org.h2.store.WriterThread)
Line 1655: - locked <0x00007f72940086d0> (a java.lang.ClassNotFoundException)
Line 1663: - locked <0x00007f7294008490> (a java.lang.Object)
Line 1675: - locked <0x00007f72940060c8> (a java.lang.Object)
Line 1680: - locked <0x00007f72a397f288> (a java.lang.Object)
Line 1683: - locked <0x00007f72a0ee74b8> (a org.apache.sshd.server.session.ServerUserAuthService)
Line 1692: - locked <0x00007f72a397f298> (a java.lang.Object)
Line 1711:    java.lang.Thread.State: BLOCKED (on object monitor)
Line 1718: - locked <0x00007f72a006af40> (a java.lang.Object)
Line 1721: - locked <0x00007f72a1b0ca38> (a org.apache.sshd.server.session.ServerUserAuthService)
Line 1730: - locked <0x00007f72a006af50> (a java.lang.Object)
Line 1749:    java.lang.Thread.State: BLOCKED (on object monitor)
Line 1756: - locked <0x00007f728216cc28> (a java.lang.Object)
Line 1759: - locked <0x00007f72a82ce9e0> (a org.apache.sshd.server.session.ServerUserAuthService)
Line 1768: - locked <0x00007f728216cc38> (a java.lang.Object)
Line 1968:    java.lang.Thread.State: BLOCKED (on object monitor)
Line 2011: - locked <0x00007f18bcf4b428> (a com.google.gerrit.server.plugins.PluginLoader)
Line 2013: - locked <0x00007f18bcf4b428> (a com.google.gerrit.server.plugins.PluginLoader)
Line 2044: - locked <0x00007f30b64f10b0> (a java.lang.ref.NativeReferenceQueue$Lock)
Line 2100: - locked <0x00007f18be073268> (a java.util.ArrayList)
Line 2111: - locked <0x00007f18be04cc80> (a java.util.ArrayList)
Line 2122: - locked <0x00007f18be099850> (a java.util.ArrayList)
Line 2133: - locked <0x00007f18be0bfe38> (a java.util.ArrayList)
Line 2314: - locked <0x00007f18be17fd18> (a java.util.TaskQueue)
Line 2331: - locked <0x00007f18be04d378> (a org.h2.store.WriterThread)
Line 2342: - locked <0x00007f18ea2f57f0> (a org.h2.store.FileLock)
Line 2344: - locked <0x00007f18ea2f46d8> (a org.h2.engine.Database)
Line 2349: - locked <0x00007f18be000000> (a org.h2.engine.Engine)
Line 2361: - locked <0x00007f18bcd75428> (a com.google.gerrit.server.cache.h2.H2CacheImpl$SqlStore)
Line 2391: - locked <0x00007f18be0c0268> (a java.util.ArrayList)
Line 2409: - locked <0x00007f18ed0f8378> (a org.h2.store.WriterThread)
Line 2463: - locked <0x00007f18e06d72f0> (a org.h2.store.WriterThread)
Line 2482:    java.lang.Thread.State: BLOCKED (on object monitor

ב-יום שני, 16 ביוני 2025 בשעה 14:45:24 UTC+3, Luca Milanesio כתב/ה:

Luca Milanesio

unread,
Jun 16, 2025, 8:42:29 AMJun 16
to Repo and Gerrit Discussion, Luca Milanesio

On 16 Jun 2025, at 13:23, Guy Levkowitz <sil...@gmail.com> wrote:

some are locked via  GitilesFilter, but other on something else , 
I have not start the  gerrit.sh run - I can give it a try , but the stuck it take sometime 1 day sometime 2 - it is not persistent - today we have it twice , and after I restart the service all goes well and work fine till our server get stuck again 

I can only give you suggestions on what I see and you’ve shared on the mailing list.

Did you check the overall metrics to see if you have some other red alerts?
(unusual cache evictions, thread pools, workload higher than the box capacity)
The above fragment doesn’t say much, without the call stacks.

Luca.

Start Export

unread,
Jun 16, 2025, 9:20:46 PMJun 16
to Luca Milanesio, Repo and Gerrit Discussion
w

To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/1A0D79B5-8C65-4523-84FD-10F74FA0A849%40gmail.com.
Reply all
Reply to author
Forward
0 new messages