Reason for BLOCKED thread

[Layona Kunjumon]

Hi Folks,

We have seen below BLOCKED thread while gerrit server load reaching high. Could you please check and help the reason for this?

gerrit version: 2.16.22

"SSH-Stream-Worker-37" #694 prio=1 os_prio=0 tid=0x00007f202801f800 nid=0x1381d waiting for monitor entry [0x00007f1ed85d7000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:165)
- waiting to lock <0x00007fed7b75f6b8> (a com.google.common.base.Suppliers$NonSerializableMemoizingSupplier)
at com.google.gerrit.server.events.SupplierSerializer.serialize(SupplierSerializer.java:27)
at com.google.gerrit.server.events.SupplierSerializer.serialize(SupplierSerializer.java:23)
at com.google.gson.internal.bind.TreeTypeAdapter.write(TreeTypeAdapter.java:81)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.write(TypeAdapterRuntimeTypeWrapper.java:69)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.write(ReflectiveTypeAdapterFactory.java:127)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.write(ReflectiveTypeAdapterFactory.java:245)
at com.google.gson.Gson.toJson(Gson.java:704)
at com.google.gson.Gson.toJson(Gson.java:683)
at com.google.gson.Gson.toJson(Gson.java:638)
at com.google.gson.Gson.toJson(Gson.java:618)
at com.google.gerrit.sshd.commands.StreamEvents.write(StreamEvents.java:279)
at com.google.gerrit.sshd.commands.StreamEvents.writeEvents(StreamEvents.java:259)
at com.google.gerrit.sshd.commands.StreamEvents.access$000(StreamEvents.java:52)
at com.google.gerrit.sshd.commands.StreamEvents$1.run(StreamEvents.java:98)
at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:83)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:646)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Regards,

Layona

[Matthias Sohn]

On Wed, Sep 29, 2021 at 11:46 AM Layona Kunjumon <layonal...@gmail.com> wrote:

Hi Folks,

We have seen below BLOCKED thread while gerrit server load reaching high. Could you please check and help the reason for this?

gerrit version: 2.16.22

you should plan to upgrade to a supported release, see https://www.gerritcodereview.com/support.html

 

"SSH-Stream-Worker-37" #694 prio=1 os_prio=0 tid=0x00007f202801f800 nid=0x1381d waiting for monitor entry [0x00007f1ed85d7000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:165)
- waiting to lock <0x00007fed7b75f6b8> (a com.google.common.base.Suppliers$NonSerializableMemoizingSupplier)

This thread tries to write stream events and is blocked by another thread having acquired the lock on <0x00007fed7b75f6b8>

Search for other occurrences of <0x00007fed7b75f6b8> in the same thread dump in order to find out what the thread is doing

which holds this lock.

 

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/922d79ce-6fb0-4b90-beea-77ad9edc7972n%40googlegroups.com.

[Layona Kunjumon]

Hi Matthias,

Thanks for your response.

 

you should plan to upgrade to a supported release, see https://www.gerritcodereview.com/support.html

We already have gerrit version upgrade in upcoming schedule.

 

This thread tries to write stream events and is blocked by another thread having acquired the lock on <0x00007fed7b75f6b8>

Search for other occurrences of <0x00007fed7b75f6b8> in the same thread dump in order to find out what the thread is doing

which holds this lock.

I have searched in thread dump and found that it is locked by another ssh stream worker.

"SSH-Stream-Worker-9" #611 prio=1 os_prio=0 tid=0x00007f2130005000 nid=0x13729 runnable [0x00007f1edd5d5000]

java.lang.Thread.State: RUNNABLE
at java.lang.ThreadLocal$ThreadLocalMap.cleanSomeSlots(ThreadLocal.java:661)
at java.lang.ThreadLocal$ThreadLocalMap.set(ThreadLocal.java:483)
at java.lang.ThreadLocal$ThreadLocalMap.access$100(ThreadLocal.java:298)
at java.lang.ThreadLocal.set(ThreadLocal.java:203)
at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:836)
at com.sun.proxy.$Proxy88.create(Unknown Source)
at com.google.gerrit.server.query.change.ChangeData$Factory.create(ChangeData.java:295)
at com.google.gerrit.server.change.ChangeKindCacheImpl.getChangeKindInternal(ChangeKindCacheImpl.java:434)
at com.google.gerrit.server.change.ChangeKindCacheImpl.getChangeKind(ChangeKindCacheImpl.java:362)
at com.google.gerrit.server.events.EventFactory.asPatchSetAttribute(EventFactory.java:540)
at com.google.gerrit.server.events.EventFactory.asPatchSetAttribute(EventFactory.java:493)
at com.google.gerrit.server.events.StreamEventsApiListener$3.get(StreamEventsApiListener.java:187)
at com.google.gerrit.server.events.StreamEventsApiListener$3.get(StreamEventsApiListener.java:182)
at com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:167)
- locked <0x00007fed7b75f6b8> (a com.google.common.base.Suppliers$NonSerializableMemoizingSupplier)