Found one Java-level deadlock

[Andy]

when use repo sync gerrit code, We often encounter issues where the team gets stuck while fetching code

so I get  jvm thread dumps find 1 deadlock. how can I fix that.

Found one Java-level deadlock:
=============================
"sshd-SshDaemon[7308c29a](port=22)-nio2-thread-6":
  waiting to lock monitor 0x00007f045402e600 (object 0x00007f0a08bbdf68, a org.apache.sshd.common.channel.ChannelOutputStream),
  which is held by "SSH git-upload-pack /ICVSBG/CHUDC/AP/E04/SDK2.5/aosp/platform/frameworks/base (jenkins_dl)"
"SSH git-upload-pack /ICVSBG/CHUDC/AP/E04/SDK2.5/aosp/platform/frameworks/base (jenkins_dl)":
  waiting to lock monitor 0x00007f03e801be80 (object 0x00007f0f679f2d20, a java.util.LinkedList),
  which is held by "sshd-SshDaemon[7308c29a](port=22)-nio2-thread-6"

Java stack information for the threads listed above:
===================================================
"sshd-SshDaemon[7308c29a](port=22)-nio2-thread-6":
at org.apache.sshd.common.channel.ChannelOutputStream.close(ChannelOutputStream.java:275)
- waiting to lock <0x00007f0a08bbdf68> (a org.apache.sshd.common.channel.ChannelOutputStream)
at org.apache.sshd.common.util.io.IoUtils.closeQuietly(IoUtils.java:191)
at org.apache.sshd.common.util.io.IoUtils.closeQuietly(IoUtils.java:146)
at org.apache.sshd.server.channel.ChannelSession.closeImmediately0(ChannelSession.java:210)
at org.apache.sshd.server.channel.ChannelSession$$Lambda$1306/0x00007f04c2a6f440.run(Unknown Source)
at org.apache.sshd.common.util.closeable.Builder$1.doClose(Builder.java:47)
at org.apache.sshd.common.util.closeable.SimpleCloseable.close(SimpleCloseable.java:63)
at org.apache.sshd.common.util.closeable.SequentialCloseable$1.operationComplete(SequentialCloseable.java:56)
at org.apache.sshd.common.util.closeable.SequentialCloseable$1.operationComplete(SequentialCloseable.java:45)
at org.apache.sshd.common.future.AbstractSshFuture.notifyListener(AbstractSshFuture.java:159)
at org.apache.sshd.common.future.DefaultSshFuture.addListener(DefaultSshFuture.java:166)
at org.apache.sshd.common.util.closeable.SequentialCloseable$1.operationComplete(SequentialCloseable.java:57)
at org.apache.sshd.common.util.closeable.SequentialCloseable$1.operationComplete(SequentialCloseable.java:45)
at org.apache.sshd.common.future.AbstractSshFuture.notifyListener(AbstractSshFuture.java:159)
at org.apache.sshd.common.future.DefaultSshFuture.addListener(DefaultSshFuture.java:166)
at org.apache.sshd.common.util.closeable.SequentialCloseable$1.operationComplete(SequentialCloseable.java:57)
at org.apache.sshd.common.util.closeable.SequentialCloseable$1.operationComplete(SequentialCloseable.java:45)
at org.apache.sshd.common.future.AbstractSshFuture.notifyListener(AbstractSshFuture.java:159)
at org.apache.sshd.common.future.DefaultSshFuture.addListener(DefaultSshFuture.java:166)
at org.apache.sshd.common.util.closeable.SequentialCloseable$1.operationComplete(SequentialCloseable.java:57)
at org.apache.sshd.common.util.closeable.SequentialCloseable$1.operationComplete(SequentialCloseable.java:45)
at org.apache.sshd.common.util.closeable.SequentialCloseable.doClose(SequentialCloseable.java:69)
at org.apache.sshd.common.util.closeable.SimpleCloseable.close(SimpleCloseable.java:63)
at org.apache.sshd.common.util.closeable.AbstractInnerCloseable.doCloseImmediately(AbstractInnerCloseable.java:48)
at org.apache.sshd.common.util.closeable.AbstractCloseable.close(AbstractCloseable.java:95)
at org.apache.sshd.common.util.closeable.ParallelCloseable.doClose(ParallelCloseable.java:65)
at org.apache.sshd.common.util.closeable.SimpleCloseable.close(SimpleCloseable.java:63)
at org.apache.sshd.common.util.closeable.AbstractInnerCloseable.doCloseImmediately(AbstractInnerCloseable.java:48)
at org.apache.sshd.common.util.closeable.AbstractCloseable.close(AbstractCloseable.java:95)
at org.apache.sshd.common.util.closeable.ParallelCloseable.doClose(ParallelCloseable.java:65)
at org.apache.sshd.common.util.closeable.SimpleCloseable.close(SimpleCloseable.java:63)
at org.apache.sshd.common.util.closeable.SequentialCloseable$1.operationComplete(SequentialCloseable.java:56)
at org.apache.sshd.common.util.closeable.SequentialCloseable$1.operationComplete(SequentialCloseable.java:45)
at org.apache.sshd.common.util.closeable.SequentialCloseable.doClose(SequentialCloseable.java:69)
at org.apache.sshd.common.util.closeable.SimpleCloseable.close(SimpleCloseable.java:63)
at org.apache.sshd.common.util.closeable.AbstractInnerCloseable.doCloseImmediately(AbstractInnerCloseable.java:48)
at org.apache.sshd.common.util.closeable.AbstractCloseable.close(AbstractCloseable.java:95)
at org.apache.sshd.common.session.helpers.SessionHelper.exceptionCaught(SessionHelper.java:1251)
at org.apache.sshd.common.session.helpers.AbstractSessionIoHandler.exceptionCaught(AbstractSessionIoHandler.java:53)
at org.apache.sshd.common.io.nio2.Nio2Session.exceptionCaught(Nio2Session.java:201)
at org.apache.sshd.common.io.nio2.Nio2Session.handleWriteCycleFailure(Nio2Session.java:522)
at org.apache.sshd.common.io.nio2.Nio2Session$2.onFailed(Nio2Session.java:480)
at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$failed$1(Nio2CompletionHandler.java:46)
at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$1871/0x00007f03bb468100.run(Unknown Source)
at java.security.AccessController.doPrivileged(java...@11.0.17/Native Method)
at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.failed(Nio2CompletionHandler.java:45)
at sun.nio.ch.Invoker.invokeUnchecked(java...@11.0.17/Invoker.java:129)
at sun.nio.ch.Invoker.invokeDirect(java...@11.0.17/Invoker.java:158)
at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implWrite(java...@11.0.17/UnixAsynchronousSocketChannelImpl.java:746)
at sun.nio.ch.AsynchronousSocketChannelImpl.write(java...@11.0.17/AsynchronousSocketChannelImpl.java:383)
at sun.nio.ch.AsynchronousSocketChannelImpl.write(java...@11.0.17/AsynchronousSocketChannelImpl.java:400)
at org.apache.sshd.common.io.nio2.Nio2Session.doWriteCycle(Nio2Session.java:466)
at org.apache.sshd.common.io.nio2.Nio2Session.startWriting(Nio2Session.java:449)
at org.apache.sshd.common.io.nio2.Nio2Session.writeBuffer(Nio2Session.java:184)
at org.apache.sshd.common.session.helpers.AbstractSession.doWritePacket(AbstractSession.java:993)
- locked <0x00007f0a08bbe9c0> (a java.lang.Object)
at org.apache.sshd.common.session.helpers.AbstractSession.sendPendingPackets(AbstractSession.java:787)
- locked <0x00007f0a08bbe9c0> (a java.lang.Object)
at org.apache.sshd.common.session.helpers.AbstractSession.handleNewKeys(AbstractSession.java:749)
- locked <0x00007f0f679f2d20> (a java.util.LinkedList)
at org.apache.sshd.common.session.helpers.AbstractSession.doHandleMessage(AbstractSession.java:481)
at org.apache.sshd.common.session.helpers.AbstractSession.handleMessage(AbstractSession.java:429)
- locked <0x00007f0a08bbea60> (a java.lang.Object)
at org.apache.sshd.common.session.helpers.AbstractSession.decode(AbstractSession.java:1466)
at org.apache.sshd.common.session.helpers.AbstractSession.messageReceived(AbstractSession.java:389)
- locked <0x00007f0f679f2d50> (a java.lang.Object)
at org.apache.sshd.common.session.helpers.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:64)
at org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:359)
at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:336)
at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:333)
at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$1005/0x00007f054de18900.run(Unknown Source)
at java.security.AccessController.doPrivileged(java...@11.0.17/Native Method)
at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
at sun.nio.ch.Invoker.invokeUnchecked(java...@11.0.17/Invoker.java:127)
at sun.nio.ch.Invoker$2.run(java...@11.0.17/Invoker.java:219)
at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(java...@11.0.17/AsynchronousChannelGroupImpl.java:112)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java...@11.0.17/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java...@11.0.17/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java...@11.0.17/Thread.java:829)
"SSH git-upload-pack /ICVSBG/CHUDC/AP/E04/SDK2.5/aosp/platform/frameworks/base (jenkins_dl)":
at org.apache.sshd.common.session.helpers.AbstractSession.enqueuePendingPacket(AbstractSession.java:928)
- waiting to lock <0x00007f0f679f2d20> (a java.util.LinkedList)
at org.apache.sshd.common.session.helpers.AbstractSession.writePacket(AbstractSession.java:882)
at org.apache.sshd.common.channel.AbstractChannel.writePacket(AbstractChannel.java:774)
at org.apache.sshd.common.channel.throttle.DefaultChannelStreamWriter.writeData(DefaultChannelStreamWriter.java:46)
at org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:251)
- locked <0x00007f0a08bbdf68> (a org.apache.sshd.common.channel.ChannelOutputStream)
at org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:150)
- locked <0x00007f0a08bbdf68> (a org.apache.sshd.common.channel.ChannelOutputStream)
at org.eclipse.jgit.util.io.TimeoutOutputStream.write(TimeoutOutputStream.java:92)
at org.eclipse.jgit.transport.UploadPack$ResponseBufferedOutputStream.write(UploadPack.java:2478)
at org.eclipse.jgit.transport.SideBandOutputStream.writeBuffer(SideBandOutputStream.java:141)
at org.eclipse.jgit.transport.SideBandOutputStream.write(SideBandOutputStream.java:120)
at org.eclipse.jgit.internal.storage.pack.PackOutputStream.write(PackOutputStream.java:99)
at org.eclipse.jgit.internal.storage.file.Pack.copyAsIs2(Pack.java:554)
at org.eclipse.jgit.internal.storage.file.Pack.copyAsIs(Pack.java:389)
at org.eclipse.jgit.internal.storage.file.WindowCursor.copyObjectAsIs(WindowCursor.java:188)
at org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjectImpl(PackWriter.java:1768)
at org.eclipse.jgit.internal.storage.pack.PackWriter.writeObject(PackWriter.java:1745)
at org.eclipse.jgit.internal.storage.pack.PackOutputStream.writeObject(PackOutputStream.java:138)
at org.eclipse.jgit.internal.storage.file.WindowCursor.writeObjects(WindowCursor.java:196)
at org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1733)
at org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1720)
at org.eclipse.jgit.internal.storage.pack.PackWriter.writePack(PackWriter.java:1270)
at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:2419)
at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:2248)
at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:1094)
at org.eclipse.jgit.transport.UploadPack.uploadWithExceptionPropagation(UploadPack.java:858)
at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:766)
at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:101)
at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:109)
at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:33)
at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:74)
at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:490)
- locked <0x00007f0a148d3c20> (a com.google.gerrit.sshd.BaseCommand$TaskThunk)
at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
at java.util.concurrent.Executors$RunnableAdapter.call(java...@11.0.17/Executors.java:515)
at java.util.concurrent.FutureTask.run(java...@11.0.17/FutureTask.java:264)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java...@11.0.17/ScheduledThreadPoolExecutor.java:304)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:612)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java...@11.0.17/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java...@11.0.17/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java...@11.0.17/Thread.java:829)

Found 1 deadlock.

[Luca Milanesio]

Hi Andy,

[Andy]

Hi  Luca， We use gerrit 3.4.8

[Matthias Sohn]

On Tue, Jul 22, 2025 at 2:12 PM Andy <vs33...@gmail.com> wrote:

Hi  Luca， We use gerrit 3.4.8

Community support for this version ended in Nov 2022.

You should consider updating to a supported version, 3.10.7, 3.11.4 or 3.12.1.

See https://endoflife.date/gerrit

 

在2025年7月22日星期二 UTC+8 19:40:51<Luca Milanesio> 写道：

Hi Andy,


> On 22 Jul 2025, at 08:54, Andy <vs33...@gmail.com> wrote:
>
> when use repo sync gerrit code, We often encounter issues where the team gets stuck while fetching code

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/445259cd-a215-43b5-8569-fa1683946b30n%40googlegroups.com.