Gerrit 2.10.3.1 hanged on SSHd

Remy Bohmer

unread,

Apr 21, 2015, 3:11:39 AM4/21/15

to Repo and Gerrit Discussion

Hi,

This morning our Gerrit server froze on 2.10.3.1 which it never did
before like this (neither on 2.10.1). The web-interface was dead for
the users, and following stacktrace was in the logs when it happened.

Any idea?

Kind regards,

Remy

[2015-04-21 07:38:43,931] ERROR com.google.gerrit.sshd.BaseCommand :
Internal server error (user <some-user> account 287) during
git-upload-pack '/<some-git-project>'
org.apache.sshd.common.channel.WindowClosedException: Already closed
at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:163)
at org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:116)
at org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:84)
at org.eclipse.jgit.util.io.TimeoutOutputStream.write(TimeoutOutputStream.java:113)
at org.eclipse.jgit.transport.SideBandOutputStream.writeBuffer(SideBandOutputStream.java:171)
at org.eclipse.jgit.transport.SideBandOutputStream.write(SideBandOutputStream.java:151)
at org.eclipse.jgit.internal.storage.pack.PackOutputStream.write(PackOutputStream.java:126)
at org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs2(PackFile.java:535)
at org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs(PackFile.java:359)
at org.eclipse.jgit.internal.storage.file.WindowCursor.copyObjectAsIs(WindowCursor.java:190)
at org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjectImpl(PackWriter.java:1528)
at org.eclipse.jgit.internal.storage.pack.PackWriter.writeObject(PackWriter.java:1505)
at org.eclipse.jgit.internal.storage.pack.PackOutputStream.writeObject(PackOutputStream.java:164)
at org.eclipse.jgit.internal.storage.file.WindowCursor.writeObjects(WindowCursor.java:196)
at org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1493)
at org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1481)
at org.eclipse.jgit.internal.storage.pack.PackWriter.writePack(PackWriter.java:1036)
at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:1417)
at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:1271)
at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:717)
at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:628)
at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:80)
at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:101)
at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)
at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:70)
at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:442)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:370)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
[2015-04-21 07:38:55,690] WARN
com.google.gerrit.sshd.GerritServerSession : Exception caught
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225)
at sun.nio.ch.IOUtil.read(IOUtil.java:198)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
at org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:302)
at org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:45)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:694)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:668)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:657)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:67)
at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1121)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
[2015-04-21 07:39:02,074] WARN
com.google.gerrit.sshd.GerritServerSession : Exception caught
org.apache.sshd.common.SshException: Received 97 on unknown channel 0
at org.apache.sshd.common.session.AbstractConnectionService.getChannel(AbstractConnectionService.java:301)
at org.apache.sshd.common.session.AbstractConnectionService.channelClose(AbstractConnectionService.java:262)
at org.apache.sshd.common.session.AbstractConnectionService.process(AbstractConnectionService.java:171)
at org.apache.sshd.common.session.AbstractSession.doHandleMessage(AbstractSession.java:431)
at org.apache.sshd.common.session.AbstractSession.handleMessage(AbstractSession.java:326)
at org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:780)
at org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:308)
at org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:54)
at org.apache.sshd.common.io.mina.MinaService.messageReceived(MinaService.java:105)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:854)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:48)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:943)
at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:535)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:714)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:668)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:657)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:67)
at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1121)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
[2015-04-21 07:39:53,056] WARN
com.google.gerrit.sshd.GerritServerSession : Exception caught
org.apache.sshd.common.SshException: Received 97 on unknown channel 0
at org.apache.sshd.common.session.AbstractConnectionService.getChannel(AbstractConnectionService.java:301)
at org.apache.sshd.common.session.AbstractConnectionService.channelClose(AbstractConnectionService.java:262)
at org.apache.sshd.common.session.AbstractConnectionService.process(AbstractConnectionService.java:171)
at org.apache.sshd.common.session.AbstractSession.doHandleMessage(AbstractSession.java:431)
at org.apache.sshd.common.session.AbstractSession.handleMessage(AbstractSession.java:326)
at org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:780)
at org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:308)
at org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:54)
at org.apache.sshd.common.io.mina.MinaService.messageReceived(MinaService.java:105)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:854)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:48)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:943)
at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:535)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:714)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:668)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:657)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:67)
at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1121)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
[2015-04-21 07:40:03,617] WARN
com.google.gerrit.sshd.GerritServerSession : Exception caught
org.apache.sshd.common.SshException: Received 97 on unknown channel 0
at org.apache.sshd.common.session.AbstractConnectionService.getChannel(AbstractConnectionService.java:301)
at org.apache.sshd.common.session.AbstractConnectionService.channelClose(AbstractConnectionService.java:262)
at org.apache.sshd.common.session.AbstractConnectionService.process(AbstractConnectionService.java:171)
at org.apache.sshd.common.session.AbstractSession.doHandleMessage(AbstractSession.java:431)
at org.apache.sshd.common.session.AbstractSession.handleMessage(AbstractSession.java:326)
at org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:780)
at org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:308)
at org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:54)
at org.apache.sshd.common.io.mina.MinaService.messageReceived(MinaService.java:105)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:854)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:48)
at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:943)
at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542)
at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:535)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:714)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:668)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:657)
at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:67)
at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1121)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
[2015-04-21 07:40:38,786] WARN
org.eclipse.jetty.util.thread.QueuedThreadPool :
HTTP{STARTED,5<=9<=200,i=0,q=50} rejected
org.eclipse.jetty.io.AbstractConnection$2@36f85d71
[2015-04-21 07:40:38,787] WARN org.eclipse.jetty.io.SelectorManager :
Could not process key for channel
java.nio.channels.SocketChannel[connected local=/<some-ip>:8080
remote=/<some-ip>:50068]
java.util.concurrent.RejectedExecutionException:
org.eclipse.jetty.io.AbstractConnection$2@36f85d71
at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:361)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:373)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:375)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:109)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:572)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:543)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:722)
[2015-04-21 07:40:39,089] WARN
org.eclipse.jetty.util.thread.QueuedThreadPool :
HTTP{STARTED,5<=9<=200,i=0,q=50} rejected
org.eclipse.jetty.io.AbstractConnection$2@a4a8052
[2015-04-21 07:40:39,090] WARN org.eclipse.jetty.io.SelectorManager :
Could not process key for channel
java.nio.channels.SocketChannel[connected local=/<some-ip>:8080
remote=/<some-ip>:50956]
java.util.concurrent.RejectedExecutionException:
org.eclipse.jetty.io.AbstractConnection$2@a4a8052
at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:361)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:373)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:375)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:109)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:572)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:543)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:722)
[2015-04-21 07:40:39,489] WARN
org.eclipse.jetty.util.thread.QueuedThreadPool :
HTTP{STARTED,5<=9<=200,i=0,q=50} rejected
org.eclipse.jetty.io.AbstractConnection$2@1e9053c3
[2015-04-21 07:40:39,490] WARN org.eclipse.jetty.io.SelectorManager :
Could not process key for channel
java.nio.channels.SocketChannel[connected local=/<some-ip>:8080
remote=/<some-ip>:50069]
java.util.concurrent.RejectedExecutionException:
org.eclipse.jetty.io.AbstractConnection$2@1e9053c3
at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:361)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:373)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:375)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:109)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:572)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:543)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:722)
[2015-04-21 07:40:39,828] WARN
org.eclipse.jetty.util.thread.QueuedThreadPool :
HTTP{STARTED,5<=9<=200,i=0,q=50} rejected
org.eclipse.jetty.io.AbstractConnection$2@1958b3ac
[2015-04-21 07:40:39,828] WARN org.eclipse.jetty.io.SelectorManager :
Could not process key for channel
java.nio.channels.SocketChannel[connected local=/<some-ip>:8080
remote=/<some-ip>:50071]
java.util.concurrent.RejectedExecutionException:
org.eclipse.jetty.io.AbstractConnection$2@1958b3ac
at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:361)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:373)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:375)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:109)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:572)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:543)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:722)
[2015-04-21 07:40:40,503] WARN
org.eclipse.jetty.util.thread.QueuedThreadPool :
HTTP{STARTED,5<=9<=200,i=0,q=50} rejected
org.eclipse.jetty.io.AbstractConnection$2@68e82916
[2015-04-21 07:40:40,504] WARN org.eclipse.jetty.io.SelectorManager :
Could not process key for channel
java.nio.channels.SocketChannel[connected local=/<some-ip>:8080
remote=/<some-ip>:50073]
java.util.concurrent.RejectedExecutionException:
org.eclipse.jetty.io.AbstractConnection$2@68e82916
at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:361)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:373)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:375)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:109)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:572)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:543)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:722)
[2015-04-21 07:40:48,139] WARN
org.eclipse.jetty.util.thread.QueuedThreadPool :
HTTP{STARTED,5<=9<=200,i=0,q=50} rejected
org.eclipse.jetty.io.AbstractConnection$2@7c7d32ee
[2015-04-21 07:40:48,139] WARN org.eclipse.jetty.io.SelectorManager :
Could not process key for channel
java.nio.channels.SocketChannel[connected local=/<some-ip>:8080
remote=/<some-ip>:50075]
java.util.concurrent.RejectedExecutionException:
org.eclipse.jetty.io.AbstractConnection$2@7c7d32ee
at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:361)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:373)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:375)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:109)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:572)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:543)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:722)
[2015-04-21 07:40:48,146] WARN
org.eclipse.jetty.util.thread.QueuedThreadPool :
HTTP{STARTED,5<=9<=200,i=0,q=50} rejected
org.eclipse.jetty.io.AbstractConnection$2@70dcdc51
[2015-04-21 07:40:48,147] WARN org.eclipse.jetty.io.SelectorManager :
Could not process key for channel
java.nio.channels.SocketChannel[connected local=/<some-ip>:8080
remote=/<some-ip>:50076]
java.util.concurrent.RejectedExecutionException:
org.eclipse.jetty.io.AbstractConnection$2@70dcdc51
at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:361)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:373)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:375)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:109)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:572)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:543)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:722)
[2015-04-21 07:40:48,167] WARN
org.eclipse.jetty.util.thread.QueuedThreadPool :
HTTP{STARTED,5<=9<=200,i=0,q=50} rejected
org.eclipse.jetty.io.AbstractConnection$2@1d47f388
[2015-04-21 07:40:48,167] WARN org.eclipse.jetty.io.SelectorManager :
Could not process key for channel
java.nio.channels.SocketChannel[connected local=/<some-ip>:8080
remote=/<some-ip>:50079]
java.util.concurrent.RejectedExecutionException:
org.eclipse.jetty.io.AbstractConnection$2@1d47f388
at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:361)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:373)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:375)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:109)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:572)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:543)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:722)
[2015-04-21 07:40:50,383] WARN
org.eclipse.jetty.util.thread.QueuedThreadPool :
HTTP{STARTED,5<=9<=200,i=0,q=50} rejected
org.eclipse.jetty.io.AbstractConnection$2@5b7164c1
[2015-04-21 07:40:50,383] WARN org.eclipse.jetty.io.SelectorManager :
Could not process key for channel
java.nio.channels.SocketChannel[connected local=/<some-ip>:8080
remote=/<some-ip>:52793]
java.util.concurrent.RejectedExecutionException:
org.eclipse.jetty.io.AbstractConnection$2@5b7164c1
at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:361)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:373)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:375)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:109)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:572)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:543)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:722)
[2015-04-21 07:40:50,402] WARN
org.eclipse.jetty.util.thread.QueuedThreadPool :
HTTP{STARTED,5<=9<=200,i=0,q=50} rejected
org.eclipse.jetty.io.AbstractConnection$2@3c43e30a
[2015-04-21 07:40:50,403] WARN org.eclipse.jetty.io.SelectorManager :
Could not process key for channel
java.nio.channels.SocketChannel[connected local=/<some-ip>:8080
remote=/<some-ip>:52792]
java.util.concurrent.RejectedExecutionException:
org.eclipse.jetty.io.AbstractConnection$2@3c43e30a
at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:361)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:373)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:375)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:109)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:572)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:543)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:722)
[2015-04-21 07:40:50,423] WARN
org.eclipse.jetty.util.thread.QueuedThreadPool :
HTTP{STARTED,5<=9<=200,i=0,q=50} rejected
org.eclipse.jetty.io.AbstractConnection$2@5ecaea14
[2015-04-21 07:40:50,423] WARN org.eclipse.jetty.io.SelectorManager :
Could not process key for channel
java.nio.channels.SocketChannel[connected local=/<some-ip>:8080
remote=/<some-ip>:52791]
java.util.concurrent.RejectedExecutionException:
org.eclipse.jetty.io.AbstractConnection$2@5ecaea14
at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:361)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:373)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:375)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:109)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:572)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:543)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:722)
[2015-04-21 07:40:50,742] WARN
org.eclipse.jetty.util.thread.QueuedThreadPool :
HTTP{STARTED,5<=9<=200,i=0,q=50} rejected
org.eclipse.jetty.io.AbstractConnection$2@2f3ee8fc
[2015-04-21 07:40:50,742] WARN org.eclipse.jetty.io.SelectorManager :
Could not process key for channel
java.nio.channels.SocketChannel[connected local=/<some-ip>:8080
remote=/<some-ip>:50074]
java.util.concurrent.RejectedExecutionException:
org.eclipse.jetty.io.AbstractConnection$2@2f3ee8fc
at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:361)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:373)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:375)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:109)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:572)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:543)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:722)
[2015-04-21 07:40:51,082] WARN
org.eclipse.jetty.util.thread.QueuedThreadPool :
HTTP{STARTED,5<=9<=200,i=0,q=50} rejected
org.eclipse.jetty.io.AbstractConnection$2@bcce78b
[2015-04-21 07:40:51,082] WARN org.eclipse.jetty.io.SelectorManager :
Could not process key for channel
java.nio.channels.SocketChannel[connected local=/<some-ip>:8080
remote=/<some-ip>:52802]
java.util.concurrent.RejectedExecutionException:
org.eclipse.jetty.io.AbstractConnection$2@bcce78b
at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:361)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:373)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:375)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:109)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:572)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:543)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:722)
[2015-04-21 07:40:51,767] WARN
org.eclipse.jetty.util.thread.QueuedThreadPool :
HTTP{STARTED,5<=9<=200,i=0,q=50} rejected
org.eclipse.jetty.io.AbstractConnection$2@5918b704
[2015-04-21 07:40:51,767] WARN org.eclipse.jetty.io.SelectorManager :
Could not process key for channel
java.nio.channels.SocketChannel[connected local=/<some-ip>:8080
remote=/<some-ip>:52804]
java.util.concurrent.RejectedExecutionException:
org.eclipse.jetty.io.AbstractConnection$2@5918b704
at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:361)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:373)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:375)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:109)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:572)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:543)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:722)
[2015-04-21 07:40:56,410] WARN
org.eclipse.jetty.util.thread.QueuedThreadPool :
HTTP{STARTED,5<=9<=200,i=0,q=50} rejected
org.eclipse.jetty.io.AbstractConnection$2@3bd31c85
[2015-04-21 07:40:56,411] WARN org.eclipse.jetty.io.SelectorManager :
Could not process key for channel
java.nio.channels.SocketChannel[connected local=/<some-ip>:8080
remote=/<some-ip>:55752]
java.util.concurrent.RejectedExecutionException:
org.eclipse.jetty.io.AbstractConnection$2@3bd31c85
at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:361)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:373)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:375)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:109)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:572)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:543)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:722)
[2015-04-21 07:40:56,412] WARN
org.eclipse.jetty.util.thread.QueuedThreadPool :
HTTP{STARTED,5<=9<=200,i=0,q=50} rejected
org.eclipse.jetty.io.AbstractConnection$2@55407f45
[2015-04-21 07:40:56,412] WARN org.eclipse.jetty.io.SelectorManager :
Could not process key for channel
java.nio.channels.SocketChannel[connected local=/<some-ip>:8080
remote=/<some-ip>:60489]
java.util.concurrent.RejectedExecutionException:
org.eclipse.jetty.io.AbstractConnection$2@55407f45
at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:361)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:373)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:375)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:109)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:572)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:543)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:722)
[2015-04-21 07:40:57,259] WARN
org.eclipse.jetty.util.thread.QueuedThreadPool :
HTTP{STARTED,5<=9<=200,i=0,q=50} rejected
org.eclipse.jetty.io.AbstractConnection$2@5da7477d
[2015-04-21 07:40:57,259] WARN org.eclipse.jetty.io.SelectorManager :
Could not process key for channel
java.nio.channels.SocketChannel[connected local=/<some-ip>:8080
remote=/<some-ip>:52808]
java.util.concurrent.RejectedExecutionException:
org.eclipse.jetty.io.AbstractConnection$2@5da7477d
at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:361)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:373)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:375)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:109)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:572)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:543)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:722)
[2015-04-21 07:40:57,314] WARN
org.eclipse.jetty.util.thread.QueuedThreadPool :
HTTP{STARTED,5<=9<=200,i=0,q=50} rejected
org.eclipse.jetty.io.AbstractConnection$2@2f7c4668
[2015-04-21 07:40:57,314] WARN org.eclipse.jetty.io.SelectorManager :
Could not process key for channel
java.nio.channels.SocketChannel[connected local=/<some-ip>:8080
remote=/<some-ip>:49190]
java.util.concurrent.RejectedExecutionException:
org.eclipse.jetty.io.AbstractConnection$2@2f7c4668
at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:361)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:373)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:375)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:109)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:572)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:543)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:722)
[2015-04-21 07:41:01,796] WARN
org.eclipse.jetty.util.thread.QueuedThreadPool :
HTTP{STARTED,5<=9<=200,i=0,q=50} rejected
org.eclipse.jetty.io.AbstractConnection$2@5abec088
[2015-04-21 07:41:01,796] WARN org.eclipse.jetty.io.SelectorManager :
Could not process key for channel
java.nio.channels.SocketChannel[connected local=/<some-ip>:8080
remote=/<some-ip>:49948]
java.util.concurrent.RejectedExecutionException:
org.eclipse.jetty.io.AbstractConnection$2@5abec088
at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:361)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:373)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
at org.eclipse.jetty.io.AbstractConnection$FillingState.onEnter(AbstractConnection.java:375)
at org.eclipse.jetty.io.AbstractConnection.next(AbstractConnection.java:267)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:557)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:82)
at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:109)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:572)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:543)
at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:722)
[2015-04-21 07:41:09,823] WARN
org.eclipse.jetty.util.thread.QueuedThreadPool :
HTTP{STARTED,5<=9<=200,i=0,q=50} rejected
org.eclipse.jetty.io.AbstractConnection$2@700bfb2a

Saša Živkov

unread,

Apr 21, 2015, 3:20:11 AM4/21/15

to Remy Bohmer, Repo and Gerrit Discussion

On Tue, Apr 21, 2015 at 9:11 AM, Remy Bohmer <li...@bohmer.net> wrote:

Hi,

This morning our Gerrit server froze on 2.10.3.1 which it never did
before like this (neither on 2.10.1). The web-interface was dead for
the users, and following stacktrace was in the logs when it happened.

Any idea?

Have you created a thread dump of the Gerrit process before restarting it?

If not, then please do that if/when it gets frozen again and post it here.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Martin Fick

unread,

Apr 21, 2015, 11:16:57 AM4/21/15

to repo-d...@googlegroups.com, Remy Bohmer

On Tuesday, April 21, 2015 09:11:33 AM Remy Bohmer wrote:
> Hi,
>
> This morning our Gerrit server froze on 2.10.3.1 which it
> never did before like this (neither on 2.10.1). The
> web-interface was dead for the users, and following
> stacktrace was in the logs when it happened.
>
> Any idea?
>
> Kind regards,
>
> Remy
>
>
> [2015-04-21 07:38:43,931] ERROR
> com.google.gerrit.sshd.BaseCommand : Internal server
> error (user <some-user> account 287) during
> git-upload-pack '/<some-git-project>'
> org.apache.sshd.common.channel.WindowClosedException:
> Already closed at
> org.apache.sshd.common.channel.Window.waitForSpace(Window

I think that stack trace shows up when users disconnect
during a jgit operation, I wouldn't read much into it.
Perhaps your server went into a "stop the world" java gc
mode for a while? Pehaps it was swapping during this
period? That can bring a server to its knees and prevent it
from responding. It is best to disable swap on any box
running a jvm that does not do manual gc. ;)

-Martin

--
The Qualcomm Innovation Center, Inc. is a member of Code
Aurora Forum, hosted by The Linux Foundation

Remy Bohmer

unread,

Apr 21, 2015, 4:11:45 PM4/21/15

to Martin Fick, Repo and Gerrit Discussion

Hi,

The first part of the trace can indeed be a disconnecting user.

Also: Many users complained about hanging fetch/push operations.
Before and after reboot.
All-in-all: We see a lot of stability issues via the SSH channel. This
afternoon we downgraded back to 2.10.2, and the problems have not been
seen since... It seems to run as stable as before the upgrade to
2.10.3.1...

The GC operation was scheduled for Saturday (weekly), so that should
not have been running at thia time.

Kind regards,

Remy

Saša Živkov

unread,

Apr 22, 2015, 6:44:45 AM4/22/15

to Remy Bohmer, Martin Fick, Repo and Gerrit Discussion

On Tue, Apr 21, 2015 at 10:11 PM, Remy Bohmer <li...@bohmer.net> wrote:

Hi,

The first part of the trace can indeed be a disconnecting user.

Also: Many users complained about hanging fetch/push operations.
Before and after reboot.
All-in-all: We see a lot of stability issues via the SSH channel. This

Again, when you see the hanging requests PLEASE make a thread dump

or several of them (waiting 10-20 seconds between).

In the $GERRIT_HOME/logs/gerrit.pid you have the process ID of the Gerrit process.

This command is what you need:

$ jstack $(cat gerrit.pid)

afternoon we downgraded back to 2.10.2, and the problems have not been
seen since... It seems to run as stable as before the upgrade to
2.10.3.1...

In the 2.10.3 the mina sshd library was (again) upgraded to the 0.14.0 version, see [1].

In the 2.10.2 we still had the 0.9.x version and I was convinced by several people

that the upgrade is safe :-|

[1] https://gerrit-documentation.storage.googleapis.com/ReleaseNotes/ReleaseNotes-2.10.3.html#_upgrades

Lundh, Gustaf

unread,

Apr 23, 2015, 9:16:16 AM4/23/15

to Saša Živkov, Remy Bohmer, Martin Fick, Repo and Gerrit Discussion

What’s the plan going forward? A mina sshd downgrade and a new 2.10 release?

Best regards

Gustaf

Saša Živkov

unread,

Apr 23, 2015, 9:21:01 AM4/23/15

to Lundh, Gustaf, Remy Bohmer, Martin Fick, Repo and Gerrit Discussion

On Thu, Apr 23, 2015 at 3:16 PM, Lundh, Gustaf <Gustaf...@sonymobile.com> wrote:

What’s the plan going forward? A mina sshd downgrade and a new 2.10 release?

I don't know yet. Without a thread dump of a hanging push I cannot tell if it is caused by

the mina sshd or by something else.

Remy Bohmer

unread,

Apr 23, 2015, 2:25:46 PM4/23/15

to Saša Živkov, Repo and Gerrit Discussion, Lundh, Gustaf, Martin Fick

Hi,

I may have some more info on this issue. We upgraded from 2.10.1 to 2.10.3.1 and then the problems started. To get rid of the problems fast in production, we downgraded back to version in between: 2.10.2. Except for the known issues with this 2.10.2 version we saw nothing new. However, the complaints on failing push/pulls restarted today. Looking deeper here it were the same users that all used an older git 1.8.2 client. Not sure how this is related here, but at least it made clear that these issues were not caused by the SSHd change in 2.10.3

This leaves open the freeze of the web server. The stacktrace at the bottom of my first mail shows something related here. I do not understand these...

So , for now I would not suspect the SSHd change. We will reinstall 2.10.3.1 in a week from now and see if we can reproduce the freeze...

Thanks and Kind regards,

Remy

Op 23 apr. 2015 15:20 schreef "Saša Živkov" <ziv...@gmail.com>:

sascha...@gmail.com

unread,

Jul 29, 2015, 10:11:10 AM7/29/15

to Repo and Gerrit Discussion, li...@bohmer.net

We are facing the same issue with 2.10.6. It looks like that one request blocks all other.

I hope the attachments help.

jstack.txt

sshd_log.txt

error_log.txt

Jack Huang

unread,

Jan 8, 2019, 10:10:20 PM1/8/19

to Repo and Gerrit Discussion

I also have the similar problem. I use gerrit in docker, the version is openfrontier/gerrit:2.15.3

when the users run git clone/pull/push command, it is always that the user must wait for it for a while, generally the users need to wait for it for serveral minutes.

and the logs of Gerrit server are as follows:

[2019-01-07 08:06:39,454] [sshd-SshServer[4eace42b]-nio2-thread-4] WARN org.apache.sshd.server.session.ServerConnectionService : Received SSH_MSG_CHANNEL_CLOSE on unknown channel 25

[2019-01-07 08:06:50,799] [sshd-SshServer[4eace42b]-nio2-thread-1] WARN org.apache.sshd.server.session.ServerConnectionService : Received SSH_MSG_CHANNEL_CLOSE on unknown channel 24

[2019-01-07 08:06:54,241] [sshd-SshServer[4eace42b]-nio2-thread-7] WARN org.apache.sshd.server.session.ServerConnectionService : Received SSH_MSG_CHANNEL_CLOSE on unknown channel 23

I want to know that do you have resloved this problem? or does the Gerrit server have a bug with sshd backend?

at org.eclipse.jetty.io.A...

Martin Fick

unread,

Jan 9, 2019, 1:12:18 PM1/9/19

to repo-d...@googlegroups.com, Jack Huang

On Tuesday, January 8, 2019 7:10:20 PM MST Jack Huang wrote:
> I also have the similar problem. I use gerrit in docker, the version is
> openfrontier/gerrit:2.15.3

Please try not to highjack someone else's thread.

> when the users run git clone/pull/push command, it is always that the user
> must wait for it for a while, generally the users need to wait for it for
> serveral minutes.
>
> and the logs of Gerrit server are as follows:
> [2019-01-07 08:06:39,454] [sshd-SshServer[4eace42b]-nio2-thread-4] WARN
> org.apache.sshd.server.session.ServerConnectionService : Received
> SSH_MSG_CHANNEL_CLOSE on unknown channel 25
> [2019-01-07 08:06:50,799] [sshd-SshServer[4eace42b]-nio2-thread-1] WARN
> org.apache.sshd.server.session.ServerConnectionService : Received
> SSH_MSG_CHANNEL_CLOSE on unknown channel 24
> [2019-01-07 08:06:54,241] [sshd-SshServer[4eace42b]-nio2-thread-7] WARN
> org.apache.sshd.server.session.ServerConnectionService : Received
> SSH_MSG_CHANNEL_CLOSE on unknown channel 23
>
>
> I want to know that do you have resloved this problem? or does the Gerrit
> server have a bug with sshd backend?

Are there other jobs/users using up all of your ssh threads? How many are
configured and what does ssh gerrit show-queue show?

Reply all

Reply to author

Forward