Troubleshooting when hooks do not run

[Will Saxon]

Hello,

We encountered a situation this morning where none of our gerrit hooks were being executed at all. We run gerrit 2.5.0 using tomcat 7.0.29. Tomcat was restarted due to a system restart at 7:30AM without any logged errors, but no hooks were run until we discovered the problem and restarted tomcat again at 1:30pm.

Typically if there are hook script errors we see stderr logged in the catalina.out file. There were no hook errors logged today. Running the hooks by hand worked fine, so there were no problems with the hooks themselves or the changes being handled. The web app and ssh daemon seemed to be operating normally.

There was only one log entry between initial startup at 7:30AM and the restart at 1:30PM, but it doesn't look like it should be related at all:

[2013-02-05 09:38:18,439] WARN  org.apache.sshd.server.session.ServerSession : Exception caught

java.io.IOException: Connection reset by peer

        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)

        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)

        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:218)

        at sun.nio.ch.IOUtil.read(IOUtil.java:191)

        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)

        at org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:280)

        at org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:44)

        at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:677)

        at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:651)

        at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:640)

        at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:67)

        at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1111)

        at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

        at java.lang.Thread.run(Thread.java:722)

 

Just an aborted SSH connection, which we've paired with some auth errors by a new user.

So what would cause hooks not to fire at all, and what can we do to prevent these possibilities?

Thanks,

-Will