I've been struggling a bit recently with Gerrit's bundled Jetty
instance. A number of things have been happening, including (from
logs/error_log):
---
(note the timestamps)
[2012-04-13 11:11:45,403] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,403] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
---
and
---
[2012-04-13 11:13:11,812] WARN org.eclipse.jetty.util.log : EXCEPTION
java.lang.IllegalMonitorStateException
at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(Unknown Source)
at java.util.concurrent.locks.ReentrantLock.unlock(Unknown Source)
at java.util.concurrent.ArrayBlockingQueue.size(Unknown Source)
at org.eclipse.jetty.util.thread.QueuedThreadPool.dispatch(QueuedThreadPool.java:329)
at org.eclipse.jetty.server.nio.SelectChannelConnector$1.dispatch(SelectChannelConnector.java:73)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:561)
at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:219)
at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:121)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:841)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:450)
at java.lang.Thread.run(Unknown Source)
---
The stack trace happens reasonably regularly. Probably related to both
of the above is that the Gerrit web interface hangs and the process hogs
100% CPU until it's sent a SIGKILL.
The Gerrit instance is running in an OpenVZ container with 2GB of RAM
and 12GB of disk on a Proxmox[1] server. Gerrit is the only process
which generates any noticeable load inside this container (which is a
Debian environment) and the server itself is running Linux 2.6.32-6-pve
(a proxmox patched kernel). Gerrit is running in an Oracle Java VM,
version 1.6.0_31.
I haven't yet tried running a standalone Jetty and running Gerrit inside
that... is it worth a shot?
Cheers,
Andrew
This is a known problem with Jetty. I'm not certain if they have fixed
this in newer versions.
I think what happens is a client disconnects ungracefully while Jetty
has de-selected its file descriptor for reading. This generates an
error condition on the socket, but the error cannot be delivered until
Jetty re-selects the file descriptor for reading. Its selecting for
writing, so Jetty is woken, but there is no event to process... so
Jetty gets annoyed, logs this warning, and tries again with the same
result.
Hmm... Okay - so I'm thinking of running Gerrit inside a tomcat instance
as both of the issues appear Jetty specific at this point. Any pointers
or thoughts on this? It's nice how easy it is to use the built-in Jetty
code, I'll miss that if I have to move to tomcat permanently!
Any ideas on the IllegalMonitorStateException? Do you suggest I take
that up with the Jetty project?
Cheers,
Andrew
This is sad, searching for `jetty "Dispatched Failed!"` on Google
basically just points to a ton of threads about Gerrit. I was hoping
to find a release notes document suggesting they fixed it in a more
recent version. :-(
Tomcat might not have this bug. GoogleServletEngine doesn't have this
bug (it has others that make it "fun" but not this one). We could look
at changing the embedded web server out for a different container. GSE
has the downside that it can only serve from "/" if I recall, which
makes it harder to use in a reverse proxy situation where you want to
serve from "/r/" or something.
I chose to embed Jetty into Gerrit eons ago because I hated
configuring Tomcat. So I'm really not the right person to ask about
how to setup Gerrit under Tomcat.