Gerrit 2.2.2.1 and 2.3 Jetty exceptions and failures

658 views
Skip to first unread message

Andrew Jeffery

unread,
Apr 12, 2012, 10:13:08 PM4/12/12
to Repo and Gerrit Discussion
Hi all,

I've been struggling a bit recently with Gerrit's bundled Jetty
instance. A number of things have been happening, including (from
logs/error_log):

---
(note the timestamps)
[2012-04-13 11:11:45,403] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,403] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
[2012-04-13 11:11:45,404] WARN org.eclipse.jetty.util.log : Dispatched Failed!
---

and

---
[2012-04-13 11:13:11,812] WARN org.eclipse.jetty.util.log : EXCEPTION
java.lang.IllegalMonitorStateException
at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(Unknown Source)
at java.util.concurrent.locks.ReentrantLock.unlock(Unknown Source)
at java.util.concurrent.ArrayBlockingQueue.size(Unknown Source)
at org.eclipse.jetty.util.thread.QueuedThreadPool.dispatch(QueuedThreadPool.java:329)
at org.eclipse.jetty.server.nio.SelectChannelConnector$1.dispatch(SelectChannelConnector.java:73)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:561)
at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:219)
at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:121)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:841)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:450)
at java.lang.Thread.run(Unknown Source)
---

The stack trace happens reasonably regularly. Probably related to both
of the above is that the Gerrit web interface hangs and the process hogs
100% CPU until it's sent a SIGKILL.

The Gerrit instance is running in an OpenVZ container with 2GB of RAM
and 12GB of disk on a Proxmox[1] server. Gerrit is the only process
which generates any noticeable load inside this container (which is a
Debian environment) and the server itself is running Linux 2.6.32-6-pve
(a proxmox patched kernel). Gerrit is running in an Oracle Java VM,
version 1.6.0_31.

I haven't yet tried running a standalone Jetty and running Gerrit inside
that... is it worth a shot?

Cheers,

Andrew

[1] http://www.proxmox.com/products/proxmox-ve

signature.asc

Shawn Pearce

unread,
Apr 13, 2012, 12:36:05 AM4/13/12
to andrew....@ultra-avalon.com, Repo and Gerrit Discussion
On Thu, Apr 12, 2012 at 19:13, Andrew Jeffery
<andrew....@ultra-avalon.com> wrote:
> I've been struggling a bit recently with Gerrit's bundled Jetty
> instance. A number of things have been happening, including (from
> logs/error_log):
>
> ---
> (note the timestamps)
> [2012-04-13 11:11:45,403] WARN  org.eclipse.jetty.util.log : Dispatched Failed!
> [2012-04-13 11:11:45,403] WARN  org.eclipse.jetty.util.log : Dispatched Failed!

This is a known problem with Jetty. I'm not certain if they have fixed
this in newer versions.

I think what happens is a client disconnects ungracefully while Jetty
has de-selected its file descriptor for reading. This generates an
error condition on the socket, but the error cannot be delivered until
Jetty re-selects the file descriptor for reading. Its selecting for
writing, so Jetty is woken, but there is no event to process... so
Jetty gets annoyed, logs this warning, and tries again with the same
result.

Andrew Jeffery

unread,
Apr 13, 2012, 6:54:06 AM4/13/12
to Shawn Pearce, Repo and Gerrit Discussion
Hi Shawn,

Hmm... Okay - so I'm thinking of running Gerrit inside a tomcat instance
as both of the issues appear Jetty specific at this point. Any pointers
or thoughts on this? It's nice how easy it is to use the built-in Jetty
code, I'll miss that if I have to move to tomcat permanently!

Any ideas on the IllegalMonitorStateException? Do you suggest I take
that up with the Jetty project?

Cheers,

Andrew

signature.asc

Shawn Pearce

unread,
Apr 13, 2012, 10:09:36 AM4/13/12
to andrew....@ultra-avalon.com, Repo and Gerrit Discussion

This is sad, searching for `jetty "Dispatched Failed!"` on Google
basically just points to a ton of threads about Gerrit. I was hoping
to find a release notes document suggesting they fixed it in a more
recent version. :-(

Tomcat might not have this bug. GoogleServletEngine doesn't have this
bug (it has others that make it "fun" but not this one). We could look
at changing the embedded web server out for a different container. GSE
has the downside that it can only serve from "/" if I recall, which
makes it harder to use in a reverse proxy situation where you want to
serve from "/r/" or something.

I chose to embed Jetty into Gerrit eons ago because I hated
configuring Tomcat. So I'm really not the right person to ask about
how to setup Gerrit under Tomcat.

Andrew Jeffery

unread,
Apr 29, 2012, 8:15:41 PM4/29/12
to Repo and Gerrit Discussion
Just as a follow-up, we have a couple of Jenkins instances talking to
gerrit via the Gerrit Trigger plugin. It appears that after upgrading
each of the Jenkins instances and the trigger plugin that the issues
have cleared up. The versions we're now running are (in case it helps
anyone in the future):

* Gerrit 2.3
* Jenkins 1.461
* Jenkins GIT plugin 1.1.7
* Gerrit Trigger 2.5.1

Andrew
signature.asc
Reply all
Reply to author
Forward
0 new messages