CometD deadlock issue

110 views
Skip to first unread message

Neil

unread,
Nov 1, 2011, 2:37:57 PM11/1/11
to cometd-users
We are facing deadlock issue with cometd, didn’t find anything on this
issue on Jetty or CometD forums.
Hope you can show some pointers. Thank you.


Deadlock details:

Found one Java-level deadlock:
=============================
"btpool0-777":
waiting to lock monitor 0x0000000004de4b70 (object
0xfffffd7f57168c40, a org.eclipse.jetty.util.ArrayQueue),
which is held by "btpool0-591"

"btpool0-591":
waiting to lock monitor 0x0000000004de48d0 (object
0xfffffd7f7e46d820, a
org.eclipse.jetty.continuation.FauxContinuation),
which is held by "btpool0-602"

"btpool0-602":
waiting to lock monitor 0x0000000004de4b70 (object
0xfffffd7f57168c40, a org.eclipse.jetty.util.ArrayQueue),
which is held by "btpool0-591"

Java stack information for the threads listed above:

===================================================

"btpool0-777":
at
org.cometd.server.ServerSessionImpl.doDeliver(ServerSessionImpl.java:
235)
- waiting to lock <0xfffffd7f57168c40> (a
org.eclipse.jetty.util.ArrayQueue)
at
org.cometd.server.BayeuxServerImpl.doPublish(BayeuxServerImpl.java:
784)
at
org.cometd.server.ServerChannelImpl.publish(ServerChannelImpl.java:
217)
at
org.cometd.server.BayeuxServerImpl.handle(BayeuxServerImpl.java:604)
at
org.cometd.server.transport.LongPollingTransport.handle(LongPollingTransport.java:
212)
at org.cometd.server.CometdServlet.service(CometdServlet.java:
152)
at org.cometd.server.CometdServlet.service(CometdServlet.java:
121)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1097)
at
org.eclipse.jetty.continuation.ContinuationFilter.doFilter(ContinuationFilter.java:
92)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1088)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:
360)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:
216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:
181)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:
729)
at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:
206)
at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:
114)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:
152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:
505)
at org.mortbay.jetty.HttpConnection
$RequestHandler.content(HttpConnection.java:843)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:647)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:
211)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:
380)
at org.mortbay.jetty.bio.SocketConnector
$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.QueuedThreadPool
$PoolThread.run(QueuedThreadPool.java:488)

"btpool0-591":
at
org.eclipse.jetty.continuation.FauxContinuation.resume(FauxContinuation.java:
186)
- waiting to lock <0xfffffd7f7e46d820> (a
org.eclipse.jetty.continuation.FauxContinuation)
at org.cometd.server.transport.LongPollingTransport
$LongPollScheduler.schedule(LongPollingTransport.java:464)
at
org.cometd.server.ServerSessionImpl.flush(ServerSessionImpl.java:471)
- locked <0xfffffd7f57168c40> (a
org.eclipse.jetty.util.ArrayQueue)
at
org.cometd.server.ServerSessionImpl.doDeliver(ServerSessionImpl.java:
246)
at
org.cometd.server.BayeuxServerImpl.doPublish(BayeuxServerImpl.java:
784)
at
org.cometd.server.ServerChannelImpl.publish(ServerChannelImpl.java:
217)
at
org.cometd.server.BayeuxServerImpl.handle(BayeuxServerImpl.java:604)
at
org.cometd.server.transport.LongPollingTransport.handle(LongPollingTransport.java:
212)
at org.cometd.server.CometdServlet.service(CometdServlet.java:
152)
at org.cometd.server.CometdServlet.service(CometdServlet.java:
121)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1097)
at
org.eclipse.jetty.continuation.ContinuationFilter.doFilter(ContinuationFilter.java:
92)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1088)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:
360)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:
216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:
181)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:
729)
at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:
206)
at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:
114)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:
152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:
505)
at org.mortbay.jetty.HttpConnection
$RequestHandler.content(HttpConnection.java:843)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:647)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:
211)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:
380)
at org.mortbay.jetty.bio.SocketConnector
$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.QueuedThreadPool
$PoolThread.run(QueuedThreadPool.java:488)

"btpool0-602":
at
org.cometd.server.ServerSessionImpl.setScheduler(ServerSessionImpl.java:
431)
- waiting to lock <0xfffffd7f57168c40> (a
org.eclipse.jetty.util.ArrayQueue)
at org.cometd.server.transport.LongPollingTransport
$LongPollScheduler.onTimeout(LongPollingTransport.java:484)
at
org.eclipse.jetty.continuation.FauxContinuation.onTimeout(FauxContinuation.java:
76)
at
org.eclipse.jetty.continuation.FauxContinuation.expire(FauxContinuation.java:
365)
at
org.eclipse.jetty.continuation.FauxContinuation.fauxSuspend(FauxContinuation.java:
419)
at
org.eclipse.jetty.continuation.FauxContinuation.exit(FauxContinuation.java:
325)
- locked <0xfffffd7f7e46d820> (a
org.eclipse.jetty.continuation.FauxContinuation)
at
org.eclipse.jetty.continuation.ContinuationFilter.doFilter(ContinuationFilter.java:
103)
at org.mortbay.jetty.servlet.ServletHandler
$CachedChain.doFilter(ServletHandler.java:1088)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:
360)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:
216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:
181)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:
729)
at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:
206)
at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:
114)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:
152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:
505)
at org.mortbay.jetty.HttpConnection
$RequestHandler.content(HttpConnection.java:843)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:647)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:
211)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:
380)
at org.mortbay.jetty.bio.SocketConnector
$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.QueuedThreadPool
$PoolThread.run(QueuedThreadPool.java:488)

Simone Bordet

unread,
Nov 1, 2011, 4:06:13 PM11/1/11
to cometd...@googlegroups.com
Hi,

On Tue, Nov 1, 2011 at 19:37, Neil <shanbhag...@gmail.com> wrote:
> We are facing deadlock issue with cometd, didn’t find anything on this
> issue on Jetty or CometD forums.
> Hope you can show some pointers. Thank you.

What exact version of CometD and Jetty are you using ?

Simon
--
http://cometd.org
http://intalio.com
http://bordet.blogspot.com
----
Finally, no matter how good the architecture and design are,
to deliver bug-free software with optimal performance and reliability,
the implementation technique must be flawless.   Victoria Livschitz

Neil

unread,
Nov 1, 2011, 4:12:14 PM11/1/11
to cometd-users
This appears to be happening when a second client request is received
at the same time that the long poll is timeing out. The Continuation
timeout synchronizes on itself and then calls the LongPollScheduler
timeout method. LongPollScheduler calls setScheduler on the
ServerSessionImpl which tries to lock the queue. The second client
request is trying to flush the queue of messages back to the client so
it first synchronizes on the queue. Resume is then called on the
continuation for the long poll request which synchronizes on itself.

One thought is that a check should be added to the
LongPollScheduler.schedule method to check and see if the Continuation
has expired before calling resume. This would allow the timeout to
complete its processing and return the long poll back to the client
and prevent the deadlock.

change:
public void schedule()
{
decBrowserId();
_continuation.resume();
}
to:
public void schedule()
{
decBrowserId();
if (!_continuation.isExpired()) {
_continuation.resume();

Neil

unread,
Nov 1, 2011, 4:15:18 PM11/1/11
to cometd-users
We are using Jetty 6.1.1 and CometD 2.1.1 versions.

On Nov 1, 3:06 pm, Simone Bordet <sbor...@intalio.com> wrote:
> Hi,
>
> On Tue, Nov 1, 2011 at 19:37, Neil <shanbhag.neel...@gmail.com> wrote:
> > We are facing deadlock issue with cometd, didn’t find anything on this
> > issue on Jetty or CometD forums.
> > Hope you can show some pointers. Thank you.
>
> What exact version of CometD and Jetty are you using ?
>
> Simon
> --http://cometd.orghttp://intalio.comhttp://bordet.blogspot.com

Simone Bordet

unread,
Nov 1, 2011, 4:25:25 PM11/1/11
to cometd...@googlegroups.com
Hi,

On Tue, Nov 1, 2011 at 21:15, Neil <shanbhag...@gmail.com> wrote:
> We are using Jetty 6.1.1 and CometD 2.1.1 versions.

These are both pretty old.
Jetty is on 6.1.26 and CometD at 2.3.1.
I'd suggest also to move to Jetty 7, as Jetty 6 is in maintenance mode only.

Your suggested fix is not completely right, since the condition of
whether the continuation is not expired is not enough.

I am not sure your setup it right, also. It seems that you're using
the SocketConnector, which is not the recommended solution for
scalability of CometD (SelectChannelConnector should be used instead).
And furthermore, you're using faux continuations, even if you're using
Jetty6, and that is really not correct. This is probably due to the
fact that you're missing the jetty continuation classes in your
webapp.

Having said that, the bug is real, although it is exposed by a non
optimal Jetty/CometD configuration.

Please file a bug at http://bugs.cometd.org.

Simon
--
http://cometd.org
http://intalio.com

Neil

unread,
Nov 3, 2011, 6:12:10 PM11/3/11
to cometd-users
Thank you Simon.

We changed SocketConnector to SelectChannelConnector, this change made
jetty to use Jetty6continuation instead of Faux Continuation!
This is because, with SocketConnector, __waitingContinuation is null,
and in ContinuationSupport.getContinuation() method, FauxContinuation
is created with following code.

if (__jetty6)
{
Object
c=request.getAttribute("org.mortbay.jetty.ajax.Continuation");
try
{
if (c==null || __waitingContinuation==null ||
__waitingContinuation.isInstance(c))
continuation=new FauxContinuation(request);
else
continuation=
__newJetty6Continuation.newInstance(request,c);

request.setAttribute(Continuation.ATTRIBUTE,continuation);
return continuation;
}
catch(Exception e)
{
throw new RuntimeException(e);
}
}


On Nov 1, 3:25 pm, Simone Bordet <sbor...@intalio.com> wrote:
> Hi,
>
> On Tue, Nov 1, 2011 at 21:15, Neil <shanbhag.neel...@gmail.com> wrote:
> > We are using Jetty 6.1.1 and CometD 2.1.1 versions.
>
> These are both pretty old.
> Jetty is on 6.1.26 and CometD at 2.3.1.
> I'd suggest also to move to Jetty 7, as Jetty 6 is in maintenance mode only.
>
> Your suggested fix is not completely right, since the condition of
> whether the continuation is not expired is not enough.
>
> I am not sure your setup it right, also. It seems that you're using
> the SocketConnector, which is not the recommended solution for
> scalability of CometD (SelectChannelConnector should be used instead).
> And furthermore, you're using faux continuations, even if you're using
> Jetty6, and that is really not correct. This is probably due to the
> fact that you're missing the jetty continuation classes in your
> webapp.
>
> Having said that, the bug is real, although it is exposed by a non
> optimal Jetty/CometD configuration.
>
> Please file a bug athttp://bugs.cometd.org.
>
> Simon
> --http://cometd.orghttp://intalio.comhttp://bordet.blogspot.com
Reply all
Reply to author
Forward
0 new messages