Netty exception during ConnectionConfiguration.connect

[Jacobus Geluk]

Hi, I get the following exception when doing a connect. What could be cause of this?

java.lang.IllegalStateException: await*() in I/O thread causes a dead lock or sudden performance drop. Use addListener() instead or call await*() from a different thread. 
at org.jboss.netty.channel.DefaultChannelFuture.checkDeadLock(DefaultChannelFuture.java:343) ~[netty-3.5.8.Final.jar:na] 
at org.jboss.netty.channel.DefaultChannelFuture.awaitUninterruptibly(DefaultChannelFuture.java:255) ~[netty-3.5.8.Final.jar:na] 
at com.clarkparsia.stardog.snarl.client.AbstractClient.<init>(Unknown Source) ~[stardog-1.1.jar:na] 
at com.clarkparsia.stardog.snarl.client.SNARLClientImpl.<init>(Unknown Source) ~[stardog-1.1.jar:na] 
at com.clarkparsia.stardog.snarl.client.SNARLClientImpl$FastClassByGuice$fbba359a.newInstance(<generated>) ~[guice-3.0.jar:na] 
at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) ~[guice-3.0.jar:na] 
at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60) ~[guice-3.0.jar:na] 
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) ~[guice-3.0.jar:na] 
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) ~[guice-3.0.jar:na] 
at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978) ~[guice-3.0.jar:na] 
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024) ~[guice-3.0.jar:na] 
at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974) ~[guice-3.0.jar:na] 
at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:632) ~[guice-assistedinject-3.0.jar:na] 
at $Proxy39.createClient(Unknown Source) ~[na:na] 
at com.clarkparsia.stardog.snarl.client.SNARLDriver.connect(Unknown Source) ~[stardog-1.1.jar:na] 
at com.clarkparsia.stardog.api.DriverManager.getConnection(Unknown Source) ~[stardog-1.1.jar:na] 
at com.clarkparsia.stardog.api.ConnectionConfiguration.at(Unknown Source) ~[stardog-1.1.jar:na] 
at com.clarkparsia.stardog.api.ConnectionConfiguration.connect(Unknown Source) ~[stardog-1.1.jar:na]

[Mike Grove]

On Tue, Dec 11, 2012 at 3:29 PM, Jacobus Geluk <jacobu...@refinery29.com> wrote:

Hi, I get the following exception when doing a connect. What could be cause of this?

No, the cause is unclear since it's coming out of Netty.  The exception is thrown while the SNARL client is waiting on the connection establish event and is not an issue I've seen before, nor should it be the cause of any deadlock.

I'd first recommend using the correct version of Netty, your trace leads me to believe you're using 3.5.8 while Stardog ships, and is tested, with 3.5.2.  Try using 3.5.2 and see if the behavior persists.  If that does not resolve the issue, please provide a clear, concise description of what you're doing when the error occurs (with code, preferably) so that we can try to reproduce it locally.  If using the correct version of Netty works, I'd still like to know what you're doing to cause this issue so that I can allow for this issue in the event we upgrade to a version later than 3.5.2.

Cheers,

Mike

 

--
-- --
You received this message because you are subscribed to the C&P "Stardog" group.
To post to this group, send email to sta...@clarkparsia.com
To unsubscribe from this group, send email to
stardog+u...@clarkparsia.com
For more options, visit this group at
http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
 
 

[Robert Butler]

I actually think I know what this is. I ran into this in my own system while connection to Stardog. The issue is a check performed by the Netty pipeline. In my case I had:

HTTP Request -> Netty (my pipeline) -> my code -> Stardog client -> Netty (Stardog pipeline) -> SNARL protocol

The issue is that Netty thinks you are calling the await() method to wait for more data, however, this will cause deadlock since you are blocking the I/O thread while waiting. However, in my case, this isn't what I was doing since the two Netty pipelines are distinct and separate. The way I resolved it is by essentially introducing a "request handler" that would handle the request on a separate thread pool, making sure Netty saw two distinct threads accessing the pipelines instead of one.

Hope that helps,

- Robert

Robert Butler

Pancake Technology, LLC

972.861.0525

P.O. Box 271416
Flower Mound, TX 75027

@roberttheiv

http://www.linkedin.com/in/roberttheiv

[Mike Grove]

On Tue, Dec 11, 2012 at 3:58 PM, Robert Butler <rbu...@pancaketech.com> wrote:

I actually think I know what this is. I ran into this in my own system while connection to Stardog. The issue is a check performed by the Netty pipeline. In my case I had:

HTTP Request -> Netty (my pipeline) -> my code -> Stardog client -> Netty (Stardog pipeline) -> SNARL protocol

The issue is that Netty thinks you are calling the await() method to wait for more data, however, this will cause deadlock since you are blocking the I/O thread while waiting. However, in my case, this isn't what I was doing since the two Netty pipelines are distinct and separate. The way I resolved it is by essentially introducing a "request handler" that would handle the request on a separate thread pool, making sure Netty saw two distinct threads accessing the pipelines instead of one.

Does this only happen with the embedded server?

Cheers,

Mike

[Robert Butler]

No. In fact, I don't think it would happen with the embedded server. Personally, I have switched away from using Stardog in embedded mode.

This issue here is that the client connection code uses Netty. If the code calling the Stardog client libs additionally uses Netty to receive requests then you have a double Netty pipeline.

So, with the client you have:

HTTP Request -> Netty (my pipeline) -> my code -> Stardog client -> Netty (Stardog pipeline) -> SNARL protocol     |*--network-->*|     Stardog server

With embedded you have:

HTTP Request -> Netty (my pipeline) -> my code -> Stardog client -> direct memory calls to in-memory server (if I understand it correctly)

There is no second Netty pipeline in that case. My resolution was:

HTTP Request -> Netty (my pipeline) -> my code -> |*-- thread pool --*| -> my code -> Stardog client -> Netty (Stardog pipeline) -> SNARL protocol     |*--network-->*|   Stardog server

The Netty pipelines are now separated across thread boundaries and there is no conflict. As far as I understand it, there's no reason that the first case wouldn't work if Netty wasn't making the check but I'm not familiar enough with the internals of Netty to know for sure. At any rate, inserting the thread pool was a logical refactoring of my code that I thought was good architecturally anyway, so that's what I did.

Does that make sense?

- Robert

P.S. Sorry for the horrible grammar in my previous email. I wish I had proofed that a bit.

Robert Butler

Pancake Technology, LLC

972.861.0525

P.O. Box 271416
Flower Mound, TX 75027

@roberttheiv

http://www.linkedin.com/in/roberttheiv

[Mike Grove]

On Tue, Dec 11, 2012 at 4:20 PM, Robert Butler <rbu...@pancaketech.com> wrote:

No. In fact, I don't think it would happen with the embedded server. Personally, I have switched away from using Stardog in embedded mode.

This issue here is that the client connection code uses Netty. If the code calling the Stardog client libs additionally uses Netty to receive requests then you have a double Netty pipeline.

Ok, that makes sense given what I think they're deadlock detector is doing; there's a ThreadLocal that keeps the executor that they're using for the detection, so if you have two netty clients in the same thread, I think you'd always hit this problem.

Cheers,

Mike

[jacobu...@gmail.com]

Hmm, yes, could be my problem too then. I am using vert-x (http://vertx.io/) where I have an HTTP server "verticle" which sends a request via the vert-x event bus to a "Sparql Verticle" which uses the Stardog client to connect to a Stardog server via SNARL.

I can't downgrade netty to the same version that Stardog has been tested with since vert-x requires this newer version.

So this sounds as if I have to rethink this architecture a little.

[jacobu...@gmail.com]

I have a deeper stack trace here that could explain what is going on, as you can see there are calls to netty from Stardog and from Vert-x on the same stack:

at org.jboss.netty.channel.DefaultChannelFuture.checkDeadLock(DefaultChannelFuture.java:343) ~[netty-3.5.8.Final.jar:na] 
at org.jboss.netty.channel.DefaultChannelFuture.awaitUninterruptibly(DefaultChannelFuture.java:255) ~[netty-3.5.8.Final.jar:na] 
at com.clarkparsia.stardog.snarl.client.AbstractClient.<init>(Unknown Source) ~[stardog-1.1.jar:na] 
at com.clarkparsia.stardog.snarl.client.SNARLClientImpl.<init>(Unknown Source) ~[stardog-1.1.jar:na] 
at com.clarkparsia.stardog.snarl.client.SNARLClientImpl$FastClassByGuice$fbba359a.newInstance(<generated>) ~[guice-3.0.jar:na] 
at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) ~[guice-3.0.jar:na] 
at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60) ~[guice-3.0.jar:na] 
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) ~[guice-3.0.jar:na] 
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) ~[guice-3.0.jar:na] 
at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978) ~[guice-3.0.jar:na] 
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024) ~[guice-3.0.jar:na] 
at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974) ~[guice-3.0.jar:na] 
at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:632) ~[guice-assistedinject-3.0.jar:na] 
at $Proxy39.createClient(Unknown Source) ~[na:na] 
at com.clarkparsia.stardog.snarl.client.SNARLDriver.connect(Unknown Source) ~[stardog-1.1.jar:na] 
at com.clarkparsia.stardog.api.DriverManager.getConnection(Unknown Source) ~[stardog-1.1.jar:na] 
at com.clarkparsia.stardog.api.ConnectionConfiguration.at(Unknown Source) ~[stardog-1.1.jar:na] 
at com.clarkparsia.stardog.api.ConnectionConfiguration.connect(Unknown Source) ~[stardog-1.1.jar:na] 

at com.r29.athena.stardog.SparqlStardogClient.getConnection(SparqlStardogClient.java:226) ~[classes/:na] 
at com.r29.athena.stardog.SparqlStardogClient.execute(SparqlStardogClient.java:80) ~[classes/:na] 
at com.r29.athena.stardog.SparqlStardogClient.executeForEach(SparqlStardogClient.java:190) ~[classes/:na] 
at com.r29.athena.service.core.Sparql.handleSelect(Sparql.java:97) ~[classes/:na] 
at com.r29.athena.service.core.Sparql$4.apply(Sparql.java:60) ~[classes/:na] 
at com.r29.athena.service.core.Sparql$4.apply(Sparql.java:1) ~[classes/:na] 
at com.r29.athena.service.core.EventBusHandler$1.handle(EventBusHandler.java:98) ~[classes/:na] 
at com.r29.athena.service.core.EventBusHandler$1.handle(EventBusHandler.java:1) ~[classes/:na] 
at org.vertx.java.core.eventbus.impl.DefaultEventBus$7.run(DefaultEventBus.java:577) ~[vertx-core-1.3.0.final.jar:na] 
at org.vertx.java.core.impl.Context$2.run(Context.java:122) ~[vertx-core-1.3.0.final.jar:na] 
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processEventQueue(AbstractNioWorker.java:453) [netty-3.5.8.Final.jar:na] 
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:330) [netty-3.5.8.Final.jar:na] 
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35) [netty-3.5.8.Final.jar:na] 
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102) [netty-3.5.8.Final.jar:na] 
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.5.8.Final.jar:na] 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_09] 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_09] 
at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]

[Robert Butler]

Yes, this looks like exactly what I was seeing with Netty's deadlock detection.

Robert Butler

Pancake Technology, LLC

972.861.0525

P.O. Box 271416
Flower Mound, TX 75027

@roberttheiv

http://www.linkedin.com/in/roberttheiv

[Mike Grove]

On Wed, Dec 12, 2012 at 3:03 PM, <jacobu...@gmail.com> wrote:

I have a deeper stack trace here that could explain what is going on, as you can see there are calls to netty from Stardog and from Vert-x on the same stack:

Yes, this is what I'd expect to see given Rob's suggestion on the issue and my review of that part of the Netty codebase.

If you can insert an org.jboss.netty.handler.execution.ExecutionHandler into the verticle Netty Channel pipeline, or otherwise get that call into different threads, you should be ok.  The different versions of Netty might otherwise not be an issue.

Cheers,

Mike

 

[jacobu...@gmail.com]

Ok, clear. So this means that the Stardog client can not be used in the same thread when the calling thread is responding to a netty event itself right?

[Mike Grove]

On Wed, Dec 12, 2012 at 3:14 PM, <jacobu...@gmail.com> wrote:

Ok, clear. So this means that the Stardog client can not be used in the same thread when the calling thread is responding to a netty event itself right?

Yes, that's the consequence of how Netty does deadlock detection on an IO thread.

Cheers,

Mike