Akka's ask pattern can't be used in netty-based application due to threading issue

[wolfy]

Let's examine the following case:
1) We are using netty in our application
2) We are receiving some data in OUR netty worker
3) We pass received data to actor and waiting for answer using ask pattern

In such case akka will fail with the stacktrace like this:

java.lang.IllegalStateException: await*() in I/O thread causes a dead lock or sudden performance drop. Use addListener() instead or call await*() from a different thread.
    at org.jboss.netty.channel.DefaultChannelFuture.checkDeadLock(DefaultChannelFuture.java:343)
    at org.jboss.netty.channel.DefaultChannelFuture.awaitUninterruptibly(DefaultChannelFuture.java:255)
    at akka.remote.netty.ActiveRemoteClient$$anonfun$1.apply$mcV$sp(Client.scala:171)
    at akka.util.Switch.transcend(LockUtil.scala:27)
    at akka.util.Switch.switchOn(LockUtil.scala:48)
    at akka.remote.netty.ActiveRemoteClient.connect(Client.scala:151)
    at akka.remote.netty.NettyRemoteTransport.send(NettyRemoteSupport.scala:241)
    at akka.remote.RemoteActorRef.$bang(RemoteActorRefProvider.scala:234)
    at akka.actor.ActorRef.tell(ActorRef.scala:106)
    at akka.pattern.AskSupport$class.ask(AskSupport.scala:85)
    at akka.pattern.package$.ask(package.scala:41)
    at akka.pattern.Patterns$.ask(Patterns.scala:46)
    at akka.pattern.Patterns.ask(Patterns.scala)
    at sev.omnitrack.WaitForReply.askProcessing(WaitForReply.java:21)
    at sev.omnitrack.akka.nightmare.RequestReplyOneJvmAwaitBrokenTest$1$1$1.messageReceived(RequestReplyOneJvmAwaitBrokenTest.java:91)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:563)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
    at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:71)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:563)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:91)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:385)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:256)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)

Error message is very descriptive and clear. Akka's netty transport use unsafe and deprecated api, which may interfere with another netty's worker state.

Please find the full demo attached.

And one more funny issue: I don't know why, but if any other actor are sending messages to our simultaneoulsy with test IT WILL NOT FAIL.
So, if you uncomment

            //greeter.tell(new Greeting("hooita3"));

all will work fine.

Looks like very ugly race condition, isn't it?
 

[√iktor Ҡlang]

Hi Wolfy,

Ummm,no, it's actually you who are using Netty in the wrong way:

"Do not call await() inside ChannelHandler

The event handler methods in ChannelHandler is often called by an I/O thread unless an ExecutionHandler is in the ChannelPipeline. If await() is called by an event handler method, which is called by the I/O thread, the I/O operation it is waiting for might never be complete because await() can block the I/O operation it is waiting for, which is a dead lock."

Now: Akka uses ExecutionHandler so that's perfectly fine, however you seem to run your Netty without an ExecutionHandler, so in _your_ Netty IO Thread you are doing a remote send, and that will cause Akka remoting to block for the outbound channel to be connected, which is causing your issue. If you add an ExechutionHandler to your Netty pipeline everything will be just fine.

 

Please find the full demo attached.

And one more funny issue: I don't know why, but if any other actor are sending messages to our simultaneoulsy with test IT WILL NOT FAIL.
So, if you uncomment

            //greeter.tell(new Greeting("hooita3"));

all will work fine.

Looks like very ugly race condition, isn't it?

No, not at all, it's the first outbound send that will obtain the writelock, the others will be stalled in a readlock.

Cheers,

√

 

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user?hl=en.
 
 

--
Viktor Klang

Akka Tech Lead

Typesafe - The software stack for applications that scale

Twitter: @viktorklang

[Roland Kuhn]

17 aug 2012 kl. 11:18 skrev Ketoth Xupack:

Do not call await() inside ChannelHandler

Well... We are using camel (camel-netty component) and we are doing some akka-remoting on client message processing. And processing is really happens in Netty I/O thread (whoops...). Actually we can workaround this problem by digging in NettyEndpoint configuration. Or do remoting in separate Thread (for example just spawn new thread each time we need to do akka remoting)... 

 

However I think it's a good idea to not to use awaitUninterruptibly() in akka-remote at all. You just can't be sure your awaits will never ever called in other Netty I/O threads. I believe all awaits could be eliminated by using Futures / ChannelFutues and ChannelFutureListeners. 

Thoughts?

Being one of the few who have actually tried doing that, I can tell you it's way harder than it sounds. But we are looking into a completely different, actor-based design for the RemoteTransport layer, which will fix this issue.

Regards,

Roland

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user?hl=en.
 
 

Roland Kuhn
Typesafe – The software stack for applications that scale.
twitter: @rolandkuhn

[√iktor Ҡlang]

On Fri, Aug 17, 2012 at 11:26 AM, Roland Kuhn <goo...@rkuhn.info> wrote:

17 aug 2012 kl. 11:18 skrev Ketoth Xupack:

Do not call await() inside ChannelHandler

Well... We are using camel (camel-netty component) and we are doing some akka-remoting on client message processing. And processing is really happens in Netty I/O thread (whoops...). Actually we can workaround this problem by digging in NettyEndpoint configuration. Or do remoting in separate Thread (for example just spawn new thread each time we need to do akka remoting)... 

 

However I think it's a good idea to not to use awaitUninterruptibly() in akka-remote at all. You just can't be sure your awaits will never ever called in other Netty I/O threads. I believe all awaits could be eliminated by using Futures / ChannelFutues and ChannelFutureListeners. 

Thoughts?

Being one of the few who have actually tried doing that, I can tell you it's way harder than it sounds. But we are looking into a completely different, actor-based design for the RemoteTransport layer, which will fix this issue.

In fact, we already have a guy working full time on it.

 

Regards,

Roland

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user?hl=en.
 
 

Roland Kuhn
Typesafe – The software stack for applications that scale.
twitter: @rolandkuhn

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user?hl=en.
 
 

--

Viktor Klang

Akka Tech Lead

Typesafe - The software stack for applications that scale

Twitter: @viktorklang

[√iktor Ҡlang]

On Fri, Aug 17, 2012 at 11:41 AM, Ketoth Xupack <ketoth...@gmail.com> wrote:

Thanks you guys! We really appreciate all of your hard work and effort :)

Thanks!

 

Anyway... Is there any chance 2.1 release would have new remote transport layer?

Yes, but that chance is slightly smaller than the chance of me becoming a Microsoft MVP.

Cheers,

√

 

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user?hl=en.
 
 

[√iktor Ҡlang]

On Fri, Aug 17, 2012 at 12:13 PM, Ketoth Xupack <ketoth...@gmail.com> wrote:

Thanks anyway. Hope you'll became Microsoft MVP soon :D

;-)

Hopefully the experiment will be fruitful and after a lot of testing and tuning we'll be able to release it in either a 2.1.x or for 2.2

Cheers,

√

 

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user?hl=en.