Netty error in akka-remote

Bruce Mitchener

unread,

Jun 28, 2011, 2:32:37 PM6/28/11

to akka...@googlegroups.com

Howdy hAkkers,

Occasionally, we get this:

[ERROR] [2011-06-28 18:00:11:768] [akka:event-driven:dispatcher:event:handler-26] a.e.s.Slf4jEventHandler:

[akka.remote.netty.ActiveRemoteClientHandler]

[await*() in I/O thread causes a dead lock or sudden performance drop. Use addListener() instead or call await*() from a different thread.]

[java.lang.IllegalStateException: await*() in I/O thread causes a dead lock or sudden performance drop. Use addListener() instead or call await*() from a different thread.

at org.jboss.netty.channel.DefaultChannelFuture.checkDeadLock(DefaultChannelFuture.java:296)

at org.jboss.netty.channel.DefaultChannelFuture.awaitUninterruptibly(DefaultChannelFuture.java:208)

at akka.remote.netty.RemoteClient.sendPendingRequests(NettyRemoteSupport.scala:304)

at akka.remote.netty.ActiveRemoteClientHandler.channelConnected(NettyRemoteSupport.scala:530)

at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:110)

at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:362)

We're not really sure why, but it only happens in one of our processes. We seem to get a lot of lost connections which seems odd. (We run in EC2 if that matters.)

Anyone seen this before or have any thoughts?

I'll be happy to file an issue when I wake back up in 6 hours if necessary.

That happens on Akka 1.1.2 ...

- Bruce

Raymond Roestenburg

unread,

Jun 28, 2011, 3:07:30 PM6/28/11

to akka...@googlegroups.com

Maybe try 1.1.3? I think it's out.. Otherwise 1.2 snapshot

--

Raymond Roestenburg

code: http://github.com/RayRoestenburg

blog: http://roestenburg.agilesquad.com

twtr: @RayRoestenburg

--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

√iktor Ҡlang

unread,

Jun 28, 2011, 3:31:19 PM6/28/11

to akka...@googlegroups.com

Reimplement the resend of messages using recursion with a Netty future listener. So no need to do any blocking in it.
Open a ticket.
I'm on holiday so one of our awesome guys will take a stab at it.

Cheers,
V

Eugene Vigdorchik

unread,

Jun 28, 2011, 3:02:31 PM6/28/11

to akka...@googlegroups.com

Hello Bruce,
Please file a request for this. I think the exception is
self-explanatory: we shouldn't really block I/O thread while we
process channelConnected after the connection is back again.
The large number of lost connections I believe allows for the problem
to manifest itself, however why you get them is another question.

Cheers,
Eugene.

valodzka

unread,

Jul 26, 2011, 6:23:34 PM7/26/11

to Akka User List

Same problem with after upgrading from 1.0 to 1.1.3.
It's difficult to reproduce, but it repeated regularly on server
(every couple of days) and breaks application work.

On Jun 28, 10:07 pm, Raymond Roestenburg

<raymond.roestenb...@gmail.com> wrote:
> Maybe try 1.1.3? I think it's out.. Otherwise 1.2 snapshot
>
> --
> Raymond Roestenburg
>
> code:http://github.com/RayRoestenburg
> blog:http://roestenburg.agilesquad.com
> twtr: @RayRoestenburg
>

> On Jun 28, 2011, at 20:32, Bruce Mitchener <bruce.mitche...@gmail.com> wrote:
>
>
>
>
>
>
>
> > Howdy hAkkers,
>
> > Occasionally, we get this:
>
> > [ERROR] [2011-06-28 18:00:11:768] [akka:event-driven:dispatcher:event:handler-26] a.e.s.Slf4jEventHandler:
> > [akka.remote.netty.ActiveRemoteClientHandler]
> > [await*() in I/O thread causes a dead lock or sudden performance drop. Use addListener() instead or call await*() from a different thread.]
> > [java.lang.IllegalStateException: await*() in I/O thread causes a dead lock or sudden performance drop. Use addListener() instead or call await*() from a different thread.
> > at org.jboss.netty.channel.DefaultChannelFuture.checkDeadLock(DefaultChannelFu ture.java:296)

> > at org.jboss.netty.channel.DefaultChannelFuture.awaitUninterruptibly(DefaultCh annelFuture.java:208)

> > at akka.remote.netty.RemoteClient.sendPendingRequests(NettyRemoteSupport.scala :304)
> > at akka.remote.netty.ActiveRemoteClientHandler.channelConnected(NettyRemoteSup port.scala:530)
> > at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleC hannelUpstreamHandler.java:110)

> > at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPip eline.java:362)

√iktor Ҡlang

unread,

Jul 26, 2011, 6:29:45 PM7/26/11

to akka...@googlegroups.com

Are you using remote actor refs in onComplete-callbacks?

If so, wrap the contents of callback in Future {}, we'll fix this for 1.2

Cheers,
V

valodzka

unread,

Jul 26, 2011, 6:40:15 PM7/26/11

to Akka User List

No, I don't..

Code is very simple, something like

ActorRef ref = Actors.remote().actorFor(...)
...
while (!Thread.interrupted()) {
Object o = q.take();
ref.sendOneWay(o);
}

Where q is blocking queue, objects is putting in queue from other
threads.

May be problem occurs if there too many objects and akka doesn't have
time to send it all.
Here full stack trace:

[ERROR] [7/26/11 10:20 PM] [New I/O client worker #151-8]
[ActiveRemoteClientHandler] await*() in I/O thread causes a dead lock

or sudden performance drop. Use addListener() instead or call await*()
from a different thread.
java.lang.IllegalStateException: await*() in I/O thread causes a dead
lock or sudden performance drop. Use addListener() instead or call
await*() from a different thread.
at

org.jboss.netty.channel.DefaultChannelFuture.checkDeadLock(DefaultChannelFuture.java:
296)
at
org.jboss.netty.channel.DefaultChannelFuture.awaitUninterruptibly(DefaultChannelFuture.java:
208)
at
akka.remote.netty.RemoteClient.sendPendingRequests(NettyRemoteSupport.scala:

313)
at
akka.remote.netty.ActiveRemoteClientHandler.channelConnected(NettyRemoteSupport.scala:
544)
at
org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:
66)
at
org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:
66)
at
org.jboss.netty.channel.Channels.fireChannelConnected(Channels.java:
233)
at org.jboss.netty.channel.socket.nio.NioWorker
$RegisterTask.run(NioWorker.java:793)
at
org.jboss.netty.channel.socket.nio.NioWorker.processRegisterTaskQueue(NioWorker.java:
257)
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:199)
at
org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:
46)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

On Jul 27, 1:29 am, √iktor Ҡlang <viktor.kl...@gmail.com> wrote:
> Are you using remote actor refs in onComplete-callbacks?
>
> If so, wrap the contents of callback in Future {}, we'll fix this for 1.2
>
> Cheers,
> V

√iktor Ҡlang

unread,

Jul 27, 2011, 3:16:29 AM7/27/11

to akka...@googlegroups.com

On Wed, Jul 27, 2011 at 12:23 AM, valodzka <valo...@gmail.com> wrote:

Same problem with after upgrading from 1.0 to 1.1.3.
It's difficult to reproduce, but it repeated regularly on server
(every couple of days) and breaks application work.

This is caused by resending of messages, the work-around is to turn it off in your akka.conf

retry-message-send-on-failure = off

I'll see if it's possible to fix it for 1.2

Cheers,
√

--
Viktor Klang

Akka Tech Lead

Typesafe - Enterprise-Grade Scala from the Experts

Twitter: @viktorklang

Bruce Mitchener

unread,

Jul 27, 2011, 3:44:48 AM7/27/11

to akka...@googlegroups.com

What's the recommended way to use akka-remote in 1.1.x?

A problem with security cookies is also tied to resending...

- Bruce

2011/7/27 √iktor Ҡlang <viktor...@gmail.com>

√iktor Ҡlang

unread,

Jul 27, 2011, 3:53:00 AM7/27/11

to akka...@googlegroups.com

On Wed, Jul 27, 2011 at 9:44 AM, Bruce Mitchener <bruce.m...@gmail.com> wrote:

What's the recommended way to use akka-remote in 1.1.x?

I think the question needs to be a bit more specific.

A problem with security cookies is also tied to resending...

What ticket?

Cheers,
√

Bruce Mitchener

unread,

Jul 27, 2011, 4:04:07 AM7/27/11

to akka...@googlegroups.com

2011/7/27 √iktor Ҡlang <viktor...@gmail.com>

On Wed, Jul 27, 2011 at 9:44 AM, Bruce Mitchener <bruce.m...@gmail.com> wrote:

What's the recommended way to use akka-remote in 1.1.x?

I think the question needs to be a bit more specific.

Well, if we turn off resend-on-reconnect and other things happening still cause the connection to be dropped (like that future mapping not existing did), then there's a greater chance of problems ...

And more generally, are there are other flags which are better to have turned on / off due to bugs?

A problem with security cookies is also tied to resending...

What ticket?

None yet ... haven't had the time to reduce it to a test case, although I suspect that it will be similar to my second Rockwell test case that demonstrates other problems with reconnect. Working on a deadlock instead...

- Bruce

√iktor Ҡlang

unread,

Jul 27, 2011, 4:18:53 AM7/27/11

to akka...@googlegroups.com

On Wed, Jul 27, 2011 at 10:04 AM, Bruce Mitchener <bruce.m...@gmail.com> wrote:

2011/7/27 √iktor Ҡlang <viktor...@gmail.com>

On Wed, Jul 27, 2011 at 9:44 AM, Bruce Mitchener <bruce.m...@gmail.com> wrote:
What's the recommended way to use akka-remote in 1.1.x?

I think the question needs to be a bit more specific.

Well, if we turn off resend-on-reconnect and other things happening still cause the connection to be dropped (like that future mapping not existing did), then there's a greater chance of problems ...

And more generally, are there are other flags which are better to have turned on / off due to bugs?

Not that I know of.

A problem with security cookies is also tied to resending...

What ticket?

None yet ... haven't had the time to reduce it to a test case, although I suspect that it will be similar to my second Rockwell test case that demonstrates other problems with reconnect. Working on a deadlock instead...

Great, thanks

√

- Bruce

--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

Tymon Tobolski

unread,

Aug 28, 2011, 4:24:09 PM8/28/11

to Akka User List

I have same problem when sending message to remote actor from
unfiltered application running on netty.

java.lang.IllegalStateException: await*() in I/O thread causes a dead
lock or sudden performance drop. Use addListener() instead or call
await*() from a different thread.
at

org.jboss.netty.channel.DefaultChannelFuture.checkDeadLock(DefaultChannelFuture.java:
296)
at
org.jboss.netty.channel.DefaultChannelFuture.awaitUninterruptibly(DefaultChannelFuture.java:
208)

at akka.remote.netty.ActiveRemoteClient$$anonfun$connect$1.apply$mcV
$sp(NettyRemoteSupport.scala:381)
at akka.util.Switch.liftedTree1$1(LockUtil.scala:126)
at akka.util.Switch.transcend(LockUtil.scala:125)
at akka.util.Switch.switchOn(LockUtil.scala:138)
at
akka.remote.netty.ActiveRemoteClient.connect(NettyRemoteSupport.scala:
370)
at akka.remote.netty.NettyRemoteClientModule
$class.withClientFor(NettyRemoteSupport.scala:109)
at
akka.remote.netty.NettyRemoteSupport.withClientFor(NettyRemoteSupport.scala:
589)
at akka.remote.netty.NettyRemoteClientModule
$class.send(NettyRemoteSupport.scala:90)
at akka.remote.netty.NettyRemoteSupport.send(NettyRemoteSupport.scala:
589)
at akka.actor.RemoteActorRef.postMessageToMailbox(ActorRef.scala:
1158)
at akka.actor.ScalaActorRef$class.$bang(ActorRef.scala:1311)
at akka.actor.RemoteActorRef.$bang(ActorRef.scala:1135)

When using local actor everything is ok. I checked 1.1.3 and 1.2-RC3.
Any ideas?

On Jul 27, 10:18 am, √iktor Ҡlang <viktor.kl...@gmail.com> wrote:
> On Wed, Jul 27, 2011 at 10:04 AM, Bruce Mitchener <bruce.mitche...@gmail.com
>
>
>
>
>
>
>
>
>
> > wrote:
> > 2011/7/27 √iktor Ҡlang <viktor.kl...@gmail.com>

> Typesafe <http://www.typesafe.com/> - Enterprise-Grade Scala from the
> Experts
>
> Twitter: @viktorklang

Tymon Tobolski

unread,

Aug 28, 2011, 4:25:57 PM8/28/11

to Akka User List

Ah, forgot to mention - setting

retry-message-send-on-failure = off

did not help.

On Aug 28, 10:24 pm, Tymon Tobolski <i%teamon...@gtempaccount.com>
wrote:

> I have same problem when sending message to remote actor from
> unfiltered application running on netty.
>
> java.lang.IllegalStateException: await*() in I/O thread causes a dead
> lock or sudden performance drop. Use addListener() instead or call
> await*() from a different thread.
> at

> org.jboss.netty.channel.DefaultChannelFuture.checkDeadLock(DefaultChannelFu ture.java:
> 296)
> at
> org.jboss.netty.channel.DefaultChannelFuture.awaitUninterruptibly(DefaultCh annelFuture.java:

√iktor Ҡlang

unread,

Aug 28, 2011, 5:06:28 PM8/28/11

to akka...@googlegroups.com

Open ticket, attach failing test.

Thanks,
V

Tymon Tobolski

unread,

Aug 28, 2011, 6:37:47 PM8/28/11

to Akka User List

I dont have access to creating ticket at assembla, so I'll post it
here.

https://gist.github.com/1177321 - how to reproduce

Unfortunately I don't know how to reproduce this issue without using
unfiltered.

On Aug 28, 11:06 pm, √iktor Ҡlang <viktor.kl...@gmail.com> wrote:
> Open ticket, attach failing test.
>
> Thanks,
> V

> On Aug 28, 2011 10:24 PM, "Tymon Tobolski" <i%teamon...@gtempaccount.com>

> wrote:> I have same problem when sending message to remote actor from
> > unfiltered application running on netty.
>
> > java.lang.IllegalStateException: await*() in I/O thread causes a dead
> > lock or sudden performance drop. Use addListener() instead or call
> > await*() from a different thread.
> > at
>

> org.jboss.netty.channel.DefaultChannelFuture.checkDeadLock(DefaultChannelFu ture.java:> 296)
> > at
>
> org.jboss.netty.channel.DefaultChannelFuture.awaitUninterruptibly(DefaultCh annelFuture.java:> 208)

√iktor Ҡlang

unread,

Aug 28, 2011, 6:50:27 PM8/28/11

to akka...@googlegroups.com

I suspect that there could be a bug in Netty, so it believes that a an IO thread in Unfiltered belongs to the same bootstrap as Akka remoting.
What happens if you add an executionhandler to the pipeline in Unfiltereds bootstrap?

Cheers,
V

Tymon Tobolski

unread,

Aug 28, 2011, 8:01:05 PM8/28/11

to Akka User List

Looks like adding execution handler like this:
https://gist.github.com/1177321#file_test_with_executor.scala (before
plans) resolved this problem.

Thanks a lot.

Anyway, do you have any hints on what Execution Handler stratego to
use for webapp + akka?

On Aug 29, 12:50 am, √iktor Ҡlang <viktor.kl...@gmail.com> wrote:
> I suspect that there could be a bug in Netty, so it believes that a an IO
> thread in Unfiltered belongs to the same bootstrap as Akka remoting.
> What happens if you add an executionhandler to the pipeline in Unfiltereds
> bootstrap?
>
> Cheers,
> V

> On Aug 29, 2011 12:37 AM, "Tymon Tobolski" <i%teamon...@gtempaccount.com>

> wrote:
>
>
>
>
>
>
>
> > I dont have access to creating ticket at assembla, so I'll post it
> > here.
>

> >https://gist.github.com/1177321- how to reproduce

√iktor Ҡlang

unread,

Aug 29, 2011, 6:02:54 AM8/29/11

to akka...@googlegroups.com

Nope, you'll have to experiment :)

Cheers,
V

Reply all

Reply to author

Forward