OutOfMemoryError: Direct buffer memory

[Rinde van Lon]

Hi list,

I'm having an issue with my Akka 1.2 application. I'm having the following exception: "java.lang.OutOfMemoryError: Direct buffer memory" It turns out that my issue is probably caused by a bug in Netty (compare this bug report with the stack trace on the bottom of this email). As this bug is not fixed yet, I'm wondering if anyone has any suggestions for a workaround for this Netty bug?

I have a server which manages connections with a number of nodes. At the server an actor is created for every node that connects to the server. The actors on the server periodically poll the nodes (by sending a message). Now if I let too many nodes (> 75) connect to the server the exception is thrown at the server.

I have been investigating several solutions for this problem, but with no succes so far. The only thing that worked is using only a small number of nodes in my system, but this is of course very unsatisfactory. In this thread a similar issue is described. I have tried increasing the direct memory with -XX:MaxDirectMemorySize but this doesn't help.

Below is the stack trace that I got from my program:

[ERROR]   [11/23/11 1:38 PM] [akka:event-driven:dispatcher:global-20] [LocalActorRef] Direct buffer memory

java.lang.OutOfMemoryError: Direct buffer memory

at java.nio.Bits.reserveMemory(Bits.java:633)

at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:98)

at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)

at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$Preallocation.<init>(SocketSendBufferPool.java:155)

at org.jboss.netty.channel.socket.nio.SocketSendBufferPool.<init>(SocketSendBufferPool.java:42)

at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:84)

at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.<init>(NioClientSocketPipelineSink.java:74)

at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:135)

at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:105)

at akka.remote.netty.ActiveRemoteClient$$anonfun$connect$1.apply$mcV$sp(NettyRemoteSupport.scala:431)

at akka.util.Switch.liftedTree1$1(LockUtil.scala:126)

at akka.util.Switch.transcend(LockUtil.scala:125)

at akka.util.Switch.switchOn(LockUtil.scala:138)

at akka.remote.netty.ActiveRemoteClient.connect(NettyRemoteSupport.scala:427)

at akka.remote.netty.NettyRemoteClientModule$class.withClientFor(NettyRemoteSupport.scala:112)

at akka.remote.netty.NettyRemoteSupport.withClientFor(NettyRemoteSupport.scala:649)

at akka.remote.netty.NettyRemoteClientModule$class.send(NettyRemoteSupport.scala:94)

at akka.remote.netty.NettyRemoteSupport.send(NettyRemoteSupport.scala:649)

at akka.actor.RemoteActorRef.postMessageToMailboxAndCreateFutureResultWithTimeout(ActorRef.scala:1249)

at akka.actor.ScalaActorRef$class.$qmark(ActorRef.scala:1453)

at akka.actor.RemoteActorRef.$qmark(ActorRef.scala:1207)

at rinde.cloud.core.Server$NodeDelegate.pollActor(Server.scala:249)

at rinde.cloud.core.Server$NodeDelegate$$anonfun$7.apply(Server.scala:229)

at rinde.cloud.core.Server$NodeDelegate$$anonfun$7.apply(Server.scala:227)

at scala.PartialFunction$$anon$1.apply(PartialFunction.scala:45)

at akka.actor.FSM$class.processEvent(FSM.scala:501)

at rinde.cloud.core.Server$NodeDelegate.processEvent(Server.scala:165)

at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:492)

at akka.actor.FSM$$anonfun$receive$1.apply(FSM.scala:464)

at akka.actor.FSM$$anonfun$receive$1.apply(FSM.scala:452)

at akka.actor.Actor$class.apply(Actor.scala:545)

at rinde.cloud.core.Server$NodeDelegate.apply(Server.scala:165)

at akka.actor.LocalActorRef.invoke(ActorRef.scala:905)

at akka.dispatch.MessageInvocation.invoke(MessageHandling.scala:25)

at akka.dispatch.ExecutableMailbox$class.processMailbox(ExecutorBasedEventDrivenDispatcher.scala:216)

at akka.dispatch.ExecutorBasedEventDrivenDispatcher$$anon$4.processMailbox(ExecutorBasedEventDrivenDispatcher.scala:122)

at akka.dispatch.ExecutableMailbox$class.run(ExecutorBasedEventDrivenDispatcher.scala:188)

at akka.dispatch.ExecutorBasedEventDrivenDispatcher$$anon$4.run(ExecutorBasedEventDrivenDispatcher.scala:122)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

at java.lang.Thread.run(Thread.java:680)

at akka.dispatch.MonitorableThread.run(ThreadPoolBuilder.scala:184)

Best regards,

Rinde

[√iktor Ҡlang]

Hi,

try to increase the read-timeout in the client section of the remote section in your akka configuration. This will reduce the number of created and destroyed client bootstraps.

Also, strongly recommend to try out the 1.3-RC1, it solves some problems with the remoting and looks to be the one and only RC before 1.3 goes final (since it seems to be awesome)

Cheers,
√

--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

--
Viktor Klang

Akka Tech Lead

Typesafe - Enterprise-Grade Scala from the Experts

Twitter: @viktorklang

[Rinde van Lon]

Hi,

Thanks for your quick reply! I have tried increasing the read-timeout to 60 and 120 (both at the nodes as at the server), unfortunately these values didn't have any effect..

Since 1.3 seems to be near final, I'm now thinking of upgrading to it. However, do you think this release will actually solve this issue? I'm a bit doubtful since the bug is actually in the Netty library.. Also, I'm wondering why it seems that I am the only one having this issue. Given that what I'm trying to do is not that spectacular or exotic, I suspect that I'm using Akka in a wrong or unintended way. Does anybody have any suggestions? Also, I'm happy to share more details of my app but I don't know which details are relevant for this issue..

Cheers,
Rinde

2011/11/23 √iktor Ҡlang <viktor...@gmail.com>

[√iktor Ҡlang]

On Wed, Nov 23, 2011 at 4:47 PM, Rinde van Lon <rinde...@gmail.com> wrote:

Hi,

Thanks for your quick reply! I have tried increasing the read-timeout to 60 and 120 (both at the nodes as at the server), unfortunately these values didn't have any effect..

That's a shame.
 

Since 1.3 seems to be near final, I'm now thinking of upgrading to it. However, do you think this release will actually solve this issue? I'm a bit doubtful since the bug is actually in the Netty library.. Also, I'm wondering why it seems that I am the only one having this issue.

Perhaps you're not, or perhaps it only affects your OS version, on a specific hardware configuration or specific JVM, or a combo of all of those.
 

Given that what I'm trying to do is not that spectacular or exotic, I suspect that I'm using Akka in a wrong or unintended way. Does anybody have any suggestions? Also, I'm happy to share more details of my app but I don't know which details are relevant for this issue..

I have no idea either, what does the Netty boys say?

Cheers,
√
 

[Rinde van Lon]

Ok, I did a test which I should probably have done a lot earlier.. I tested it on a different machine, and I had no issues here.

The machine on which it worked has the following specs:

Linux <machine name> 2.6.32-35-generic #78-Ubuntu SMP Tue Oct 11 16:11:24 UTC 2011 x86_64 GNU/Linux

java version "1.6.0_20"

OpenJDK Runtime Environment (IcedTea6 1.9.10) (6b20-1.9.10-0ubuntu1~10.04.2)

OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)

The machine on which I got the exception:

System Version: Mac OS X 10.6.8 (10K549)

Kernel Version: Darwin 10.8.0

java version "1.6.0_26"

Java(TM) SE Runtime Environment (build 1.6.0_26-b03-384-10M3425)

Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02-384, mixed mode)

So apparently this specific setup on the mac server is responsible for the errors. I have no clue what the exact reason might be, for now I will just avoid deploying my server on this machine. Thanks for the comments!

Best regards,

[√iktor Ҡlang]

On Wed, Nov 23, 2011 at 6:18 PM, Rinde van Lon <rinde...@gmail.com> wrote:

Ok, I did a test which I should probably have done a lot earlier.. I tested it on a different machine, and I had no issues here.

The machine on which it worked has the following specs:

Linux <machine name> 2.6.32-35-generic #78-Ubuntu SMP Tue Oct 11 16:11:24 UTC 2011 x86_64 GNU/Linux

java version "1.6.0_20"

OpenJDK Runtime Environment (IcedTea6 1.9.10) (6b20-1.9.10-0ubuntu1~10.04.2)

OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)

The machine on which I got the exception:

System Version: Mac OS X 10.6.8 (10K549)

Kernel Version: Darwin 10.8.0

java version "1.6.0_26"

Java(TM) SE Runtime Environment (build 1.6.0_26-b03-384-10M3425)

Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02-384, mixed mode)

So apparently this specific setup on the mac server is responsible for the errors. I have no clue what the exact reason might be, for now I will just avoid deploying my server on this machine. Thanks for the comments!

You're very welcome,
I'd really appreciate a write-up/case-study on your experiences with Akka down the road.

Happy hAkking!

Cheers,
√