too many open files, akka 2.1.2, java api

[Alper Akture]

I'm using akka java api, 2.1.2. I'm creating about 500 actors, each with about 10 child actors. Monitoring the open file descriptors count, I see it steadily rise until I get a "too many open files" error (I've increased ulimit to 65535, and it seems to teeter on exceeding that limit). I'm still checking my app to make sure I'm not leaving any open connections, but so far it appears it is not. If I do an "ls -lrt" on the /proc/PID/fd directory, I see a lot of these types of files opened: 

l-wx------ 1 root root 64 2013-06-18 17:12 9265 -> pipe:[15485976]

lr-x------ 1 root root 64 2013-06-18 17:12 9264 -> pipe:[15486059]

lr-x------ 1 root root 64 2013-06-18 17:12 9263 -> pipe:[15485976]

l-wx------ 1 root root 64 2013-06-18 17:12 9262 -> pipe:[15486039]

l-wx------ 1 root root 64 2013-06-18 17:12 9261 -> pipe:[15486099]

lr-x------ 1 root root 64 2013-06-18 17:12 9260 -> pipe:[15486099]

I'm new to akka, and not sure if it's the way I'm using the actors or communicating between actors. Does this seem high for open files? Any pointers would be appreciated!

[Alper Akture]

It appears this was an issue with the way I was handling failures, sending a Kill message to an actor. The preStart was getting called, creating new child actors, without releasing the existing ones first. 

[Roland Kuhn]

Hi Alper,

do I guess correctly that you are using a PinnedDispatcher? In this case, please don’t do that :-)

Background: the only thing creating such pipes in a JVM AFAIK are threads, and you seem to be creating a lot of them. Actors are supposed to share threads, i.e. you do not need one per actor.

Regards,

Roland

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Dr. Roland Kuhn
Akka Tech Lead
Typesafe – Reactive apps on the JVM.
twitter: @rolandkuhn

[Alper Akture]

Thanks for the reply Roland. I'm not actually using a PinnedDispatcher. After some further study though, it appears the file handles are created by a client jar I'm using. We're building a load test app for a game. The game server's client library I believe uses netty, and it seems there are up to 50 file descriptors created once the connection is established between the client and server. I need to double check, but maybe I saw less than 200 actual threads. What's strange is, when I run the same code outside of akka, I only see about 7 file descriptors created per actor (once the connection is established). If you have any insight on that please share!

Thanks again,
Alper 

[√iktor Ҡlang]

Guesswork: Perhaps it stores things in ThreadLocals and since Akka uses multiple threads it causes the client lib to create one set of FDs for each thread?

Cheers,

√

--

Viktor Klang
Director of Engineering

Typesafe

Twitter: @viktorklang

[Alper Akture]

Thanks Viktor, that's entirely possible.  I'll see if I can get some more info from the client provider. The game server/client jar we're using is called SmartFox. I did some further testing with increasing number of actors. It's pretty linear. And with 1000 actors (each actor is a simulated user, and has 12 or so child actors), it's 3k threads and 50k file handles.

[√iktor Ҡlang]

Where do these threads and file handles come from? 3k threads sounds extreme(ly bad).

Cheers,

√

[Alper Akture]

From what I can gather, it's in the client lib for the game server we're using. Each actor needs to open a connection (vi their 3rd party api) to the server. Here's a trace on a failure. I run the same number of actors (and child actors) without connecting to the game server, and there are no such issues. I have been able to run with 1200 Actors, even with the high threads and file descriptor counts (not at all ideal, but maybe I can live with it).

Exception in thread "Thread-200" Exception in thread "Thread-199" org.jboss.netty.channel.ChannelException: Failed to create a selector.
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.openSelector(AbstractNioWorker.java:198)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:127)
    at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:40)
    at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:34)
    at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:26)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.<init>(AbstractNioWorkerPool.java:56)
    at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:29)
    at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:152)
    at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:117)
    at sfs2x.client.core.sockets.TCPSocketLayer.initNetty(TCPSocketLayer.java:89)
    at sfs2x.client.core.sockets.TCPSocketLayer.connect(TCPSocketLayer.java:221)
    at sfs2x.client.bitswarm.BitSwarmClient.connect(BitSwarmClient.java:228)
    at sfs2x.client.SmartFox$7.run(SmartFox.java:534)
    at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.IOException: Too many open files in system
    at sun.nio.ch.IOUtil.makePipe(Native Method)
    at sun.nio.ch.KQueueSelectorImpl.<init>(KQueueSelectorImpl.java:84)
    at sun.nio.ch.KQueueSelectorProvider.openSelector(KQueueSelectorProvider.java:42)
    at java.nio.channels.Selector.open(Selector.java:227)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.openSelector(AbstractNioWorker.java:196)
    ... 13 more

[√iktor Ҡlang]

Alright, happy hAkking!

Cheers,

√

[Alper Akture]

Indeed! Thank you!

[Alper Akture]

One interesting observation, I see some threads that I believe are created by akka named "Hashed wheel timer #" when my client code establishes a connection with the server. I don't see these being created when I run the same code outside of akka (ie, standalone java app).

So it appears their client creates two threads per connection, but I get this third thread when I run in akka. Any idea why that happens? Could it be something I'm doing incorrectly with the framework?

Here's one from a dump:

"Hashed wheel timer #8" prio=10 tid=0x00007f663047c800 nid=0x2638 waiting on condition [0x00007f6613852000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:464)
    at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:373)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
    at java.lang.Thread.run(Thread.java:724)

On Wednesday, June 26, 2013 3:28:06 PM UTC-7, √ wrote:

[√iktor Ҡlang]

Hi Alper,

Just realized: of course you're getting the HashedWheelTimer too many, it's your 3rd party lib that creates a new HWT for each instance instead of reusing a single one. You should look into if it's possible to provide a HWT from the outside to your 3rd party library.

Cheers,

√

[Alper Akture]

Yes, was able to confirm the HashedWheelTimer is created by the 3rd party lib we use. I'll see if they have any recommendations. Many thanks once again.

[√iktor Ҡlang]

Happy hAkking!

Cheers,

√