From what I can gather, it's in the client lib for the game server we're using. Each actor needs to open a connection (vi their 3rd party api) to the server. Here's a trace on a failure. I run the same number of actors (and child actors) without connecting to the game server, and there are no such issues. I have been able to run with 1200 Actors, even with the high threads and file descriptor counts (not at all ideal, but maybe I can live with it).
Exception in thread "Thread-200" Exception in thread "Thread-199" org.jboss.netty.channel.ChannelException: Failed to create a selector.
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.openSelector(AbstractNioWorker.java:198)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:127)
at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:40)
at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:34)
at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:26)
at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.<init>(AbstractNioWorkerPool.java:56)
at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:29)
at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:152)
at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:117)
at sfs2x.client.core.sockets.TCPSocketLayer.initNetty(TCPSocketLayer.java:89)
at sfs2x.client.core.sockets.TCPSocketLayer.connect(TCPSocketLayer.java:221)
at sfs2x.client.bitswarm.BitSwarmClient.connect(BitSwarmClient.java:228)
at sfs2x.client.SmartFox$7.run(SmartFox.java:534)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.IOException: Too many open files in system
at sun.nio.ch.IOUtil.makePipe(Native Method)
at sun.nio.ch.KQueueSelectorImpl.<init>(KQueueSelectorImpl.java:84)
at sun.nio.ch.KQueueSelectorProvider.openSelector(KQueueSelectorProvider.java:42)
at java.nio.channels.Selector.open(Selector.java:227)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.openSelector(AbstractNioWorker.java:196)
... 13 more