Too many open files in system

[David Smith]

Hi,

I wanted to try out clj-gatling to see how I could push Aleph.  When trying a simple get request with 10,000 users at the same time I instantly got the below error.  I was a little surprised as I thought at worst this overload would be handled by Aleph and return a 503.  I would also like some guidance on how I can go about getting Aleph to handle a lot of open connections?  Is this a system limitation rather than a Java/Aleph problem?  If so, can anyone advise me on how to configure a mac to allow more connections?

Exception in thread "nioEventLoopGroup-3-1" java.lang.InternalError

at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:838)

at sun.misc.URLClassPath.getResource(URLClassPath.java:199)

at java.net.URLClassLoader$1.run(URLClassLoader.java:358)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

at java.lang.ClassLoader.loadClass(ClassLoader.java:412)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

at java.util.ResourceBundle$RBClassLoader.loadClass(ResourceBundle.java:450)

at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2566)

at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1436)

at java.util.ResourceBundle.findBundle(ResourceBundle.java:1400)

at java.util.ResourceBundle.findBundle(ResourceBundle.java:1354)

at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1296)

at java.util.ResourceBundle.getBundle(ResourceBundle.java:795)

at java.util.logging.Level.computeLocalizedLevelName(Level.java:257)

at java.util.logging.Level.getLocalizedLevelName(Level.java:316)

at java.util.logging.SimpleFormatter.format(SimpleFormatter.java:161)

at java.util.logging.StreamHandler.publish(StreamHandler.java:196)

at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:105)

at java.util.logging.Logger.log(Logger.java:616)

at io.netty.util.internal.logging.JdkLogger.log(JdkLogger.java:606)

at io.netty.util.internal.logging.JdkLogger.warn(JdkLogger.java:482)

at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:119)

at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.io.FileNotFoundException: /Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home/jre/lib/ext/localedata.jar (Too many open files in system)

[David Smith]

I had a look at http://shenfeng.me/600k-concurrent-connection-http-kit.html and saw that I need to increase the number of files my process is allowed to open.  On a mac this can be changed with:

sudo sysctl -w kern.maxfiles=4999999
sudo sysctl -w kern.maxfilesperproc=4999999
ulimit -n 4999999

However there were still quite a lot of errors.  Out of interest I will now try to copy the http-kit method and see how the two compare.

[Zach Tellman]

If we can't get a file descriptor, we can't open a connection, and therefore can't return a 503.  If you look at the latest round of Clojure webserver bencharks, you can see Aleph handling 60k connections without a fuss, and there's no reason you can't do another order of magnitude if you set up the environment correctly.  I honestly haven't done any system tuning towards this end on OS X, so I can't give much guidance there.  If you can get hold of a Linux test environment, there's a wealth of guides out there for this sort of tuning.

--
You received this message because you are subscribed to the Google Groups "Aleph" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aleph-lib+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[David Smith]

Thanks, I managed to get things going on Linux, I guess I'm more after someone being able to advise me on Yosemite as what I've found doesn't seem to work.  I'm just going to give up and stick to a Linux VM for now though.