Too many open files exception.

[Girish Chaudhari]

Hi All,

We have Presto + EMR cluster up and running. We are getting following exception for some of our queries.Is there any config parameter we can set to overcome this exception.

(All our data is in S3 , At this point we have set hive.s3.max-connections=500 in hive.properties file). Any kind of help would be greatly appreciated.

com.facebook.presto.spi.PrestoException: Unable to execute HTTP request: Too many open files

at com.facebook.presto.hive.HiveSplitSource.propagatePrestoException(HiveSplitSource.java:139)

at com.facebook.presto.hive.HiveSplitSource.isFinished(HiveSplitSource.java:117)

at com.facebook.presto.split.ConnectorAwareSplitSource.isFinished(ConnectorAwareSplitSource.java:63)

at com.facebook.presto.split.BufferingSplitSource.fetchSplits(BufferingSplitSource.java:59)

at com.facebook.presto.split.BufferingSplitSource.lambda$fetchSplits$1(BufferingSplitSource.java:65)

at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:952)

at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926)

at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)

at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:561)

at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:580)

at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)

at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:77)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: com.amazonaws.AmazonClientException: Unable to execute HTTP request: Too many open files

at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:724)

at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:466)

at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:427)

at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:376)

at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4039)

at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3976)

at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:776)

at com.facebook.presto.hive.PrestoS3FileSystem.listPrefix(PrestoS3FileSystem.java:473)

at com.facebook.presto.hive.PrestoS3FileSystem.access$000(PrestoS3FileSystem.java:107)

at com.facebook.presto.hive.PrestoS3FileSystem$1.<init>(PrestoS3FileSystem.java:265)

at com.facebook.presto.hive.PrestoS3FileSystem.listLocatedStatus(PrestoS3FileSystem.java:263)

at com.facebook.presto.hadoop.HadoopFileSystem.listLocatedStatus(HadoopFileSystem.java:30)

at com.facebook.presto.hive.HadoopDirectoryLister.list(HadoopDirectoryLister.java:32)

at com.facebook.presto.hive.util.HiveFileIterator.getLocatedFileStatusRemoteIterator(HiveFileIterator.java:113)

at com.facebook.presto.hive.util.HiveFileIterator.computeNext(HiveFileIterator.java:86)

at com.facebook.presto.hive.util.HiveFileIterator.computeNext(HiveFileIterator.java:41)

at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145)

at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140)

at com.facebook.presto.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:234)

at com.facebook.presto.hive.BackgroundHiveSplitLoader.access$300(BackgroundHiveSplitLoader.java:85)

at com.facebook.presto.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:186)

at com.facebook.presto.hive.util.ResumableTasks.safeProcessTask(ResumableTasks.java:45)

at com.facebook.presto.hive.util.ResumableTasks.lambda$submit$1(ResumableTasks.java:33)

... 4 more

Caused by: java.net.SocketException: Too many open files

at java.net.Socket.createImpl(Socket.java:460)

at java.net.Socket.getImpl(Socket.java:520)

at java.net.Socket.setSoTimeout(Socket.java:1141)

at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:119)

at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)

at sun.reflect.GeneratedMethodAccessor547.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)

at com.amazonaws.http.conn.$Proxy165.connect(Unknown Source)

at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)

at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)

at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)

at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)

at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)

at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)

at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)

at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:875)

at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:715)

... 26 more

[Nezih Yigitbasi]

You should increase the max open file descriptor limit for the Presto processes.

[Girish Chaudhari]

Thanks for the reply Nezih, I would appreciate if you could tell me where can I set this .

Do i need to set in /etc/security/limits.conf ?

Like this ,

presto soft nofile 32768
presto hard nofile 65536

presto soft nproc 32768
presto hard nproc 65536

Correct me if i am wrong. ?

[David Phillips]

That looks correct. You can run the following query to verify the limits on all machines:

    SELECT openfiledescriptorcount, maxfiledescriptorcount FROM jmx.current."java.lang:type=operatingsystem";

This query requires the JXM connector to be setup: https://prestodb.io/docs/current/connector/jmx.html