possible bug in readonly store not all file closed?

[Xiao Zhou]

We got exception: 

Caused by: voldemort.VoldemortException: Fetch request on node 4 (http://10.6.1.175:8081/read-only/mgmt) failed: Error while performing operation: Exception in Fetcher = Error thrown while trying to get data from Hadoop filesystem : java.io.IOException: Failed on local exception: java.net.SocketException: Too many open files; Host Details : local host is: "java.net.UnknownHostException: dc1-voldemort05.dc01.revsci.net: dc1-voldemort05.dc01.revsci.net"; destination host is: "dc1-had01-m001.dc01.revsci.net":9000;

        at voldemort.store.readonly.swapper.HttpStoreSwapper$1.call(HttpStoreSwapper.java:96)

        at voldemort.store.readonly.swapper.HttpStoreSwapper$1.call(HttpStoreSwapper.java:71)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

We had increased the limit of allowed file handle for system to 20000 but still get this error. The number of voldemrot data and index file on the node is 6000. Is it possible there is a bug in readonly store not close all the files?

The version we are using is 1.6.0.

[Brendan Harris]

I don't think so. I'd suggest doing an "lsof -p <pid of voldemort server> | wc -l>"

To get a count of open files. My guess is that the user environment is not properly inheriting the "nofiles" hard and soft limits that you set.

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at http://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

[Brendan Harris (a.k.a. stotch on irc.oftc.net)]

On Thursday, July 10, 2014 4:32:53 PM UTC-7, Brendan Harris (a.k.a. stotch on irc.oftc.net) wrote:

I don't think so. I'd suggest doing an "lsof -p <pid of voldemort server> | wc -l>"

Sorry, I wrote that from my phone and it really did not come out right. Here's a clearer description:

- Get the PID of the voldemort process

- When it hits the open file limit, run the following:

sudo lsof -p <PID> | wc -l

That will output the number of files open by the voldemort process.

Additionally, you can add a "ulimit -a > $HOME/ulimit.out; echo "--------------------" >> $HOME/ulimit.out; ulimit -aH >> $HOME/ulimit.out" in the voldemort start script. This will print the soft then hard limits of various capabilities of the user, including max open files to $HOME/ulimit.out.

Brendan