Too many open files
12 views
Skip to first unread message
Ian Brooks
Aug 22, 2016, 7:02:37 AM8/22/16
to Async HBase
Hi,
I'm working on a project that uses flume-ng.1.6.0 and asynchbase 1.5.0 to stream data into HBase.
I am experiencing an issue under high load where the flume sink fails after a short time with too many open files. From what I can see every time the region cache gets updated, the old socket connections seem to be left behind which eventually leads to the too many open files.
I also tried with asynchbase version 1.7.0 and had the same issues.
I think the initial region cache updates are due to HBase not keeping up or responding correctly, or if the regions are moved by a region going down. but the asynchbase library should handle this and close the old connections.
Any suggestions on resolving this issue?
-Ian
I'm working on a project that uses flume-ng.1.6.0 and asynchbase 1.5.0 to stream data into HBase.
I am experiencing an issue under high load where the flume sink fails after a short time with too many open files. From what I can see every time the region cache gets updated, the old socket connections seem to be left behind which eventually leads to the too many open files.
I also tried with asynchbase version 1.7.0 and had the same issues.
I think the initial region cache updates are due to HBase not keeping up or responding correctly, or if the regions are moved by a region going down. but the asynchbase library should handle this and close the old connections.
Any suggestions on resolving this issue?
-Ian
Reply all
Reply to author
Forward
0 new messages