estimating open file handle limits for broker

[Kevin Hutson]

I have a question about increasing the open file handles limits for our Kafka broker. We were warned to increase this before going to production.

I've found a couple of articles on how to do increase the limits.

http://mail-archives.apache.org/mod_mbox/kafka-users/201501.mbox/%3CCAB25XEUqjob0rd2Ub=vNC9eBdhoA+cf_JY...@mail.gmail.com%3E

http://grokbase.com/t/kafka/users/1478vzm6n7/too-many-open-files-broker-error

better explanation: http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/

We are using BOSH with debian images.

Our limits are already pretty high with the stemcell that we have.

~# ulimit -Hn

1048576

~# ulimit -Sn

65536

At the very least, I think we might increase the soft limit so we aren't warned at such a low number.

But, 1048576 is pretty large. And, these other articles seem to be using much smaller numbers (like 10K).

Is there a way to estimate how large of a number I should use for the hard limit?

thanks for any information you can provide.

best,

Kevin

[Gwen Shapira]

First, 1,048,576 should be more than enough.

We found out that the Java process actually bumps up the soft limit up to the hard limit when it starts, so from what I can tell, you don't need to worry about the soft limit (you can check per-process limits and see!)

To estimate the number you are going to need, you will need to know how many files you'll have on the broker and how many connections you'll have.

Estimating number of files is especially difficult, since this can change from partition to partition, and also within the same partition over time. 

Gwen

--
You received this message because you are subscribed to the Google Groups "Confluent Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to confluent-platf...@googlegroups.com.
To post to this group, send email to confluent...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/confluent-platform/d61151be-9099-446f-b92a-a0fa6e9719dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.