Couldn't bind: 24: Too many open files

[Ali Bozorgkhan]

Hi,

I am running 25 jobs on a single machine, each with 50 concurrent requests. After a 10 minutes or so, I get this error in my proxy middleware:

Couldn't bind: 24: Too many open files.

I tried increasing the ulimit on the system but no luck. Is there anyway I could fix it?

[Pablo Hoffman]

How did you increased the ulimit?. Are you sure it was increased properly?

--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users...@googlegroups.com.
To post to this group, send email to scrapy...@googlegroups.com.
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/groups/opt_out.

[Ali Bozorgkhan]

I just found the solution:

http://stackoverflow.com/questions/19995855/increase-max-open-files-for-ubuntu-upstart-initctl

The problem is as scrapyd starts as a service, it will ignore any configuration in '/etc/security/limits.conf'. I solved the problem by adding this line to '/etc/init/scrapyd.conf':

limit nofile 524288 524288

--
You received this message because you are subscribed to a topic in the Google Groups "scrapy-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/scrapy-users/rArexq4tI7I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to scrapy-users...@googlegroups.com.

[Ali Bozorgkhan]

Thanks pablo for your reply, you are right, I did't fully explain what I did because at first, I thought it is something that needs to be set in scrapy, but I just realized it is a server problem.

[Pablo Hoffman]

No problem Ali, thanks for posting follow up with your solution.

I think you should also be able to adjust the ulimit by adding the following line to /etc/default/scrapyd (which has less chances of being overwritten by a scrapyd package upgrade):

ulimit -n 524288

Pablo.