Concurrent process execution problem in Scrapyd
29 views
Skip to first unread message
Germán Rosales
Jun 12, 2015, 8:30:39 AM6/12/15
to scrapy...@googlegroups.com
Hello everyone, looking for any suggestions/ideas on this matter :
I am running scrapyd 1.1 + scrapy 0.24.6 with a single spider that crawls over one domain. The development machine that host scrapyd's instance has 4 cores processor. I would like a setup to process N jobs simultaneously but scrapyd is processing 1 to 4 at a time regardless of how many jobs are pending and the
More info:
http://stackoverflow.com/questions/30672910/parallelism-performance-problems-with-scrapyd-and-single-spider
I am running scrapyd 1.1 + scrapy 0.24.6 with a single spider that crawls over one domain. The development machine that host scrapyd's instance has 4 cores processor. I would like a setup to process N jobs simultaneously but scrapyd is processing 1 to 4 at a time regardless of how many jobs are pending and the
max_proc_per_cpu parameter. Also i have noticed that scrapyd indeed use the 4 cores while working on 4 process at a time but would like to improve performance for batch execution using most of the processor capacity as possible and currently is using and maximum of 26% of each core.More info:
http://stackoverflow.com/questions/30672910/parallelism-performance-problems-with-scrapyd-and-single-spider
Reply all
Reply to author
Forward
0 new messages