Scrapy and Scrapyd different process time
79 views
Skip to first unread message
k bez
May 30, 2017, 6:18:24 AM5/30/17
to scrapy-users
I have a strange problem and i could not find a solution for it.
When i use scrapy crawl command for a single URL and i have process time about 1 sec.
When i use scrapyd to process the same URL then I got 5 secs process time without HTTPCACHE_ENABLED and 3.5 secs with HTTPCACHE_ENABLED True( i saw same results both from console log and web UI runtime column).
Keep in mind that i have (?) all relatives settings to lowest (DOWNLOAD_DELAY=0, poll_interval=0, concurrent high values, etc).
Any idea why I have so strange process time?
When i use scrapy crawl command for a single URL and i have process time about 1 sec.
When i use scrapyd to process the same URL then I got 5 secs process time without HTTPCACHE_ENABLED and 3.5 secs with HTTPCACHE_ENABLED True( i saw same results both from console log and web UI runtime column).
Keep in mind that i have (?) all relatives settings to lowest (DOWNLOAD_DELAY=0, poll_interval=0, concurrent high values, etc).
Any idea why I have so strange process time?
Reply all
Reply to author
Forward
0 new messages