Why engine fetch requests from scheduler first other than the start_urls generated ones?

[Jianhao Chen]

From HERE I found that Scrapy engine fetch requests from scheduler before the start_urls generated ones.

In my usage, I enqueued thousands of start urls (which from various domains) to the queue and the crawling goes not so fast (maybe networking issues). The problems comes up with me was that the spider itself extracts links and follows them, then Scrapy will fetch the requests from scheduler. It makes the concurrency lower.

I would like to learn about the design purpose of this mechanism.
BRs.

[Dimitris Kouzis - Loukas]

Are you asking for http://doc.scrapy.org/en/latest/topics/broad-crawls.html ? Finishing all the start_urls before going wide?

[Jianhao Chen]

Yes. While scrapy engine get the requests from scheduler first, not from start_urls.