For the website list given, a depth of 1 takes an insane amount of time to run

[David]

Is my code horribly unoptimized, or is everyone else having a similar problem? For example, www.starbucks.ca at a depth of 1 takes more than 20 minutes for me. 

[Wesley May]

I may've given you some URLs that have a lot of outgoing links.

I think a reasonable speed is around 10 seconds per link. I wouldn't worry about speed unless it's taking > 30 seconds per link. For development you might want to use a really small URL list.

[Nazanin]

It is taking a lot of time to crawl the URL's provided, and it is because of the huge number of outgoing links that each URL has, in general it took our code about 40 minutes, is it something that we need to optimize? We have tried adding different rules to make search more optimized that is why our crawler is taking so long, can you let us know if we have to worry about crawler's speed?

[Wesley May]

Yeah my list is maybe a little bit crazy. Don't worry about the crawler's speed on my list. If you do it all in 40 minutes, that sounds absolutely fine.