For the website list given, a depth of 1 takes an insane amount of time to run

42 views
Skip to first unread message

David

unread,
Dec 6, 2012, 1:20:58 AM12/6/12
to csc-32...@googlegroups.com
Is my code horribly unoptimized, or is everyone else having a similar problem? For example, www.starbucks.ca at a depth of 1 takes more than 20 minutes for me. 

Wesley May

unread,
Dec 6, 2012, 7:08:06 AM12/6/12
to csc-32...@googlegroups.com
I may've given you some URLs that have a lot of outgoing links.

I think a reasonable speed is around 10 seconds per link. I wouldn't worry about speed unless it's taking > 30 seconds per link. For development you might want to use a really small URL list.

Nazanin

unread,
Dec 6, 2012, 7:50:02 PM12/6/12
to csc-32...@googlegroups.com
It is taking a lot of time to crawl the URL's provided, and it is because of the huge number of outgoing links that each URL has, in general it took our code about 40 minutes, is it something that we need to optimize? We have tried adding different rules to make search more optimized that is why our crawler is taking so long, can you let us know if we have to worry about crawler's speed?

Wesley May

unread,
Dec 6, 2012, 9:55:01 PM12/6/12
to csc-32...@googlegroups.com
Yeah my list is maybe a little bit crazy. Don't worry about the crawler's speed on my list. If you do it all in 40 minutes, that sounds absolutely fine.
Reply all
Reply to author
Forward
0 new messages