I took one of my smaller server logs and imported the IP and URLs
being crawled where the user agent claims to be Slurp and pulled them
into a database.....1 24 hour period. I was expecting to find most
were not really Slurp.
As it turned out, most appear to be slurp but it looks like multiple
bots from Yahoo arrive all at the same time and rather than splitting
what's in queue to be crawled among them, they all crawl the same
URLs.
Math example on why Yahoo is now wasting bandwidth
for every 4 bots with 4 pages to crawl, 16 pages could be crawled yet
the 4 URLs get crawled 4 times each.
So if 4 bots came and each should crawl 500 URLs, that would be 2000
URLs crawled but only 500 are. Had each only been crawled once, it
would have been 25000 kb bandwidth
If 500 pages of 50kb each get crawled 4 times each, that equals
100,000 kb bandwidth
If 2000 pages of 50kb each get crawled once that equals 100,000 kb
bandwidth
So 75000kb of bandwidth is wasted in a crawl of 500 pages
Do the math for the number of URLs in your sitemap. It's ridiculous.
I should also point out I am finding more than 4 at a time but still
haven't determined exactly how many are really slurp nor I have
checked to see if some come and go while others stay. 4 was based on
an average of several 30 minute blocks using IP's where I am sure they
are Slurp.
I have also checked that there was no round robin in the sitemap which
it appears was the source of the URLs. (site uses sitemapIndex, all
URLs were from the same individual sitemap, but not all URLs in that
sitemap were crawled - next 24 hour period showed a very similar
behavior with URLs found in one of the other individual sitemaps).
Haven't imported any data from other sites yet but all are really
getting hammered by Yahoo.
