Groups
Sign in
Groups
老K实验室
Conversations
About
Send feedback
Help
SEO实验:基于海量数据制作的各类搜索引擎爬虫的爬行轨迹图/动画
10 views
Skip to first unread message
老K实验室
unread,
Sep 3, 2009, 11:08:23 PM
9/3/09
Reply to author
Sign in to reply to author
Forward
Sign in to forward
Delete
You do not have permission to delete messages in this group
Copy link
Report message
Show original message
Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message
to ee
老K实验室:最近老K实验室在研究搜索引擎爬虫行为模式的时候发现了一些比较有意思的英文文章,其中一些文章搜集了几大搜索引擎爬虫的爬行轨迹图,甚至有动画演示。虽然年代比较久远,大约都是06年前后的产物,但是还是有相当的参考价值。动画演示则非常直观,直接说明了爬虫是如何在网页和链接中爬行的,为什么目录层级数目越少的页面越容易被爬行和收录,等等。
On Bots
则是这些文章的原始出处文章,对当时的各大搜索引擎爬虫以实验的方式进行了一番研究,并针对各种数据进行了深度的分析,比较吓人的是这里面关联到的实验页面居然多达
2,147,483,647个
,时间长达一年,生成了
61万字的记录文件
。国外SEO大牛们的这类有目的、有方法、有结果、有产出的SEO实验实在是让人望尘莫及,值得学习和借鉴。
Yahoo! Slurp (2009年终于基本上算是挂掉了的万年老二——Yahoo搜索,实际上从这个实验里面显示的来看 Yahoo 搜索在06年前后还是有相当的技术优势)
large version
(4273x3090, 1.5MB) 大图
animated version over 1 year
(2005-04-13 - 2006-04-13, 13MB) 爬行超过一年的动画演示
animated version of the first 2 hours
(2006-04-14 00:40:00-02:40:00, 2.2MB) 最初两个钟头的动画演示
Googlebot (Google 爬虫)
large version
(4067x4815, 180kB) 大图
animated version
(2005-04-13 - 2006-04-13, 1.2MB) 动画演示版
成长以后的 Google Tree 看上去更像一颗花椰菜
MSN Bot Visual Animation
(MSN搜索的动画演示图,目前MSN搜索已经被
BING
取代)
参考文章:
Visual and animated representation of Search Engine Bots crawling huge websites
On Bots
Animated Search Engine Bot Crawl Visualizations
老K实验室
unread,
Sep 4, 2009, 2:02:28 AM
9/4/09
Reply to author
Sign in to reply to author
Forward
Sign in to forward
Delete
You do not have permission to delete messages in this group
Copy link
Report message
Show original message
Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message
to ee
有朋友反映之前链向图片和动画的直接链接似乎不管用,之前错误的链接现修正如下:
Yahoo! Slurp
large version
(4273x3090, 1.5MB)
animated version over 1 year
(2005-04-13 - 2006-04-13, 13MB)
animated version of the first 2 hours
(2006-04-14 00:40:00-02:40:00, 2.2MB)
Googlebot
large version
(4067x4815, 180kB)
Reply all
Reply to author
Forward
0 new messages