Web graph for May-Oct-Nov-2023 released
92 views
Skip to first unread message
Greg Lindahl
Dec 22, 2023, 2:53:36 PM12/22/23
to Common Crawl
The web graph for May-Oct-Nov-2023 is available. For full details,
please see the blog post:
https://commoncrawl.org/blog/host-and-domain-level-web-graphs-may-oct-nov-2023
Fans of the web graph will notice that this release covers the usual
number of crawls (3), but it was done the crawl after the previous web
graph release, and thus overlaps it. We did this on purpose, and will
continue to do this going forward. The benefit of this overlapping
data is that our ranking information for hosts and domains will update
more rapidly, which in turn will improve our crawling.
-- greg
please see the blog post:
https://commoncrawl.org/blog/host-and-domain-level-web-graphs-may-oct-nov-2023
Fans of the web graph will notice that this release covers the usual
number of crawls (3), but it was done the crawl after the previous web
graph release, and thus overlaps it. We did this on purpose, and will
continue to do this going forward. The benefit of this overlapping
data is that our ranking information for hosts and domains will update
more rapidly, which in turn will improve our crawling.
-- greg
Greg Lindahl
Mar 13, 2024, 10:09:38 PMMar 13
to Common Crawl
The web graph covering the most recent 3 crawls is available. For full details,
As I mentioned last time, we are now computing the webgraph after
every crawl. This means that they overlap in time. I hope people find
this fresher ranking information to be useful!
-- greg
please see the blog post:
https://www.commoncrawl.org/blog/host-and-domain-level-web-graphs-september-october-november-december-2023-and-february-march-2024
As I mentioned last time, we are now computing the webgraph after
every crawl. This means that they overlap in time. I hope people find
this fresher ranking information to be useful!
-- greg
Reply all
Reply to author
Forward
0 new messages