Groups keyboard shortcuts have been updated
Dismiss
See shortcuts

November crawl and Web Graph release

50 views
Skip to first unread message

Thom Vaughan

unread,
Nov 24, 2024, 8:19:47 AM11/24/24
to Common Crawl
Hi all,

A late announcement here that the November 2024 crawl (CC-MAIN-2024-46) and the corresponding Web Graph (cc-main-2024-sep-oct-nov) are now available.

The November crawl contains 2.68 billion web pages (or 405 TiB of uncompressed content). Page captures are from 47.5 million hosts or 38.3 million registered domains and include 1 billion new URLs, not visited in any of our prior crawls.

The corresponding Web Graph consists of 298.2 million vertices and 2.7 billion arcs at the host level, and 98 million vertices and 1.8 billion arcs at the domain level.


See our blog posts above for more information, and as ever, we welcome any questions.

TV
Reply all
Reply to author
Forward
0 new messages