Index for Common Crawl News Dataset

9 views

Skip to first unread message

unread,

Jul 18, 2026, 4:34:16 PM (2 days ago) Jul 18

to Common Crawl

Hi,

I could not find any indexes to the Common Crawl News Dataset, so I decided to create some:

There is sorted output from cdxj-indexer for each month in https://huggingface.co/datasets/brian-learns/cdx-cc-news/tree/main/cdxj and there are also parquet files.

While building it, I found this other interesting project that uses CC-NEWS

Thanks -- Brian

Reply all

Reply to author

Forward

0 new messages