warc2zip beta test
37 views
Skip to first unread message
Greg Lindahl
Aug 30, 2026, 3:49:46 PMAug 30
to Common Crawl
I invite our community to beta-test a new tool, warc2zip, that our engineer @Luca vibe-coded recently. Its goal is to enable researchers to work with WARC-formatted collections in a no-code or low-code environment. The zip file contains all of the crawl payloads as separate files, and all of the crawl metadata is stored as csv files, suitable for loading into a spreadsheet.
Our goal is to have this tool work with everyone's WARCs, so the end of README.md has a large list of public WARC examples.
Please have a look at https://github.com/commoncrawl/warc2zip and tell us how it works out! Expect bugs.
Our goal is to have this tool work with everyone's WARCs, so the end of README.md has a large list of public WARC examples.
Please have a look at https://github.com/commoncrawl/warc2zip and tell us how it works out! Expect bugs.
-- greg
Reply all
Reply to author
Forward
0 new messages