Downloading 10 million pages

38 views
Skip to first unread message

Dave Lucas

unread,
Aug 29, 2022, 3:14:46 PMAug 29
to Common Crawl
Hi Sebastian,

I have a use case where I may need to download 10 million pages. Specifically, I want the html of the pages. I will have a set of urls derived from your index list.

Any thoughts on the best way to do this? 

Thx
Dave
Reply all
Reply to author
Forward
0 new messages