How to get a list of domain names, 200 million of them, plain text, domain only, no other data

[Bruno A. H. Vincent (Webmaster & Tech)]

I've been fighting with AI for 3 days, like 100's of attempts

Maybe a human can help me?

I'm running windows 11,  i3 12100, 32 gigs of RAM good internet.

This is what I'm trying to do:

Fetch domains from common crawl, then the have them in a list like this:

a.com
b.com

I want only western countries extensions like .com, .org, .net

I have gitbash installed, can use Powershell, python,  curl, whatever is required.

How can i do this, at a loss!

[Greg Lindahl]

Bruno,

Our webgraph has that list as a separate file. I'd agree with you that this is unfortunately hard to discover!

-- greg

--
You received this message because you are subscribed to the Google Groups "Common Crawl" group.
To unsubscribe from this group and stop receiving emails from it, send an email to common-crawl...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/common-crawl/320d0c9e-c179-4640-952e-23e5570232fen%40googlegroups.com.

[Bruno A. H. Vincent (Webmaster & Tech)]

Thanks, what URL is this on?

" Our webgraph has that list as a separate file. I'd agree with you that this is unfortunately hard to discover!"

is it a text file?

[Greg Lindahl]

https://commoncrawl.org/web-graphs

In particular this 3.2 gigabyte file has what you want from the most recent web graph: https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2025-jun-jul-aug/domain/cc-main-2025-jun-jul-aug-domain-ranks.txt.gz

To view this discussion visit https://groups.google.com/d/msgid/common-crawl/030d63d4-57d7-4e76-95c4-15e5e37698edn%40googlegroups.com.