we're happy that our URL index server is popular and heavily used.
However, it's only a single server and we cannot scale it up.
We think time and hardware are better spent to improve the crawler
Please try not to overload the URL index server! And please avoid
1. bulk downloads, e.g., *all .com results over all monthly crawl archives*.
It's ok, to perform bulk queries, but please try not to fetch Terabytes
of data via the index server! Below are instructions how to download the
index files directly.
2. fetching the list of available monthly indexes too often. The content of
is changed once per month. No need to fetch it multiple times per second.
Please keep it cached!
How to download index files:
The overview page on
links to a list of index files for each monthly index, e.g.
Download it, decompress it, and fetch the files in the list by adding the prefix
or when accessing it via S3
Want to fetch index files for a single top-level domain (here .fr)?
- the file list contains a cluster.idx file
- fetch it, e.g.:
- the first field in the cluster.idx contains the SURT representation of the URL,
with the reversed host/domain name:
- it's easy to list the cdx files containing all results from the .fr TLD:
grep '^fr,' cluster.idx | cut -f2 | uniq
That's only 4 files! I'm sure you're able to find the full path/URL
in the file list. If not, I'm happy to help.
- .com results make more than 50% of the index:
grep '^com,' cluster.idx | cut -f2 | uniq | wc -l
Please, fetch the index files directly. That's even much faster
and you can get all .com URLs from a monthly index in about one hour.
I'll add or link these instructions to the overview page soon.