Missing data in index search results

[Bahar Zafer]

Dear CommonCrawl Team,

I am reaching out to report an issue with the index search functionality. It appears that index.commoncrawl.org might be truncating search results when a URL pattern matches a large number of records.

In my analysis, I processed the full WARC files from CC-MAIN-2017-04 by iterating over all records and matching URL patterns, then compared these results with the URLs returned by the index query. For URL patterns matching fewer than 10,000 records, the index search performed as expected. However, for one URL pattern that matched 96,112 URLs in the full dataset, the index query returned only 14,165 URLs. After merging and sorting the URLs alphabetically, it seems that only the first 14,165 entries are being returned. I observed the same behavior with all five URL patterns that matched between 25K and 420K URLs—the index query consistently returned fewer than 15,000 results.

I tested the query using both Python’s requests package and curl, and both methods yielded identical results. Would you recommend a different approach or tool to handle such long responses?

I can provide additional details and share my files if needed.

Thank you for your assistance.

Best regards,

Bahar

[Thom Vaughan]

Hi Bahar,

The CDX index server paginates results, which may explain why you're seeing fewer results than expected in a single query. You can use the showNumPages=true URL parameter to get the total number of available pages, and then use page=N to iterate over them.

This is documented here: https://github.com/webrecorder/pywb/wiki/CDX-Server-API#pagination-api

Let us know if you need further assistance.

TV

[Jason Grey]

I’d be helpful to have your queries. If you don’t want to post here, you can send them to me directly and I can have a look.

On Mar 9, 2025, at 3:26 AM, Bahar Zafer <bahar...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "Common Crawl" group.
To unsubscribe from this group and stop receiving emails from it, send an email to common-crawl...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/common-crawl/c64ba5ad-56de-4653-b4e2-fdaa11499fb8n%40googlegroups.com.

[Bahar Zafer]

Hi Thom and Jason, 

Many thanks for your replies and assistance. Iterating over pages worked well. 

Based on a few responses that I have checked, it seems to me that the index query is returning the results at page=0 when the page parameter was not specified. Would you verify this happens generically? Is page=0 the default value? Would you recommend skipping page=0 and iterating over the rest of pages to complete my index search? 

A simplified version of my code below: 

resp = requests.get(f'https://index.commoncrawl.org/{crawl}-index?', params={'url': url_key, 'showNumPages':'true'})

cont = json.loads(resp.content)

numpages = cont['pages']

for pagenum in range(numpages):

      response = requests.get(f"https://index.commoncrawl.org/{crawl}-index?url={url_key}&output=json&page={pagenum}")

      .....

Thank you for taking the time to help.

Best, 

Bahar 

To view this discussion visit https://groups.google.com/d/msgid/common-crawl/819796BB-3721-48EC-9749-D8D247A4AEC8%40commoncrawl.org.

[Thom Vaughan]

Hi Bahar,

Glad you've made progress.  The page numbers are zero-indexed, so page=0 is the first page.

Best,

TV

[Bahar Zafer]

Hi Thom, 

I am still confused about this. Should response1 and response2 as defined below for some url key and crawl return the same output? 

response1 = requests.get(f'https://index.commoncrawl.org/{crawl}-index?', 

params={'url': url_key, 'output': 'json', 'page'=0} )

response2 = requests.get(f'https://index.commoncrawl.org/{crawl}-index?', 

params={'url': url_key, 'output': 'json'})

Many thanks for your help. 

Best regards,

Bahar

To view this discussion visit https://groups.google.com/d/msgid/common-crawl/97dd082c-2f77-4190-ab16-e4a1e9377a63n%40googlegroups.com.

[Jason Grey]

Yes, those should return the same results.

However - the JSON/dictionary you have specified below has incorrect syntax (note the use of “=" rather than “:”) - so be sure to fix that.

Jason Grey

ja...@commoncrawl.org

To view this discussion visit https://groups.google.com/d/msgid/common-crawl/CAPKptOXMyRRn%2BQNEA2oEM50-Vo4-bkSng6K6fvugwZ9_9Z0ruA%40mail.gmail.com.

[Bahar Zafer]

Thanks, Jason! 

To view this discussion visit https://groups.google.com/d/msgid/common-crawl/170AAF74-4CD6-4970-977B-5FF529060A5A%40commoncrawl.org.