Paging for json endpoint

[Joel Strasser (joestr)]

Is it possible to do paging for the data returned as json?

For example https://crt.sh/json?q=mx24.eu with paging parameters like &offset=10&limit=10?

[Tamim]

I tried the same and it didnt work, infact it just fetches the same consistent n number of records. Is there a way to fetch atleast random ones so we can periodically create a local dataset by deduplicated fetched values?

[Anton Drukh]

I don't have a solution for pagination, but will say that I use the following URL structure - `https://crt.sh/?Identity=${domain}&output=json&exclude=expired&deduplicate=Y&limit=10`

Repeating it periodically (make sure you are aware of rate limits) will help you 'pseudo-paginate' the certs as they get published.

[Tamim]

Excluding=expired causes timeouts for me, and its limited to 900~ rows, aswell as everytime I run it returns the same data for me.

[Anton Drukh]

Yeah, I think this is quite greedy to exclude and limit by 900. Let's work with the endpoint we have, not the one we wish we had :)

My solution was to select a low limit (10), and run the query repetitively every couple of hours, practically crawling new certs as they get issued. Not the same use-case as yours, but still.

[Tamim]

Hasn't worked for me particularly. I would like to suggest using the CeRTSearcH tool created by the author. Lovely tool for pretty much "paginating" the database. Its can handle 100k records at a time from the DB, so its pretty slow tho.