API Rate Limit

Abel B.

unread,

Oct 9, 2023, 1:05:04 PM10/9/23

to arXiv API

Hey everyone,

I've been using the arXiv API in a data engineering project for some time now and have recently encountered an issue. Up until a few days ago, I was able to fetch more records, but now it seems I can only retrieve the last 1,000 papers.

Has anyone else noticed this change? Did arXiv update its rate limits or introduce new restrictions? I appreciate any insights or updates you might have on this matter.

Thank you in advance for your assistance!

Warm regards,

Abel.

Rilton Franzone

unread,

Oct 13, 2023, 3:56:26 PM10/13/23

to arXiv API

I'm also facing the same issue. When I query for something using the query param `start` < 1000 I get the results normally, but anything beyond that returns an empty response. For example, this works:

https://export.arxiv.org/api/query?search_query=cat:cs.CV+OR+cat:cs.IR+OR+cat:eess.AS+OR+cat:eess.IV+OR+cat:eess.SP+OR+cat:cs.AI+OR+cat:cs.RO+OR+cat:cs.LG+OR+cat:cs.CL+OR+cat:cs.NE+OR+cat:stat.ML&sortBy=submittedDate&start=500&sortOrder=descending&max_results=200

But this one doesn't: https://export.arxiv.org/api/query?search_query=cat:cs.CV+OR+cat:cs.IR+OR+cat:eess.AS+OR+cat:eess.IV+OR+cat:eess.SP+OR+cat:cs.AI+OR+cat:cs.RO+OR+cat:cs.LG+OR+cat:cs.CL+OR+cat:cs.NE+OR+cat:stat.ML&sortBy=submittedDate&start=1000&sortOrder=descending&max_results=200

Gaurav Ragtah

unread,

Oct 13, 2023, 3:56:26 PM10/13/23

to arXiv API

Facing exactly this as well, even with long sleep delays in between fetching records. Anyone else? Any suggestions?

Reply all

Reply to author

Forward