arXiv OAI-PMH returns 503 "first byte timeout" for specific date ranges

[Atanas S.]

Hi,

I'm experiencing intermittent 503 "first byte timeout" errors from the arXiv OAI-PMH endpoint (oaipmh.arxiv.org/oai) when querying ListRecords for specific date ranges.

Observations:

• Single-day queries (e.g. from=2026-02-09&until=2026-02-10) return 503 first byte timeout after ~183 seconds (likely Varnish timeout)
• The issue is not consistent across dates — some older dates respond quickly (e.g. Feb 20 → 200 OK in ~0.2s), while others in the same period consistently time out (e.g. Feb 9, Mar 2 → 503)
• The issue is independent of metadataPrefix (arXiv, oai_dc, arXivRaw all behave the same)
• Narrowing the query using set=cs does not resolve the issue
• Retries do not help — affected dates consistently return 503
• The response includes Retry-After: 0

Example failing request:

GET https://oaipmh.arxiv.org/oai?verb=ListRecords&metadataPrefix=arXiv&from=2026-02-09&until=2026-02-10

Any guidance would be appreciated. Thanks in advance!

[Brian Maltzan]

Hi Atanas,

503 errors are probably related to the total number of api users at that time.

Inconsistent speeds can be related type of query, and also whether that url has cached results.

Some tips - avoid high use times if you can. 8pm ET, after announcement is busy.

We have random usage spikes, maybe try a short time later. Right now, your example url is fast.

If a query is fast, then others may be using the same format- stick with that.

Hope that helps,

Brian

--
You received this message because you are subscribed to the Google Groups "arXiv API Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to api+uns...@arxiv.org.
To view this discussion visit https://groups.google.com/a/arxiv.org/d/msgid/api/0403879d-bc7c-42e2-a4bc-1b38c62ffa33n%40arxiv.org.