500 errors from OAI service for certain dates

[Richard Orr]

I'm finding certain date ranges for which the OAI-PMH interface eventually gives a 500 error when paging through the results.

Steps to reproduce:

For example, starting with this request:

http://europepmc.org/oai.cgi?verb=ListRecords&from=2018-12-20&metadataPrefix=oai_dc&until=2018-12-21

I eventually get this resumption token and the resulting request fails:

http://europepmc.org/oai.cgi?resumptionToken=oai%3Aeuropepmc.org%3A5271626!2018-12-20!2018-12-21!oai_dc!&verb=ListRecords

Debugging attempts:

Noting that the resumption token is based on the ID of the first record in the next page, I thought the cause could be either:

1. A problem constructing the response for specific records. I tried incrementing the the starting record ID until the resumption token worked and found the point where the token starts working again:
   
   Not working: 5271655
   Working (most of the time): 5271656
   
   5271655 could be the problem, but both records work individually:
   
   oai:europepmc.org:5271655
   oai:europepmc.org:5271656
   
   
2. A kind of pagination race condition where the starting ID is no longer in the specified date range. But restarting the date range from the beginning eventually fails at the same resumption token, and at the moment the timestamp on both records linked above is 2018-12-21, so I don't think that's it either. 
   
   
   

Any idea what's causing this, or suggestions for a workaround better than skipping dates?

Thanks,
Richard

--

Richard Orr

Lead Developer, Unpaywall

Our Research: We build tools to make scholarly research more open, connected, and reusable—for everyone.