openalex-cli metadata download "Rate limited" error

[Yang Shen]

Hello OpenAlex Team and Community,

I was experimenting using openalex-cli to harvest a proportion of Works, and I was harvesting metadata only.

After running the process a while, some failed messages appeared stating "Rate limited" errors. I checked credit limit with: https://api.openalex.org/rate-limit?api_key=MyAPIKey, and found there were plenty of credits remaining, as shown in the screenshot below.

The works in question seem normal when looked up in the browser, and I can confirm they were not downloaded.

Therefore, my questions are: Why are there "Rate-limited" errors when there are enough credits left? And why is the issue inconsistent, as in some works produced the error, while the majority didn't?

Also checked Github repo, but didn't find any existing issues directly related to the problem.

Best regards,

Yang Shen

[Jay Pfaffman]

Do you think you're doing more than 100 requests per second?

Jay Pfaffman <j...@literatecomputing.com>

http://www.literatecomputing.com/

--
You received this message because you are subscribed to the Google Groups "OpenAlex Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openalex-commun...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/openalex-community/6c515963-b065-40d7-854f-3726eb54fc6bn%40googlegroups.com.

[Yang Shen]

Hi Jay,

That's an interesting thought, and I tested it by calling: openalex download ... --workers 200

If based on the downloading speed, it's unlikely that there are more than 100 requests per second, as it downloads mostly stable 60-70 works/s on average. I didn't notice any spikes that lead to more than 100 works/s.

If based on number of workers (200), and assuming each worker requests immediately once the work ID is passed in, there should be more than 100 requests. But the "Rate limit" error appeared only very occasionally. In fact they didn't appear at all for the first 80k works harvested.

Best,

Yang

[Jay Pfaffman]

If you have 200 workers and are getting tens of thousands of files then it would make sense that every so often a hundred and one would fire the same second. I think if you had 99 workers you'd not have a problem. 

Jay Pfaffman <j...@literatecomputing.com>

http://www.literatecomputing.com/

To view this discussion visit https://groups.google.com/d/msgid/openalex-community/bb653109-eedc-4f5e-aa55-3ac051ce119fn%40googlegroups.com.

[Yang Shen]

Hi Jay,

I was using 50 workers (as the default) for my initial experiment shown in the screenshot. The 200 workers is to check if the problem will happen more frequently, as there are more API calls made simultaneously. From my observation, I'm not convinced it's more frequent.

That said, I do think your suspicion is valid that the API is firing over 100 calls/s and exceeded the limit, even with 50 workers. I tried to add some fixes to limit how many calls can be made within one second to the cli tool, but hasn't been successful. But I came across this PR, which handles the problem in a different way (retry the works that got a 429 Rate Limit error). At the moment, I won't rely entirely on the CLI for data harvesting.

Best,

Yang