429 Client Error: Too Many Requests for url: https://api.bcda.cms.gov/auth/token

[Paul Miller]

Was there a change to the tolerance level for this recently? Suddenly I'm getting these exceptions a lot when running two jobs at once. Normally we run up to 10 and never see this come up.

[Beneficiary Claims Data API (BCDA) Community]

Hi Paul,

Yes, there have been changes to BCDA's rate limiting in response to the recent authentication issues other customers reported.

BCDA now limits traffic from each IP Address to 300 requests in a 5 minute window. Exceeding that threshold will cause BCDA to respond with a 429 status code and the following OperationOutcome:

{
    "issue": [
        {
            "code": "throttled",
            "details": {
                "text": "Requests from this IP are currently throttled due to exceeding the limit. Try
again in 5 minutes."
            },
            "severity": "error"
        }
    ],  
"resourceType": "OperationOutcome"
}

As general guidance, we are recommending organizations follow an exponential backoff approach in automated requests to BCDA or using the value of the Retry-After header to inform when your next request can be made. These approaches can help you avoid having to implement any rate-limiting on your BCDA client reliant on the current limit of 300 requests per 5 minutes per IP Address. We apologize for any inconvenience, but believe this change will improve our service reliability for all users.

We'll update our documentation with this information as well.

Thanks,
The BCDA Team

[Paul Miller]

Thank you, I've applied the suggested changes and increased the exponential backoff factor for our retries, plus I set the job status check to every two minutes instead of the 30 seconds that we were doing before. Problem is, I'm still seeing the 429 errors occasionally. We do have a large number of ACOs that we pull data for regularly (up to 58 I believe now), and to save a ton of time, we do some of that work in parallel, so I can only assume that's causing us to hit the limit.

My follow-up question is: Does every API call count towards this total? (Job status check, token retrieval, file download, etc) My concern is that since there are potentially hundreds of files to download per ACO, and we have many ACOs, we hit that limit pretty regularly, especially on a fast network connection, and the only thing we can do is to space out our ACOs more, which will slow us down significantly.

[Beneficiary Claims Data API (BCDA) Community]

Hi Paul,

Yes, each call to BCDA from a given IP address counts towards the rate limit. We're analyzing BCDA's traffic to identify if there's a more appropriate way to enforce our rate limiting, and potentially alleviate the constraints it's put on organizations representing several entities.

We'll share our plans here as they develop and reach out directly over email if we need more information from you, Paul.

Additionally, we anticipate the number of files will decrease once our fix to the _since parameter is deployed this week. Reducing the number of files to download per job should likewise reduce the number of requests counted towards the rate limit for your IP Address(es). We'll provide more details and guidance on these changes in an upcoming post.

Thanks,
The BCDA Team