Failures requesting /api/v2/metadata

[Kevin Born]

Hello BCDA team, 

We have been seeing errors when making GET requests to /api/v2/metadata since Saturday July 5. We have made several attempts since then and are seeing a response code of either 401, 404 or 500. Can you please have a look?

Kevin Born
Pearl Health

Kevin   Born Software Engineer II kevin...@pearlhealth.com 220 5th Avenue, 17th Floor, NY, NY 10001 www.pearlhealth.com

This email address should not be used to communicate private, confidential or sensitive information, including, but not limited to, personal health information protected by the Health Insurance Portability and Accountability Act of 1996, Public Law 104-191, as amended and its implementing privacy, security and breach notification regulations ("Protected Information"). If you need to send a communication to the sender of this email or any Pearl Health personnel containing Protected Information, please send an email to he...@pearlhealth.com indicating your need, but not containing the Protected Information, so that you can be provided instructions respecting use of a secure channel.

    

[Kevin Born]

Correction - we are seeing 500s on /api/v2/metadata and were seeing 401 and 404 on other endpoints previously. Thanks!

Kevin Born Software Engineer II kevin...@pearlhealth.com

[Lightbeam Development]

We are experience this same issue.

[Jack Shoemaker]

Dear Kevin,

In our experience a 500 means the server isn't available - an ACO REACH problem. The 400 series usually means that the credentials are wrong - an us problem. YMMV. - Jack

--

Jack N Shoemaker / (336) 202-2165

 

---

From: 'Kevin Born' via Beneficiary Claims Data API (BCDA) Community <bc-...@googlegroups.com>
Sent: Monday, July 7, 2025 13:06
To: Beneficiary Claims Data API (BCDA) Community <bc-...@googlegroups.com>
Subject: [BCDA] Re: Failures requesting /api/v2/metadata

 

--
You received this message because you are subscribed to the Google Groups "Beneficiary Claims Data API (BCDA) Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bc-api+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/bc-api/8e2c08ea-dd68-4b9b-8a87-8133f7e59271n%40googlegroups.com.

The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please reply to the sender (only) and then delete this message from your inbox as well as any copies. Thank you.

The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please reply to the sender (only) and then delete this message from your inbox as well as any copies. Thank you.

Secured by Paubox - HITRUST certified

[HDAI Development]

We are experiencing the same issue for some clients.

[Nick Moore]

That should be the case, but unfortunately is not with BCDA. These HTTP status codes have documented meanings, but it requires proper implementation in the service to be accurate.

There have been many instances where 401 are returned for completed valid access tokens because of issues with BCDA configuration, etc. and cases where 404 errors are used in lieu of the proper status code. Additionally, it's not clear if the 500 errors are originating within the BCDA service or the upstream BFD service.

All that to say, is that it is still necessary for us to question the 40x errors after we have ruled out issues on our end (as best we can).

At the moment the service still appears not to be functioning.

Nick Moore Senior Staff Engineer (858) 722‑4920 |  nick....@pearlhealth.com

To view this discussion visit https://groups.google.com/d/msgid/bc-api/DM5PR0101MB319675F30A1692BC019CB9DAA64FA%40DM5PR0101MB3196.prod.exchangelabs.com.

--

Nick   Moore Senior Staff Engineer (858) 722‑4920 |  nick....@pearlhealth.com

220 5th Avenue, 17th Floor, NY, NY 10001 www.pearlhealth.com

LinkedIn

[Aaron Croshaw]

We are also seeing this issue with fetch requests.

[Beneficiary Claims Data API (BCDA) Community]

Hi all,

Sorry to hear about the increased errors you're receiving from BCDA. We're taking a look at recent traffic to the API from over the past few days based on what you've reported. In support of our investigation, please share the following:

1. The 5 character Entity ID for your organization(s). For example, ACOs in the Shared Savings Program have Entity IDs which begin with the letter 'A' followed by four numbers, like "A9999".
2. The specific request(s) to BCDA which is(are) failing and the response from BCDA
3. Any additional messaging the system is providing to accompany the errors including response headers from BCDA. Please be sure to censor Personally Identifiable Information (PII) and other sensitive information (e.g., private keys, passwords) by replacing the information with the word ‘REDACTED’. Additional details and guidance can be found here in the Google Group.
4. Whether or not you are able to retrieve a token from BCDA's production API: https://api.bcda.cms.gov/auth/token
5. Whether or not you are able to make a successful request to the BCDA sandbox API: https://sandbox.bcda.cms.gov/api/v2/metadata

Please feel free to provide any additional information that you think might be pertinent, and again be sure to redact PII and other sensitive information.

Thank you in advance,
The BCDA Team

[Kevin Born]

Thanks for the response. 

Our flow is:

1. Authenticate (/auth/token)
2. get and log metadata (/api/v2/metadata)
3. trigger job and check status until job is complete (/api/v2/Group/all/$export and api/v2/jobs)
1. re-authenticate as necessary

And here is a log of our requests across 2 ACOs over the past couple days. You can see the failures happen at various points in the flow and with varying status codes per endpoint 

Timestamp	Resource	Status Code	ACO
2025-07-05, 05:00:06 UTC	/api/v2/metadata	500	D0242
2025-07-05, 05:00:06 UTC (x4)	/api/v2/metadata	500	D0243
2025-07-05, 05:06:26 UTC (x3)	api/v2/jobs	404	D0242
2025-07-06, 05:06:32 UTC (x5)	/api/v2/metadata	500	D0242
2025-07-06, 05:05:09 UTC	api/v2/jobs	500	D0243
2025-07-06, 05:10:14 UTC	/api/v2/metadata	500	D0243
2025-07-07, 05:00:06 UTC (x2)	/auth/token	401	D0243
2025-07-07, 05:10:13 UTC (x2)	/api/v2/metadata	500	D0243
2025-07-07, 14:15:18 UTC (x3)	/auth/token	401	D0242
2025-07-07, 14:21:04 UTC)	/auth/token	401	D0243
2025-07-07, 14:26:06 UTC	/api/v2/metadata	404	D0242
2025-07-07, 14:26:08 UTC	/api/v2/metadata	500	D0243
2025-07-07, 16:01:00 UTC	api/v2/jobs	500	D0242
2025-07-07, 16:06:04 UTC (x4)	/api/v2/metadata	500	D0242
2025-07-07, 18:11:32 UTC	api/v2/jobs	500	D0242

Kevin Born Software Engineer II kevin...@pearlhealth.com

--
You received this message because you are subscribed to a topic in the Google Groups "Beneficiary Claims Data API (BCDA) Community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bc-api/1hIm0_oZYXE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bc-api+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/bc-api/012f593c-4cde-4df1-811a-794586e0a91en%40googlegroups.com.

[Jack Shoemaker]

Dear BCDA Team,

Regarding item 4 below, we test for this before going further with submit requests and downloads.

That is, obtaining the token is the first task. When that fails, we stop further processing for the day.

• Jack

--

Jack N Shoemaker / (336) 202-2165

---

From: Beneficiary Claims Data API (BCDA) Community <bc-...@googlegroups.com>
Sent: Monday, July 7, 2025 2:56:20 PM

To: Beneficiary Claims Data API (BCDA) Community <bc-...@googlegroups.com>
Subject: [BCDA] Re: Failures requesting /api/v2/metadata

--

You received this message because you are subscribed to the Google Groups "Beneficiary Claims Data API (BCDA) Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bc-api+un...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/bc-api/012f593c-4cde-4df1-811a-794586e0a91en%40googlegroups.com.

The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please reply to the sender (only) and then delete this message from your inbox as well as any copies. Thank you.

The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please reply to the sender (only) and then delete this message from your inbox as well as any copies. Thank you.

[Jack Shoemaker]

Dear Nick,

Empirically, I can't argue with you because several of our failures don't comport with what I wrote. I am reluctant to blame the publisher, hence my response.

We do daily polls. It is maddening that it works 'most of the time'. Intermittent errors are the hardest to solve.

-Jack

--

Jack N Shoemaker / (336) 202-2165

---

From: Nick Moore <nick....@pearlhealth.com>
Sent: Monday, July 7, 2025 2:45:41 PM
To: Jack Shoemaker <jshoe...@mhnchicago.org>
Cc: Beneficiary Claims Data API (BCDA) Community <bc-...@googlegroups.com>; Kevin Born <kevin...@pearlhealth.com>
Subject: Re: [BCDA] Re: Failures requesting /api/v2/metadata

 

[Schwindt, Adam]

I can confirm we have had a very similar experience since late yesterday afternoon when our automated processes kicked off.

A2973

/patient has also experienced the same issues.

You received this message because you are subscribed to the Google Groups "Beneficiary Claims Data API (BCDA) Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bc-api+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/bc-api/CAFyGo9j62%2BCR%3DZB_xy5LYm1-J5jaq8Mh8P%3DgWEgUYTPeZwXMtA%40mail.gmail.com.

--

Adam Schwindt

Sr. Director, ACO provider and vendor integration 

LTC ACO

Mobile: 317-779-4393

Email: adam.s...@LTCACO.com

This e-mail and any attachments may contain information which is confidential, proprietary, privileged or otherwise protected by law. The information is solely intended for the named addressee (or a person responsible for delivering it to the addressee). If you are not the intended recipient of this message, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this e-mail in error, please notify the sender immediately by return e-mail and delete it from your computer.

[Beneficiary Claims Data API (BCDA) Community]

Hi all,

Our apologies for the degraded BCDA service over the past few days. We've restarted the BCDA service and traffic seems to have been flowing normally again over the past hour. Our team is continuing to investigate the root cause of the service degradation. We also have some improvements to our application monitoring in progress to alert us to these types of incidents earlier.

Additionally, we will review the HTTP status codes returned by our service and how they could adjusted to better align with standard uses.

Thanks,
The BCDA Team