b1 Receive for XDR Cases Down

[Braeden Rai]

Hello,

We were testing the b1 Receive Using XDR cases, and all of them seem to immediately fail with this error:

 Error 0x0020

 Hide debugging

Exception: a problem occured while sending the XDR document

Caused by:

Requesting: xdr

The cases fail both for the new site and the old one. In the past this error seems to have occurred when the site itself is having issues, is this the case currently? If so, when can we expect these to be back up?

Thanks,

Braeden

[Kim Poletti]

Hi - Thanks for reaching out. This has been logged for review and a member of the team will reach out in the near future.

[Evan Brock]

Hello,

I am currently investigating this, can you confirm if this is still an issue? As far as I can tell, provided a valid endpoint, the system is working as intended

[Braeden Rai]

Yes, we are still having the issue. We are also investigating to double check that there are no issues on our end. If you are free today or later this week, we could get on a call to troubleshoot this with you.

[Evan Brock]

My availability this week will be limited, but I can meet sometime within the next 2 hours today or on Friday. In the meantime, can you provide me the endpoint that you're using that is getting the error?

[Braeden Rai]

We can meet this afternoon to troubleshoot, thank you.

[Evan Brock]

Hello

Let me know if this time doesn't work for you

Evan Brock is inviting you to a scheduled Zoom meeting.

Topic: Evan Brock's Zoom Meeting
Time: Jan 14, 2025 02:30 PM Eastern Time (US and Canada)

Join Zoom Meeting
https://pointclickcare.zoom.us/j/93538478511?pwd=VsCDYizcMYtazG41SU1C12bAgak3ah.1

Password: 055168

One tap mobile
+13126266799,,93538478511# US (Chicago)
+13092053325,,93538478511# US

Dial by your location
        +1 312 626 6799 US (Chicago)
        +1 309 205 3325 US
        +1 305 224 1968 US
        +1 646 558 8656 US (New York)
        +1 646 931 3860 US
        +1 301 715 8592 US (Washington DC)
        +1 564 217 2000 US
        +1 669 444 9171 US
        +1 669 900 6833 US (San Jose)
        +1 689 278 1000 US
        +1 719 359 4580 US
        +1 253 205 0468 US
        +1 253 215 8782 US (Tacoma)
        +1 346 248 7799 US (Houston)
        +1 360 209 5623 US
        +1 386 347 5053 US
        +1 507 473 4847 US
        +1 778 907 2071 Canada
        +1 780 666 0144 Canada
        +1 204 272 7920 Canada
        +1 438 809 7799 Canada
        +1 587 328 1099 Canada
        +1 647 374 4685 Canada
        +1 647 558 0588 Canada
Meeting ID: 935 3847 8511
Password: 055168
Find your local number: https://pointclickcare.zoom.us/u/abUgSJFygx

[Evan Brock]

Hello,

Our server logs reveal the following when running the XDR Test with the endpoint provided during the Zoom call:

Caused by: { code:"500" extendedCode:"0" reason:"XdsInternal error Network connection fault: org.apache.axiom.om.OMException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target" reasonPhrase:"Internal Server Error"

Which usually indicates a certificate issue for your endpoint. I would recommend ensuring the certificate you're using isn't expired, you're not using a self signed cert, and the cert is for the correct domain.

[Braeden Rai]

We've attempted to resolve this issue, are you still seeing the same error for us?

[Braeden Rai]

If anyone else from your team is able to meet today, that would also be helpful.

[Evan Brock]

Unfortunately I'm not in a position to check the logs today, but your endpoint is still failing for me while other known valid endpoints are working, so I would assume the same issue is persisting.

Most members of the team are at an event currently, but I will see what I can do

[Braeden Rai]

We got XDR Test 3 working, it turns out the certs available on the site expired in October of last year. The other Receive for XDR cases are still failing though, a few of them still the error mentioned above. We're seeing a connection made before ending almost immediately. Based on this site, it seems like the error is an internal error? Is there something to update for your certificates as well? Specifically, does the cacerts file in the JAVA VM need to be updated to trust the cert that expires in 2034?

[Evan Brock]

Hello,

I just tested your endpoint for 3, 4a, 4b and 5 and it seems to be behaving as intended now. The logs are being populated with the response, although I did note that the test will sometimes show as failed, it will still function as intended. I'll be able to get more info from the logs tomorrow if needed, but unfortunately all I could do for now is run the tests again.

[Braeden Rai]

Since 4a and 4b are supposed to cause an error and you saw them working as intended, is this error the one we should expect to see?

IOException Bad Connection

javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

[Braeden Rai]

Working with our team here, one of them was looking at Test 5 and it seems that the message doesn't actually have a document.  Just the extrinsicobject metadata for the document, the registry package and association that says the exterinsicobject is part of the registry package.

[Evan Brock]

Hello,

Sorry for the delay, just wanted to give an update

Are you currently seeing the PKIX error? Is it in the logs for the 4a and 4b tests? If so, that is not the expected output. I am not personally seeing that, the logs are being populated with a proper response.

For test 5, I was able to reproduce the issue, and the logs are definitely indicating a discrepancy between a known working endpoint and yours. They are indicating that the payload is empty, which would be why the message doesn't have a document. I need to dig deeper to find out what's causing this.

[Braeden Rai]

Yes, we are still seeing that error in the logs for 4a and 4b.

[Evan Brock]

Hello,

Can you provide any information on the protocols and ciphers that you support?

Thank you

[Evan Brock]

Also, if you'd like to get on a call to further discuss this, please provide your availability this week and I'll setup another call.

[Braeden Rai]

Hi, we're still seeing the same issue with the 4a and 4b cases. Are there any more updates, or would it be possible for us to set up additional troubleshooting calls?

[Evan Brock]

Hi Braeden,

Can you provide your availability for next week? I'd like to setup another meeting to go over this if you're experiencing the issue

Thank you

[Braeden Rai]

We're available tomorrow afternoon for troubleshooting, and most days next week if that does not work.

[Mahesh Dabi]

Hi 

We are also facing problems while sending and receiving to ETT. Let us know the call details.

Thanks

Mahesh

[Miles Williams]

Status update: 

     Still failing: 

          XDR 8

          XDR 5: "Invalid request received. ProvideAndRegisterDocumentSetRequest not complete. An unknown error has occurred."

          XDR 4a / 4b: The PKIX path building failed error noted above.

     Succeeding:

          XDR 3 Receive + Negative Cases

[Evan Brock]

Thank you for details

I've created a meeting for tomorrow, let me know if another time works better. I will keep looking into these.

Evan Brock is inviting you to a scheduled Zoom meeting.

Topic: Evan Brock's Zoom Meeting

Time: Mar 11, 2025 11:30 AM Eastern Time (US and Canada)

Join Zoom Meeting
https://pointclickcare.zoom.us/j/93037088288?pwd=XyaLttxEjEMNLTh7vvCofOGCiTOIeT.1

Password: 541906

One tap mobile
+13092053325,,93037088288# US
+13126266799,,93037088288# US (Chicago)

Dial by your location

        +1 309 205 3325 US
        +1 312 626 6799 US (Chicago)

        +1 301 715 8592 US (Washington DC)
        +1 305 224 1968 US
        +1 646 558 8656 US (New York)
        +1 646 931 3860 US

        +1 719 359 4580 US
        +1 253 205 0468 US
        +1 253 215 8782 US (Tacoma)
        +1 346 248 7799 US (Houston)
        +1 360 209 5623 US
        +1 386 347 5053 US
        +1 507 473 4847 US
        +1 564 217 2000 US
        +1 669 444 9171 US
        +1 669 900 6833 US (San Jose)
        +1 689 278 1000 US

        +1 647 374 4685 Canada
        +1 647 558 0588 Canada
        +1 778 907 2071 Canada
        +1 780 666 0144 Canada
        +1 204 272 7920 Canada
        +1 438 809 7799 Canada
        +1 587 328 1099 Canada

Meeting ID: 930 3708 8288
Password: 541906
Find your local number: https://pointclickcare.zoom.us/u/adJsJ61YRh

[Evan Brock]

Hello,

Can you confirm if replacing the self signed certificate resolved the issue with the XDR tests?

Thank you

[Braeden Rai]

Unfortunately after updating our certificates, we're still seeing the same issue occur. Are there any error logs on your end that have more details about the failure?

[Miles Williams]

Here's a status update with the cert updates we made since our call last week: 

Not working

XDR 8: "No logs to display"
XDR 9: Success, "No logs to display"
XDR 5: a problem occured while sending the XDR document
XDR 3: a problem occured while sending the XDR document
XDR 4a: IOException Bad Connection

javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

XDR 4b: same as above
XDR 20a: Not seeing the received MDN in our trace viewer
XDR 20b: same as above



Working

XDR 6
XDR 7
XDR 2
XDR 49

[Evan Brock]

Thanks for the update. I looked into this some more and can see that the self signed certificate issue is resolved, but that just allows me to see them following upon trying to reach the endpoint from the ETT:

[ServiceActivationException]: The service '/Interconnect........svc' cannot be activated due to an exception during compilation.  The exception message is: Memory gates checking failed because the free memory (508194816 bytes) is less than 5% of total memory.  As a result, the service will not be available for incoming requests.  To resolve this, either reduce the load on the machine or adjust the value of minFreeMemoryPercentageToActivateService on the serviceHostingEnvironment config element..

I'd recommend investigating your server resources or adjusting your WCF configuration minFreeMemoryPercentageToActivateService

Thank you

[Braeden Rai]

Hi,

We did increase the available memory, can you try testing again?

[Braeden Rai]

Hi, are there any updates on if you were able to test successfully?

[Evan Brock]

Hello,

Sorry for the delay, I checked last week and your certificate is working now. The tests are still failing, but I believe we need to update the server with your new certificate. I should have an update for you later this week.

[Evan Brock]

Quick update, we restarted the service last night which we expected to resolve the issue, however the issue is still showing. I'm continuing to work on this, thank you for your patience.

[Miles Williams]

Hello, 

Is there any update this week? We are still seeing the cases behave as follows:

Not Working
XDR 8: "No logs to display."
XDR 9: Success, "No logs to display."

XDR 5: a problem occured while sending the XDR document
XDR 3: a problem occured while sending the XDR document
XDR 4a: IOException Bad Connection

javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
XDR 4b: same as above
XDR 20a: Not seeing the received MDN in our trace viewer
XDR 20b: same as above

Working
XDR 6
XDR 7

XDR 1
XDR 2

[Evan Brock]

Hello,

We are still working on a solution for this. Do you happen to remember where you downloaded the XDR TLS certificates from? Were they from the new UI at https://site.healthit.gov/direct/testbycriteria/H2?

[Evan Brock]

Hello,

We have resolved this issue. Please get the new certs from https://site.healthit.gov/certificates/xdr-tls/keyAndCert.zip and reconfigure your endpoint to use them.

[Braeden Rai]

Unfortunately after installing these certs we're still experiencing the same issues. Would we be able to set up another troubleshooting call this week, or look at your logs to help troubleshoot?

[Evan Brock]

Hello,

Sorry the solution didn't fix the issue. I have checked your endpoint after updating the certs and it's still the same PKIX error.

Let me know if this time doesn't work for you

Thursday, Apr 17, 11:00AM EST

________________________________________________________________________________

Microsoft Teams Need help?

Join the meeting now

Meeting ID: 212 293 389 823 4

Passcode: jo329CV2

---

Dial in by phone

+1 213-436-3838,,90344826# United States, Los Angeles

Find a local number

Phone conference ID: 903 448 26#

For organizers: Meeting options | Reset dial-in PIN

Privacy and security

_____________________________________________

[Evan Brock]

Hello,

I've still been looking into this issue and other users that were experiencing this PKIX issue are seeing the issue resolved for their endpoints. I'm still seeing XDR test 3 working, but the endpoint you provided me to test 4a and 4b is timing out now when it used to still return the PKIX issue. Have any changes been made to that endpoint on your end?

[Braeden Rai]

I just tested again and saw the PKIX error again, it doesn't seem to be timing out. I'll be able to test some and troubleshoot next week on Monday to see if anything changes.

[Evan Brock]

Thank you for checking, after further testing I realized that the endpoint used for 4a and 4b is visible from our production environment, but not our development environment, where I was running the tests, which is why it was timing out. I am still seeing the PKIX error on our production environment.

[Evan Brock]

Hello,

We're still unable to determine why your endpoint is encountering this issue when it was resolved for other users. I would like to kindly ask that you check to ensure that your endpoint is configured with the certificates that were recently updated from https://site.healthit.gov/certificates/xdr-tls/keyAndCert.zip , they should expire in 2034

I've recently checked the logs to confirm that the status of each test is the same from our last call, but if you'd like me to setup another call so we can discuss this, I'd be happy to do so.

[Braeden Rai]

We'll do another check of our installed certificates, and let you know if another troubleshooting call will be helpful. Thanks for continuing to look into this.

[Evan Brock]

Thank you, please keep me updated

[Braeden Rai]

Hi, unfortunately, after checking with one of our developers, we do have the certificates you specified, but are still experiencing the same issues with all the b1 XDR receive cases except XDR Test 3. Is there anything else we can check? We can jump on more troubleshooting calls, if that will help.

[Evan Brock]

Thank you for checking, I'll check in later this week when I determine the next step

[Evan Brock]

Hello,

If you'd like to have another call, may I have your availability for Monday?

[Braeden Rai]

Our team should be available at 9:30 central time on Monday, does that work for you?

[Braeden Rai]

If 9:30 central does not work, let us know if sometime this afternoon is better.

[Braeden Rai]

Does 2:30 central time today work for you?

[Evan Brock]

Yes, I will setup a meeting

[Evan Brock]

________________________________________________________________________________

Microsoft Teams Need help?

Join the meeting now

Meeting ID: 235 622 406 089

Passcode: 8tU33Qf3

---

Dial in by phone

+1 240-331-5581,,60439864# United States, Bethesda

Find a local number

Phone conference ID: 604 398 64#

For organizers: Meeting options | Reset dial-in PIN

Org help | Privacy and security

________________________________________________________________________________

[Evan Brock]

Forgot to mention on the call, the XDR send tests are working now if you were having issues with them too, I think I remember seeing a thread created by you for those. It is only the XDR receive tests experiencing issues now.

[Evan Brock]

Hello,

Would you mind setting up your endpoint with this certificate? We would like to test using it on our development environment before deploying the changes to our production environment.

[Evan Brock]

Can you also provide details on how you are installing the cert? Are you adding it to your default java trust store, cacerts?

[Michael Williams]

Evan,

I was able to import the certificate from .pem file; but it does not appear to contain the private key.  Can you confirm?

Many thanks,

Michael @Epic

[Evan Brock]

That should be fine, thank you, that helped us gain some valuable information.

We finally found a fix to this issue and I will be in touch tomorrow with the next steps

Thank you for your patience

[Evan Brock]

Hello,

The fix was deployed successfully, we will communicate any required actions on Monday

[Evan Brock]

Hello, please try the tests again. If they are still failing, make sure to reinstall the certs from the XDR testing page

[Braeden Rai]

Hi, XDR tests 4a, 4b, 8, and 9 are working now, thank you! We are still seeing an issue with test 5, and will respond back after we investigate a little more.

[Miles Williams]

It seems we are rejecting the message for XDR Test 5 because it does not have an actual base 64 encoded document in it. We get this error: 

We looked at the ITI-41 standards and think this is correct behavior for us; we'd expect this request to have some element like <xdsb:Document...>.

 

Do you know why the Test 5 request might be missing the document? 

[Evan Brock]

Hello Miles,

I'll look into this and see if I can figure out why that's happening

[Evan Brock]

Hello,

We are still working on this, but as the PKIX issue is resolved and this is now a different issue, can you please create a new thread describing the issue for our tracking purposes?

Thank you

[Miles Williams]

Sure, made a new one: https://groups.google.com/g/edge-test-tool/c/ETvS-j70D74/m/hT4GgNOQAQAJ

[Arslan Iqbal]

Thanks! Locking this thread. Please open a new ticket if any other issues occur. 

-SITE/ETT Team