Recent elevated error rates for FHIR stores

[Philip Roberts]

Over the last few weeks, and particularly today and yesterday, we have been seeing an elevated level of 503 "The service is currently unavailable." errors when making requests to our FHIR stores. Particularly with writes (1.5% error rate in the last hour) but reads are also affected (0.15% error rate).

Is anyone else struggling with an elevated error rate here? I have opened another ticket with support, last time they said our write-rate was high, but we're seeing issues even in our staging project which doesn't have especially high throughput.

[Truc Le]

Thanks for the notification. Our production team has been aware of the issue and is working on it.

[Philip Roberts]

Oh that's good to know, thanks!

I there a status page we can follow along for events like this? I was expecting it to see it show up here: https://status.cloud.google.com/products/zgodfdJcHiKkGxQYixiZ/history or in our personalized status dashboard, but I don't see anything on either?

Thanks again.

--
You received this message because you are subscribed to a topic in the Google Groups "GCP Healthcare Discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gcp-healthcare-discuss/knVC9e6q2_E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gcp-healthcare-di...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/gcp-healthcare-discuss/53cdd28b-4c1a-47c9-b4b9-0e0a9ac4a125n%40googlegroups.com.

[Truc Le]

As service restoration is underway, could you please confirm if the 503 error rate observed from your end is decreasing?

Regarding the status dashboard, the severity of this issue does not necessitate an update to the service health status, given that the error rate percentage and elevated latency are predominantly observed at the 90th percentile.

Best,

Truc

[Philip Roberts]

Hi Truc,

I can confirm in the last 20 minutes or so we've seen no errors.

> Regarding the status dashboard, the severity of this issue does not necessitate an update to the service health status, given that the error rate percentage and elevated latency are predominantly observed at the 90th percentile.

Can you say more about this? What do you mean by the 90th percentile - percentile of what?

We have seen similar issues over the last month or so, where the 503 unavailable error takes around 30s to come back to us. Are you able to give me an idea of what would happen if we were to initiate a client-side request timeout of say 5s - would there be a risk of the request partially completing or something, or would this be a suitable approach to minimise the impact of these kinds of issues?

Thanks for your responsiveness, it's much appreciated!

Philip

To view this discussion visit https://groups.google.com/d/msgid/gcp-healthcare-discuss/d5bb98dd-554b-487c-9ed6-bd66c8507c11n%40googlegroups.com.

[Truc Le]

We have observed an increase in latency for the 90th percentile of our overall traffic. Please note that inclusion on the status dashboard typically requires an issue to be of sufficient severity, such as a zone outage or complete API unavailability.

We have conducted a thorough investigation of the system stack, which led to the identification of a regression affecting the underlying technology across multiple cloud products. This regression was introduced approximately 1.5 months ago and is currently being rolled back. This action is anticipated to significantly mitigate any anomalous behavior or increased latency you may have observed recently.

Concerning the client request timeout, could you please specify which APIs you are utilizing? For most FHIR APIs, you may assume a 1-minute deadline.

Best,

Truc