Cloud load balancer suddenly starts to return 502 errors; log message failed_to_connect_to_backend

[Craig Finch]

Around 7:00PM Eastern time last night, we started to see 502s at a high enough rate that end users were affected. We haven't changed anything in our application or infrastructure. Is anyone else seeing similar problems?

These errors occur with a very simple configuration: one CentOS 7 instance running Apache/PHP behind an HTTP(S) cloud load balancer. This system has been stable in production for a long time. We haven't deployed any changes to our application recently, and our last infrastructure change occurred almost two weeks ago. The error rate is high enough to trigger our external responsiveness monitor. We can't find any logs or metrics that indicate a problem anywhere in our GCP systems, except for the 502s and the failed_to_connect_to_backend errors in the load balancer logs. The problems started a period of light usage, and neither the PHP server nor its SQL database were under heavy load. 

Increasing the Apache TimeOut to 650 seconds did not reduce the rate of 502s.

Let me know if there is any other information that I can provide to help diagnose and resolve this problem.

This ticket is related, but for a Kubernetes cluster rather than a VM: https://issuetracker.google.com/issues/76200990

Related ticket: https://issuetracker.google.com/issues/79869362

[Navi Aujla (Google Cloud Support)]

Hello Craig 

As we are working on your reported private issue report 79869362, and investigating/verifying any issue on our end which may cause this reported behavior, I suggest you to keep track of your private issue report 79869362 for any further updates and post any comments.