502 "failed_to_pick_backend" errors, seems like both workers died at the same time

[Steve Anderson]

Project ID: firebase-lessthan3

Service: Action

At 8:08PM PDT tonight we experienced about one minute of 503s from our data pipeline ingenstion services on google app engine. No one at our company triggered a restart, and the logs didn't show much besides 502 failed_to_pick_backend.

What was the cause of this? And how could we prevent this in the future? This app handles our data ingestion pipeline for our analytics, and its of the upmost priority that we don't this service doesn't go down.

Thank you.

[Kenworth (Google Cloud Platform)]

This is odd, I can't seem to find any "failed_to_pick_backend" error logs for that timeframe, even on expanded search. Do you have an exact timestamp of a sample error?

I also found this Google Groups and StackOverflow case that may provide solution to your issue. It discusses the possibility whether a database connection or a Load Balancer issue, among others.