Lots of 502 errors

[Andrew Benton]

I'm not sure how much this has to do specifically with the application in question, but an otherwise well-functioning appengine app will start seeing lots and lots of 502 errors from Google on occasion.

This is extremely disruptive to the app's users and requires some manual data cleanup each time, which is frustrating. There are no logging indicators that show these 502 responses, so I can only presume that the response is generated upstream from the application's infrastructure.

Any idea what's going on? Here is some more information about the app:

It's a golang app running on the flexible environment.

It uses a custom domain, with cname pointing to ghs.googlehosted.com.

It relies on the appengine datastore, although even requests to handlers that make no datastore connections are failing with 502.

I feel like this is a routing/timeout problem between ghs.googlehosted.com and the appengine infrastructure, given the lack of logging. Can anyone from Google comment on that hypothesis?

[Kenworth (Google Cloud Platform)]

502 bad gateway usually means that the Nginx proxy responsible for handling requests for App Engine instance has not been able to get in contact with your application, the application raises an exception, an internal API call raises an exception or if the request simply takes too long.  As such, 502 does not tell you much about the real source of the problem. 

You can check if there is a pattern for the error (e.g. is it happening on the same endpoint). You can also try to perform exponential backoff in which retry a failed request only after waiting for a small delay to make sure the application does not get overloaded with requests. You might also be interested with 'error_handlers' element on your config file to possibly catch 502 errors related to over_quota or timeout, etc.