503 errors - default liveness checks and readiness checks

[Kush Goyal]

In my logs I am seeing 503 for readiness checks. These are the default checks so they do no hit my application.

Most of the 503 errors are for readiness. But in between there are 503 errors for liveness checks as well.

- Can someone help me identify the reason behind this?

- What are the implications of this? Does it restart my instances?

- How can I remove these 503 errors?

- Should I extend these health checks as documented here [1]?

It would really help if someone from the App engine team can explain this.

I am using app engine flexible with python 3 runtime.

[1] https://cloud.google.com/appengine/docs/flexible/custom-runtimes/configuring-your-app-with-app-yaml#health_checks

[yka...@google.com]

This could be due to your app taking long to startup and failing the readiness checks. Another possibility is resources on the instances are maxing out.

An instance will be restarted if it fails consecutive health checks [1].

To avoid this, you can tune health checks to be less aggressive by increasing check timeouts or intervals between checks [2]. You can also try to increase the memory in the app.yaml file as a test, optimum value depends on the app’s workload. Additionally, this doc [3] on deadline errors might be helpful to optimize your app.

[1] https://cloud.google.com/appengine/docs/flexible/python/how-instances-are-managed#health_checking

[2] https://cloud.google.com/appengine/docs/flexible/custom-runtimes/configuring-your-app-with-app-yaml#readiness_checks

[3] https://cloud.google.com/appengine/articles/deadlineexceedederrors