Container terminated due to sandbox error.

[Mohamed Zoweil]

Today we started seeing our deployed services on AppEngine standard environment fail with a lot of warnings saying "Container terminated due to sandbox error."

The code also shows a lot of 500 OPTIONS requests, which has the following INFO log message

0: {

logMessage: "This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application."

severity: "INFO"

time: "2021-11-19T23:57:23.329869Z"

}

Splitting or migrating traffic to previous versions that we deployed and were working before did not solve the problem, the service would start for few minutes then run into the same issue. Since this is a Standard Environment, we cannot stop the instances, instead we can re-deploy, and when we did that the service started working but again only for few minutes before it fails.

We suspected a recent change in our code that might have resulted in throwing those 500 errors, and thought that these errors are causing AppEngine to kill the instance (per docs it says that 20 consecutive 5xx would make AppEngine kill the instance). We deployed this fix to both prod and staging, and both worked, but after some time ONLY staging started failing this time. Prod is still working even an hour later.

The only difference in configurations between staging and prod are:

1. Prod is class F4 while staging is F2
2. Prod has min_idle_instances set to 3 while staging set to 0
3. Prod has max_instances set to 5, while it's not set in staging

Would setting min_idle_instances in staging to something > 0 help fix this, but even if, the question remains, why did things suddenly started failing.

Any help or ideas?

[Sergi (Google Cloud Platform Support)]

Hello,

Have you tried to deploy the service with Flex environment to see if it works correctly with that environment?

However, as it appears that these inquiries would be best addressed by the Cloud App Engine team as they can provide a more accurate answer about technical issues, I would suggest to please submit your inquiry directly to the team using this public issue tracker template. The Cloud App Engine team will respond and update you as needed through that thread.