Deadline Exceeded Exception on loading requests leaves bad instance running

[Jason Collins]

We've seen some Deadline Exceeded Exceptions cropping up on loading requests again. This used to be commonplace (a couple of years ago), but I haven't seen it for a long time... until recently.

Has anyone else seen this lately? We're seeing it on Py2.7. Normally, these apps warmup in a couple of seconds, so 60-70s startup times are completely abnormal.

Worse, when this occurs during a loading request, the instance is left in a "partially inflated" state - in our case, basically all subsequent requests fail with import/pythonpath issues. Our recourse, once we discover this is happening, is to manually kill the instance. This really, really sucks.

Can anyone make an argument why you would want to keep an instance that Deadline Exceeded Exceptions on a loading request? I can't. I think an instance that hits a Deadline Exceeded Exception during a loading request should be thrown away as suspect.

j

[Anand Mistry]

Hi Jason,

The engineering team is aware of this issue of instances being in a wacky state, however, I can't give you any more information.

[Sam]

Is there any update on this? In addition to instance time increases, we are also seeing huge datastore latency past few days and lot of DeadlineExceededExceptions. 

Our app id: textyserver