Multiple GAE instances being torn down for no apparent reason and problems with Instances view
50 views
Skip to first unread message
Patrick Tessier
Aug 11, 2017, 5:25:28 PM8/11/17
to Google App Engine
Hi,
I've been investigating some claims that our app is sometimes slow, which led me to discover that many times a day we have multiple Instances being shutdown, creating latency spikes as new instances spin up. Based on the Instances view in the GCP App Engine console, it appeared to be the result of exceeding the memory limit of our instance class. So I recently increased our instance class and decreased the max-concurrent-requests for our app. This change does appear to distribute most of the traffic across multiple rather than a single instance most of the time.
Unfortunately it appears that we still have multiple instances being shutdown now and then, whose memory usages (according to the Instances view) are far below what the new instance class allows. The other thing I noticed is that the traffic is then often sent to new instances, rather than to any idle instances (Dynamic) that may be present. (Note that I have later seen these dormant Dynamic instances start receiving traffic again).
The only solution that I see is to increase the minimum number of idle (Resident) instances, to accommodate having multiple instances suddenly being shutdown, but there must be a way to prevent the multiple instance shutdowns from occurring in the first place, or maybe not? Adding Resident instances would increase costs, for a reason that seems like it should be avoidable in my mind.
That's the main issue. The other problem is that I don't trust the GCP App Engine console Instances view itself. While conducting my investigation this morning, I've seen:
1) Active Instances with Start Times around 6:30AM suddenly appear in the list when I checked at 12:00PM, but they were not present in the list in the prior few hours that I had been checking the list.
2) An active instance (Dynamic) that was apparently using 0 B of Memory.
Is the Instances view having trouble lately, or am I trusting it too much as a reliable debug tool?
Thank you
I've been investigating some claims that our app is sometimes slow, which led me to discover that many times a day we have multiple Instances being shutdown, creating latency spikes as new instances spin up. Based on the Instances view in the GCP App Engine console, it appeared to be the result of exceeding the memory limit of our instance class. So I recently increased our instance class and decreased the max-concurrent-requests for our app. This change does appear to distribute most of the traffic across multiple rather than a single instance most of the time.
Unfortunately it appears that we still have multiple instances being shutdown now and then, whose memory usages (according to the Instances view) are far below what the new instance class allows. The other thing I noticed is that the traffic is then often sent to new instances, rather than to any idle instances (Dynamic) that may be present. (Note that I have later seen these dormant Dynamic instances start receiving traffic again).
The only solution that I see is to increase the minimum number of idle (Resident) instances, to accommodate having multiple instances suddenly being shutdown, but there must be a way to prevent the multiple instance shutdowns from occurring in the first place, or maybe not? Adding Resident instances would increase costs, for a reason that seems like it should be avoidable in my mind.
That's the main issue. The other problem is that I don't trust the GCP App Engine console Instances view itself. While conducting my investigation this morning, I've seen:
1) Active Instances with Start Times around 6:30AM suddenly appear in the list when I checked at 12:00PM, but they were not present in the list in the prior few hours that I had been checking the list.
2) An active instance (Dynamic) that was apparently using 0 B of Memory.
Is the Instances view having trouble lately, or am I trusting it too much as a reliable debug tool?
Thank you
Jordan (Cloud Platform Support)
Aug 14, 2017, 4:55:37 PM8/14/17
to Google App Engine
Google Groups is meant for general product discussions and not for project specific technical support. It is recommended to post details about your scaling configuration and code that is causing the 204 memory instance evictions on Stack Overflow using the support Cloud tags. If you are indeed seeing 204 errors in your logs, the only solution is to ensure your code stays within the memory limits of the instance class you have chosen in your configuration.
As for the discrepancies in the console, I'd recommend filing an issue report with detailed screenshots of the logs showing the instance ID in use before it even shows up in the App Engine instances pages, as there may indeed be a delay in the console.
Reply all
Reply to author
Forward
0 new messages