The instance manager/scheduler spins up 4 additional instances to serve a request stream from a single browser session

[Carl Schroeder]

Normally, these requests take under a second to satisfy, two seconds at most. However at 2012-10-17 09:47:44.811 4 new instances were spun up for 4 requests in a row. It is my understanding that Chrome is limited to a certain number of concurrent requests per host. How could a single browser session overwhelm Google app-engine? The result on the back end was 5 instances where only one was needed. The result on the front end was 30 seconds of waiting around. After a few minutes, the scheduler begins killing off the new instances. Then the problem repeats itself.

There were no warmup requests issued. My app is set up to handle them.

This is behavior can be reproduced.

The appid is lemurspot.

Please make sure that the next time the engineers mess with the dark powers of the scheduler, they draw their pentagrams correctly. And don't use frozen goats blood. ;)

[Andrew Mackenzie]

Do you have thread safe to true in your appengine-web.xml?

Do you have your static files declared as such so that they are handled by Google CDN?

[Jason Collins]

I am very frustrated with the scheduler recently. It seems to have way too many instances running at any time; i.e., I _wish_ it were more aggressive about killing instances, or better yet, not spinning up extras when it's not actually necessary.

The most pathological case is when you ask for 1 Min Idle Instance (so that you get warmup requests), but if that instance is actually idle, the scheduler sends the request to a new cold instance instead of allowing the warm one that's sitting there to handle it.

We have recently switched to Python 2.7, threadsafe: true, and F4 instances believing that we would see a fair decrease in the number of instances that are running at any given time. So far, not seeing any meaningful difference. Except for the additional cost...

[Brian]

FYI - I noticed similar behavior today as well. The scheduler initiated three instances for my single remote API session to my python app. Weird!

[Carl Schroeder]

Yes to both.

[Kristopher Giesing]

When I looked into this recently, I discovered that GAE was actually sending the new request to the guaranteed idle instance, but then also spinning up another instance at the same time, making it seem as if the request were being handled by the cold instance.  The 2nd instance spinning up is working as intended; GAE is guaranteeing an idle instance, and the current warm instance is handling a request.

[Jason Collins]

Kristopher,

That's the way it _should_ work, so if true, that's great news!

I hadn't checked in the past couple of weeks.

j