Any GAE EU region changes going on we should be aware of?

[Linus Larsen]

Last couple of days our app has started to behave strange, seems like our instances are shut down very rapidly. Our low traffic default instance could easily manage a whole day with just a few instances, until a couple of days ago. Now it starts around 100 instances a day for no apparent reason. A cron job is running at every half an hour, which should keep 1 instance alive all the time (or at least it used too), but now almost every of these requests fires up a new instance indicating that there was no instance running at all. 

Now, has anyone else noticing more spawning instances than before? 

It would be interesting to know why instances are shut down.

 

 

[George (Cloud Platform Support)]

Hello Linus, 

You seem to have 3 projects running at this time. Which one of the 3 is affected by the 100 instances for no reason at all behavior change? For confidentiality, you can send us the project name by private email. 

None of the 3 projects start more than a maximum of 17 instances at a time. When did this change in behavior occur, exactly? 

Looking at your versions in the console, one notices that "automatic" is chosen as value for the "Min Idle Instances" and "Max Idle Instances" parameters.   

Issue 13551 is tagged as fixed right now. The reported pub sub calls timed out, memcache down, datastore failures are unrelated issues, worthy of reporting on the public issue tracker on their own, if consistently repeating.  

[Linus Larsen]

After some digging I found out that we've made some modifications on the mobile app, which makes it do several ajax requests rapidly.

The result became somewhat strange.

1. Request1 starts a new instance

2. Request2 also starts a new instance since it takes aprox 20 secs to start a new instance, and the instance started in request1 isn't ready yet

3. Request3 also starts a new instance since the instance started in request1 isn't ready yet

Suddenly we have 3 instances running, and the response time wasn't improved since the startup time of each instance it so slow. 

I guess we can improve the behavior fiddling a bit with max/min-pending and min-idle instances settings.

But. As it seems, started instances are shut down very quickly, more quickly than they used too.

FYI, it all started last week (and it not just me who noticed these changes)

https://code.google.com/p/googleappengine/issues/detail?id=13551

And it keep on going with different kinds of issues, today pub sub calls timed out, memcache was totally down, datastore failures etc.

Very shaky, and it makes me wondering if Google is doing som changes in the EU region.

[Thanasis Delenikas]

Hi all,

I too have seen strange behavior following the datastore issue (#13551) - and this is going on every since.

Instances are a bit crazy going up and down, I've seen latency spiking (i.e. 20 seconds for a task that takes < 1sec and is datastore/memcached bound), memcache seems kind of "exhausted", tasks staying in queues for 2 minutes, etc.

All this behavior is definitely *very* different than before #13551 (which may be irrelevant but I'am mentioning it only as a time milestone).

 

Can you please let us know if you're experiencing issues with EU region?

[Kristof Degros]

Same here

I doublechecked my code as I made some changes, The dashboard shows yesterday one of the apps that I have, had 12gb traffic ... While I can browse through the logs with 2 refreshes

The dashboard is also acting up, today I already have 83 hours billed while only during 30 minutes a 2nd F1 instance was running if you look at the graph. So something is really up

Yesterday on one app I had 5 instances spin up, while I never have gone above 3 instances and I checked the logs and I had normal traffic. Nothing special, the usual traffic. I am using F1 instances

and in general my costs are around 0,0 to 0,20€ a day. Ever since 14th of Februaru I have unexplainable restarts, some 500 errors that I can not even explain. It is getting a bit crazy

I really wish these things get resolved quickly, cause I am starting to loose clients here