GAE keeps starting new instances causing lots of user facing loading requests

[Francois Masurel]

I've notice lots of user facing loading requests since a few hours (days?).

Just check the screenshot below.  Two new instances have been started to serve only 13 (12 + 1) requests though another instance was already running and available and obviously was not overloaded as it had only served 105 requests during the last hour :

My application instance settings are all set to automatic and pending latency set to 10s :

Did something change on GAE side recently ?

These user facing loading requests are really killing my business as my users are getting really upset about it.

Thanx for your help.

François

App Id = vncts1

[Francois Masurel]

After checking my logs for warming requests I can see that new instances have been started every 3 to 4 minutes for the last hours.

Is this normal behavior ?

[Francois Masurel]

It looks like old instances cannot serve any new coming requests :

Looks like GAE send almost all traffic to the new instance which is not fully initialized (lazy loading).

App Id : vncts1

On Saturday, November 17, 2012 4:21:28 PM UTC+1, Francois Masurel wrote:

[Igor Kharin]

I've seen such pattern with my apps. As far as I can see, I believe, this is perfectly OK given your performance settings.

Somehow your active instance's "queue" was full and scheduler decided to start a new instance. Since it was trying to serve the request for 20 seconds, and again, given your settings (max latency = 15s), scheduler found it bad and kept trying to send requests to the old instance. And again. And again.

There are like exactly two safe options for performance settings:

1. min idle instances = auto, max = 1, and a fairly high min latency (low cost);

2. min idle instances = 1+, max = auto (traffic spike);

Generally, AE engineers recommend not to mix "min" and "max" options and always set the opposite slider to "auto", as settings like in your case (min latency = 10s, max = 15s) confuse scheduler's heuristics. My advice would be to set "max pending latency" to auto and see what happens. Then tune the "idle instances" setting as desired:

1. max idle instances = 1 so you wouldn't pay for surplus instances;

2. OR set min idle instances to 1+ as you obviously seeing a little spike.

Also, keep in mind that resident instances are not really what we had with Always On--treat them as "fallback instances". To wrap up, in your case, if you want to keep users off of startup requests completely (20s is no good):

1. set up a cron "ping" task to keep your dynamic instance "warm";

2. set min idle instances to 1 (do not forget to handle warmup requests);

3. tune latency settings (say, min = auto, max = 1s to try to evenly distribute load between resident and dynamic instances);

Hope that'll help.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ylBW4ldXw_cJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Francois Masurel]

Hi Igor,

Thanx for your nice and long answer.  I understand and I'm ok with most of what you said.

The only thing that annoys me is that my app is a Java app with threading enabled and as you can see on the screenshots the main long-lived active instance doesn't do anything : extremely low latency and almost no traffic served (no traffic at all between second and third screenshot).

So, no, the active instance's queue is not full unless everything goes nowhere or the dashboard requests count is wrong.

What do you think ?

Thanx again for your help.

François

PS:

And without changing any settings everything has been working fine for months.

[Igor Kharin]

You are right. That's precisely why I said "somehow" there--scheduler can go wrong. Maybe the app had a little spike for a sec or it's really just tablet your instance were running on had a little hiccup. Actually, it seems like old instance were plain dead to scheduler for a while.

Here it starts to be interesting, even though we had max latency = 15s and new instances with 20s latency were ineligible, it only started like three instances until the scheduler figured out that it doesn't help and decided to fed requests to existing ones. Then performance settings come into the game: as scheduler "somehow" found instance to be slow it was trying to use new, "hotter" instances (screenshots 3-4) and nicely sent only two request to the old one. And since we haven't explicitly stated how much idle instances we want, the scheduler might have guessed (albeit, mistakenly) -- OK, this app needs two instances right now.

Just my speculative optimistic assumptions, but it's kind of remarkable. It'd be cool if the scheduler did improve that much. I've had worst: absolutely no traffic other than me, yet new instances were spinning off with no apparent reason. The issue have disappeared within a couple days. I am to wordy again, so: try to set max latency to auto, plus add a resident instance for a while. Even if problem stays, it should keep scheduler from running new instances.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/uErXDY8EB9UJ.

[Francois Masurel]

Hi Igor,

I have followed your advice and set max latency to auto and min idle instance to 1 :

I'll tell you how it goes.

Thanx again for your help.

François

[Francois Masurel]

The related issue is getting some activity lately: http://code.google.com/p/googleappengine/issues/detail?id=7865

[Francois Masurel]

My instance reloading problems are probably related to this :

https://groups.google.com/forum/?fromgroups=#!topic/google-appengine-downtime-notify/TYEvpd8PtUk

Why can't we be informed of such important updates ?  Is GAE still in beta or what ?

Please Google communicate about all those impacting changes.

Thanx.

On Saturday, November 17, 2012 4:21:28 PM UTC+1, Francois Masurel wrote:

[Francois Masurel]

Could frequent instance loadings be related to the use of multiple custom domains on the same app ?  Looks like it the case for my app : vncts1

And why GAE seems to send all traffic to the new fresh instance instead of using the other already loaded instances ?

This is pretty annoying as we use lazy loading a lot and new instances take some time to get fully initialized.

How come GAE needs 3 instances to display a simple home page with few javascripts and CSS ?

There is really something wrong with the GAE instance scheduler.

What should I say to my customers about that ?  I feel really desperate about this.

Will it improve sometimes in the future ?  or is it normal behavior for GAE and will never be fixed ?

Will I have to pay 600$ to get an answer ?

Thanx for your help.

François

On Saturday, November 17, 2012 4:21:28 PM UTC+1, Francois Masurel wrote:

[Francois Masurel]

Could it also be related to really bad Get latency lately ?

Let's compare today's Java dynamic get latency with April 25 numbers :

Today : 

04/25/2012 :

In my case, some requests running most of the time in less than 200ms can take as much as 7s without any instance loading going on.

When will things start to stabilize a bit so we can run real businesses on GAE ?

Am I the only one encountering such problems ?  I sometimes feels like being on a desert island here :-)

François

On Saturday, November 17, 2012 4:21:28 PM UTC+1, Francois Masurel wrote:

[Francois Masurel]

Example of two identical requests doing a simple MemCache fetch with quite different excecution times 385ms vs 4502ms :

2012-11-27 12:19:25.809 /chateau-de-mouchac-183001.dhtml?m=publications&l=fr&d=d 200 385ms 14kb Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.91 Safari/537.11

I 2012-11-27 12:19:25.511 com.mably.cms.web.PageCacheServlet doGet: At program start we have : 20535256 bytes

I 2012-11-27 12:19:25.512 com.mably.cms.theme.web.CookieDeviceResolver initRenderDevice: Device = {browser:chrome,type:DESKTOP,bot:false}

I 2012-11-27 12:19:25.512 com.mably.cms.web.PageCacheServlet doGet: Page is cacheable : view|aghzfnZuY3RzMXIJCxIBZBjZlQsM|publications|fr|d

I 2012-11-27 12:19:25.512 com.mably.cms.web.PageCacheUtils getPageCacheFullId: Added to ThreadLocal : mouchac.vinocities.com|default|view|aghzfnZuY3RzMXIJCxIBZBjZlQsM|publicati

I 2012-11-27 12:19:25.520 com.mably.cms.web.PageCacheServlet$PageCacheChecker check: In cache, verify OK :mouchac.vinocities.com|default|view|aghzfnZuY3RzMXIJCxIBZBjZlQsM|publi

I 2012-11-27 12:19:25.520 com.mably.cms.web.PageCacheServlet doGet: After running the program, we have : 20071256 bytes

2012-11-27 12:19:09.395 /chateau-de-mouchac-183001.dhtml?m=publications&l=fr&d=d 200 4502ms 14kb Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.91 Safari/537.11

I 2012-11-27 12:19:09.325 com.mably.cms.web.PageCacheServlet doGet: At program start we have : 23017432 bytes

I 2012-11-27 12:19:09.326 com.mably.cms.theme.web.CookieDeviceResolver initRenderDevice: Device = {browser:chrome,type:DESKTOP,bot:false}

I 2012-11-27 12:19:09.326 com.mably.cms.web.PageCacheServlet doGet: Page is cacheable : view|aghzfnZuY3RzMXIJCxIBZBjZlQsM|publications|fr|d

I 2012-11-27 12:19:09.327 com.mably.cms.web.PageCacheUtils getPageCacheFullId: Added to ThreadLocal : mouchac.vinocities.com|default|view|aghzfnZuY3RzMXIJCxIBZBjZlQsM|publicati

I 2012-11-27 12:19:09.338 com.mably.cms.web.PageCacheServlet$PageCacheChecker check: In cache, verify OK :mouchac.vinocities.com|default|view|aghzfnZuY3RzMXIJCxIBZBjZlQsM|publi

I 2012-11-27 12:19:09.339 com.mably.cms.web.PageCacheServlet doGet: After running the program, we have : 22546336 bytes

What am I supposed to say to my customers ?

[Brandon Wirtz]

Apps serve only 8 requests concurrently. If you have your min/max pending latency set wrong you may have a scenario where the 12-16 requests that a browser will make at a time pushes you to 2 instances with 1 idle.

 

 

From: google-a...@googlegroups.com [mailto:google-a...@googlegroups.com] On Behalf Of Francois Masurel
Sent: Tuesday, November 27, 2012 3:51 AM
To: google-a...@googlegroups.com
Subject: [google-appengine] Re: GAE keeps starting new instances causing lots of user facing loading requests

 

Could frequent instance loadings be related to the use of multiple custom domains on the same app ?  Looks like it the case for my app : vncts1

 

And why GAE seems to send all traffic to the new fresh instance instead of using the other already loaded instances ?

 

This is pretty annoying as we use lazy loading a lot and new instances take some time to get fully initialized.

 

How come GAE needs 3 instances to display a simple home page with few javascripts and CSS ?

 

There is really something wrong with the GAE instance scheduler.

 

What should I say to my customers about that ?  I feel really desperate about this.

 

Will it improve sometimes in the future ?  or is it normal behavior for GAE and will never be fixed ?

 

Will I have to pay 600$ to get an answer ?

 

Thanx for your help.

 

François

 

 

 

On Saturday, November 17, 2012 4:21:28 PM UTC+1, Francois Masurel wrote:

I've notice lots of user facing loading requests since a few hours (days?).

 

Just check the screenshot below.  Two new instances have been started to serve only 13 (12 + 1) requests though another instance was already running and available and obviously was not overloaded as it had only served 105 requests during the last hour :

 

My application instance settings are all set to automatic and pending latency set to 10s :

Did something change on GAE side recently ?

These user facing loading requests are really killing my business as my users are getting really upset about it.

Thanx for your help.

François

App Id = vncts1

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/L2D3Iwo-FQYJ.

[Francois Masurel]

Hi Brandon,

Thanx for chiming in.

Min latency is set to 10 and max to auto.  Should I change something ?

BTW do you know if PageSpeed can be used as some kind of CDN to offload some traffic from my instances ?

It doesn't seem so as most requests are logged twice when PS is actived, one for the PageSpeed client and one for the final client.  If your code is already optimized it seems pretty useless then.

Thanx for helping.

François

[Francois Masurel]

What is really strange is that appstats shows that everything is going fine on GAE side though logs shows a very high latency for the same request, check below:

AppStats :

Same request in the logs :

1. 2012-11-27 23:03:18.921 /?d=d 200 7591ms 23kb Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.91 Safari/537.11

2. I2012-11-27 23:03:18.720 com.mably.cms.web.PageCacheServlet doGet: At program start we have : 15761400 bytes

3. I2012-11-27 23:03:18.721 com.mably.cms.theme.web.CookieDeviceResolver initRenderDevice: Device = {browser:chrome,type:DESKTOP,bot:false}

4. I2012-11-27 23:03:18.721 com.mably.cms.web.PageCacheServlet doGet: Page is cacheable : home|d|en

5. I2012-11-27 23:03:18.721 com.mably.cms.web.PageCacheUtils getPageCacheFullId: Added to ThreadLocal : fr.filhot.net|default|home|d|en

6. I2012-11-27 23:03:18.748 com.mably.cms.web.HomeServlet verifyPageCache: com.mably.cms.cache.SerializableCacheObject@19bf7d0[object=com.mably.cms.cache.PageCacheData@aaa392,tag

7. I2012-11-27 23:03:18.777 com.mably.cms.web.PageCacheServlet$PageCacheChecker check: In cache, verify OK :fr.filhot.net|default|home|d|en

8. I2012-11-27 23:03:18.778 com.mably.cms.web.PageCacheServlet doGet: After running the program, we have : 14724712 bytes

 7591ms  instead of   58ms , huge difference no ?

I can reproduce the same behavior from home and from work, so it's not related to my internet connection.

Why do we have such high latency for this trivial request ?  A 7 seconds wait and your users are gone.

Any help will be greatly appreciated.

François

[Nik Eleftheriou]

I can confirm that we are seeing same behaviour in our app.
we can usually serve everyone with 6 instances, but in the last 6 weeks something has been changed and even though at any one time we have 6-7 instances, we have 1000 warmups a day.

so definitely something caused the scheduler to change behaviour as we only had 10 warmups a day before Oct.
We've not made any code changes to our app in 5 months.

nik.

[Per]

We had a very similar problem (see my other post). It's resolved since two days or so, after some quite bumpy weeks. Latency is down, and "spikyness" of the dashboard charts is down too, everything runs a lot smoother since yesterday.

 Maybe we're just lucky, but maybe also something has actually been reconfigured and we're just the first who moved to new smarter settings, and you'll follow soon.

Do you guys happen to use https by the way, and do you track latency externally like with Pingdom or so?

Cheers,
Per

[Nik Eleftheriou]

We use https under the appspot domain. 90% of our requests are https.
running a Java webapp that is 2 years now on appengine.

I've not been tracking latency with an external service. Our clients(retail) are a very vocal bunch. 3 seconds of "Please wait..." and they're on the phone.

[Francois Masurel]

We are only using https with appspot domain for our Facebook fan page app.

But on the other side we are using a lot of custom domains, our app is some kind of multi-host CMS.

I have read somewhere that custom domains are not performing as well as appspot domains.

François

[Peter Ondruška]

If you use appspot domain SSL is available and you might benefit of SPDY, https://developers.google.com/appengine/docs/python/runtime#SPDY

[Nik Eleftheriou]

Right about the time i noticed the scheduler had started cycling instances every 3-4 minutes, I also noticed that our QPS has dropped to 30% of what it used to be.
Instead of ~1 QPS average we now have 0.3 QPS average.
Seems to be the same for Master/Slave and HR datastore apps.

Anybody else noticed similar?

nik.

[Cesium]

I can confirm that we are observing increased latency and bothersome startups of new instances.
David

> My application instance settings are all set to automatic and pending latency set to 10s :
>

[Francois Masurel]

High latency today, things are getting worse :

On Saturday, November 17, 2012 4:21:28 PM UTC+1, Francois Masurel wrote:

[Francois Masurel]

Fasten your seat belts...

[Nik Eleftheriou]

it must be the weekend

[Francois Masurel]

Still starting a new instance (2d one in the list) and redirecting traffic to it (24 requests) though two idle instances are available doing nothing.

FYI this new instance was killed 30 seconds later.

I really can't understand this stupid behavior : start a new instance (even if others are idle), block all traffic until it is started, redirect traffic to it (if you dont get a timeout) and then kill it a few seconds later.

Must be a way to improve this, no ?

François

[Francois Masurel]

Geez, that's a lot of instances for a few Ctrl+F5 on my home page :-)

[Francois Masurel]

Again starting 5 instances (cf. below) to serve a simple Facebook page (https on appspot domain) and one is a user facing request taking 10 seconds to execute :-(

I would definitely prefer the user waiting 1-2 more seconds (BTW that's what the pending latency setting was supposed to do)  than starting a fresh new instances and wait from 10 to 20 seconds (in our case) for the new instance to be fully initialized.

May be we should be able to manage manually our instances until Google fixes the scheduler.

[Francois Masurel]

The problem is still there :

We can see above a 7 seconds user-facing loading request.

Francois

[Francois Masurel]

Yep, things seemed to have improved a bit before Xmas but we are back to wild instance loading again like before.

Here we have one resident instance, one dynamic instance started one hour ago getting almost no traffic (13 requests) but still a new dynamic instance is created :

This really needs improvement.

App ID : vncts1 with several custom domains (~20)

On Wednesday, January 9, 2013 11:56:36 PM UTC+1, Jim Briggs wrote:

I'm having the same problem and have been having it for at least a week.  Instances get started while other instances are not being used.  I wonder what the commonality is between your and my problem.  I'm also using a custom domain.  I've tried different configurations of idle instances and pending latency and nothing seems to make a difference.  I wonder why more people aren't complaining about this problem.

[Francois Masurel]

Again, starting a new instance to serve 3 requests though the resident instance has a latency of 48ms for the last minute :

Our customers keep telling us that their site is too slow.

Google could you tell us when this will be fixed ?

Thanx in advance.

François

[Santiago]

Have you tried setting up your Pending Latency from automatic to automatic? Just to see its behavior? My apps are configured on that way and we haven't had those problems... Average latency 106.2 ms all the time. 

[Francois MASUREL]

I'll do that and tell you how it is going.

At the moment it is 10.0s – Automatic.

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/FfQ_Xmsmt_EJ.

[Francois Masurel]

Still starting new dynamic instances though older ones are available :

Rather strange...

Will stay like that a few hours to see how things are doing...

[Francois Masurel]

Same thing happening a few minutes later :

[Jim Briggs]

I've been having the same problem for a couple weeks under extremely low load.  I'm wondering what the commonality is and how widespread the problem is. 

[Mark]

I am also experience this problem.  My response times are all over the place, and I get very frequent instant restarts.  

My id is bedbuzzserver (Java / High replication)

[Carl Schroeder]

We still have this bug in spades.

This bug makes java versions that are not the default version completely unusable. There are no resident instances for non-default versions. So unless your traffic is already enough to keep more than one instance in memory, you will have lots of user requests hitting cold instances. This results in consistent 15+ second response times for some java app versions. I don't see how you can get enough traffic to sustain more than one instance with 15+ second response times being the norm.

As a result, we had to port or fold away all of our secondary app versions to python because java is not viable for that use case. This was a huge duplication of effort for a small team. GAE java is dead to us for new features.

We can no longer give access to beta java versions (they are non-default) because they are unusable in low traffic scenarios. This stings the most, since our users can no longer critique any versions except the "default" one.

[Kristopher Giesing]

I set up a duplicate app ID for beta testing.  Would that work for you?

- Kris

On Thursday, January 10, 2013 1:33:21 PM UTC-8, Carl Schroeder wrote:

We still have this bug in spades.

[Carl Schroeder]

Unfortunately, the contents of the datastore drives our user experience completely. Unless the 2 apps shared a datastore view, it would have limited value to us.

This bug also sinks the ability to do A/B testing. Since the B group would be using a non-default app version, their experience will be degraded to the point where they would hate anything put in front of them. 

A duplicate app ID can be used to do limited QA testing, but it would not have access to our user's data collections, thus not be appropriate for A/B or Beta testing.

[alex]

Carl, your case seems to be fitting nicely with the new Servers feature:

On Fri, Dec 7, 2012 at 12:17 AM, Chris Ramsdale <cram...@google.com> wrote:
> We're excited to announce a new App Engine feature called "Servers" that
> allow developers to segment large-scale applications into logical components
> that are able to share stateful services and communicate in a secure
> fashion. For example, applications that have multiple frontends which
> handle web-based and mobile-based traffic or applications that have various
> backends for data analysis, billing pipelines, etc.
>
> Part of building great products is gathering quality user feedback early on.
> If you're interested, we encourage you to sign-up using the following form.
> Based on input we'll select a small group of trusted testers.
>
> https://docs.google.com/a/google.com/forms/d/1qjuLxnAHQeq2YQMCMAITbC6St_NrfzHDxrbkuTqvmfY/viewform
>
> Cheers!
>
> Chris Ramsdale
> Product Manager, Google App Engine

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-appengine/-/444KTGNdV44J.

[Carl Schroeder]

Yes it does. Unfortunately, I don't have the bandwidth currently to port our production system to an experimental service.

At this time, I need to get the GAE features (load balancing & A/B testing) in our production services running with sub 10 second response times.

[Mark]

Please can someone look at this issue??  It isn't normal behavior.  My app was working fine until a few weeks ago - when I started getting very frequent instance restarts.  I haven't uploaded any new code in a few months.

I did write up a bug: http://code.google.com/p/googleappengine/issues/detail?id=8298

Please check my logs.. for example at 10:34:48 today I have a request which took 65ms.  Then at 10:34:55 the same request takes 9941 ms.  This is happening way to frequently.  

app id: bedbuzzserver

[aswath satrasala]

Hi,

my app id:  accountingguru-india.  It is a paid account.  

I am also seeing frequent instances being initialized. 

Some data points of new instances getting created.

2013-01-14 14:01:23.674  

2013-01-14 13:59:44.969

2013-01-14 13:49:49.791

2013-01-14 13:45:37.388

-Aswath

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Gym-HWS48rMJ.

[Francois Masurel]

Starting instances like crazy again.... with quite a few user-facing loading requests (18s here) :-(

[Francois Masurel]

GAE just started two instances for a simple page refresh with a 12s wait for my users  :-(

It takes normally a few ms to load this page (simple fetch from datastore) :

You can try to refresh (Ctrl-F5) the page by yourself here :

http://www.chateaudemouchac.fr/architecture-184002.dhtml?d=d

Tell me how it goes.

François

[Francois Masurel]

Again...

[alex]

I wonder how you guys do real work while keep sending screenshots every 5 mins, which starts getting really annoying.

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/OElSXLh5J8wJ.

[Francois Masurel]

Sorry for the annoyance, I'll start a new thread for my screenshots.

I can't stop though coz that's the only thing that makes me feel better :-)

François

[Carl Schroeder]

Unfortunately, real work on our GAE java stories has to be on the back burner because of this issue. Plenty of work porting previous features though. But not so much work that we can't keep track of a couple of threads about critical infrastructure issues. ;)

Masters of multitasking we are. The lack of unicorns drives us.