too many appengine frontend instances

[blackdew]

my server make too many Frontend instances

may be it make instance per request

from 2 hours ago

and my last deploy was yesterday may be 24 hours ago

so, help and give me any advice

thanks

[Ryuuji Yoshimoto]

me too.

https://yabumi.cc/145df20f7a96a1945c3c7cd3.png

Ryuuji Yoshimoto 

[James Foster]

I'm seeing the same thing. Between 10x and 30x the regular number of instances, even though the number of requests is the same as usual. Perhaps a problem with v1.9.5?

Status page seems to be showing everything but datastore as normal, though.

https://code.google.com/status/appengine

If anyone files an issue, please link it here so that we can star it.

[Ryan Barrett]

me too. it seems like many dynamic requests, if not all, are spawning new instances. very odd. seems like the scheduler is unhappy.

[mar...@ameeker.com]

Our application has the same issue too starting a few hours ago, spike to 70+ instances while normally it should just run on 2 to 3 instances...

[Kyoung-chan Lee]

We have same issue....

Usual time : 2~3 instances.

Now : 118 instances.

[Ice13ill]

I have the same issue, about 230 open Frontend instances. All instances seam to be running on version 1.9.5.

[Prabhakar Raikwar]

I am also facing the same issue. New instances are getting created for every request.

[Teddy Yew]

I have the same issue too with Java, v1.9.5. 

[Rafael]

any response from google?

I hope they figure out a way to reimburse people for these hours (getting billed for 800 instances instead of 30): https://www.dropbox.com/s/7gd9x5uwhnoulr5/Screenshot%202014-05-09%2000.14.34.png

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

[Pardeep Singh]

We are also facing the same issue. Normal 2-4 instances are there. But its currently 60+ instances with 1-2 request per instance. I hope this issue is resolved asap.

On Friday, May 9, 2014 9:10:19 AM UTC+5:30, blackdew wrote:

[Nik Graf]

Same here. Quite a lot of instances spawned in the last couple hours. see http://cl.ly/image/0y35321f1r3J

[timh]

I am still on 1.9.4 and not seeing this.

Is everyone experiencing this rolled over to 1.9.5 ?

T

On Friday, May 9, 2014 11:40:19 AM UTC+8, blackdew wrote:

[Andrei Cosmin Fifiiţă]

The issue begun with v. 1.9.5 but now the new instaces are created with 1.9.4 so i have about 220 instances on 1.9.5 and 110 on 1.9.4.

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/tlQPHCvP-14/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.

[Andrea Cimino]

Very sad that after 12 hours I have still the issues:

9.5	6 total	0.011	14859.5 ms	70.9 MBytes
1.9.4	10 total

and no feedback from Google. And I think that in a few hours i'll be overquota again.

[Dirk Groten]

I'm having the same issue, my app stopped running due to going over the budget, but it shouldn't have started all these instances in the first place, our traffic for this app is nothing unusual.

[Eric Ka Ka Ng]

we are with >200 frontend instances too...... the overall performance is very slow, and the cost becomes huge

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.

[Miroslav Genov]

Same on my side. Is anyone doing something about this? 

[Bogdan Nourescu]

Hello,

I have the same problem on multiple applications. Each request in creating a new instance.

[Florin-Alexandru Stancu]

Hi,

I'm having the same issue with two apps.

I see over 80 instances running while my applications usually have ~ 5 each.

The instances are split: about 4-5 are for the 1.9.5 version and the rest with v1.9.4.

On Friday, May 9, 2014 6:40:19 AM UTC+3, blackdew wrote:

[Florin-Alexandru Stancu]

Hello guys,

I found this on stackoverflow:

http://stackoverflow.com/questions/23558319/app-engine-too-many-open-instances

I'll try it and see if it works ;) 

On Friday, May 9, 2014 6:40:19 AM UTC+3, blackdew wrote:

[Miroslav Genov]

Here is the information from my admin dashboard: 

QPS*	Latency*	Requests	Errors	Age	Memory	App Engine Release	Logs	Availability	Shutdown
0.000	0.0 ms	3	0	0:14:44	133.3 MBytes	1.9.4	View Logs	Resident
0.000	0.0 ms	2	0	0:04:56	123.4 MBytes	1.9.4	View Logs	Dynamic
0.017	661.0 ms	5	0	0:11:16	137.6 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	1	0	0:02:36	114.6 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	2	0	0:03:57	140.8 MBytes	1.9.4	View Logs	Dynamic
0.017	3229.0 ms	5	0	0:09:59	140.7 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	1	0	0:10:19	130.7 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	2	0	0:08:19	113.1 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	4	0	0:06:02	142.0 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	4	0	0:09:28	137.7 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	1	0	0:10:36	117.1 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	2	0	0:06:44	133.9 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	2	0	0:06:36	133.0 MBytes	1.9.4	View Logs	Dynamic
0.033	0.0 ms	1	0	0:00:49	121.7 MBytes	1.9.5	View Logs	Dynamic
0.000	0.0 ms	4	0	0:08:47	134.7 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	5	0	0:12:43	139.0 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	2	0	0:04:52	133.1 MBytes	1.9.4	View Logs	Dynamic
0.017	6317.0 ms	3	0	0:10:24	131.4 MBytes	1.9.4	View Logs	Dynamic
0.017	30587.0 ms	1	0	0:01:34	129.6 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	5	0	0:08:26	142.4 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	1	0	0:12:01	126.2 MBytes	1.9.4	View Logs	Dynamic
0.017	2349.0 ms	4	0	0:03:50	124.3 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	2	0	0:11:05	131.3 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	2	0	0:11:32	112.9 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	4	0	0:12:43	134.7 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	3	0	0:07:09	113.8 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	2	0	0:09:32	122.4 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	5	0	0:06:33	144.5 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	4	0	0:10:18	133.7 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	3	0	0:10:17	137.8 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	6	0	0:11:31	127.8 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	6	0	0:07:53	138.0 MBytes	1.9.4	View Logs	Dynamic
0.017	1793.0 ms	5	0	0:08:00	139.8 MBytes	1.9.4	View Logs	Dynamic
0.017	2775.0 ms	5	0	0:11:52	144.5 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	5	0	0:09:09	143.2 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	6	0	0:11:10	125.0 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	8	0	0:12:29	140.2 MBytes	1.9.4	View Logs	Dynamic
0.033	30643.0 ms	1	0	0:01:12	113.1 MBytes	1.9.4	View Logs	Dynamic
0.050	38595.0 ms	1	0	0:01:01	134.4 MBytes	1.9.4	View Logs	Dynamic
0.033	0.0 ms	1	0	0:00:33	103.8 MBytes	1.9.4	View Logs	Dynamic
0.017	627.0 ms	3	0	0:06:32	130.4 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	6	0	0:12:19	134.8 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	2	0	0:04:59	131.8 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	1	0	0:10:05	135.0 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	1	0	0:10:15	111.9 MBytes	1.9.4	View Logs	Dynamic
0.017	674.0 ms	8	0	0:11:06	139.4 MBytes	1.9.4	View Logs	Dynamic
0.033	16701.0 ms	2	0	0:01:32	134.3 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	6	0	0:10:50	140.7 MBytes	1.9.4	View Logs	Dynamic
0.017	1154.0 ms	3	0	0:03:57	127.2 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	2	0	0:04:14	135.4 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	4	0	0:10:17	126.5 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	6	0	0:10:58	138.4 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	2	0	0:05:49	125.7 MBytes	1.9.4	View Logs	Dynamic
0.000	0.0 ms	1	0	0:06:58	113.6 MBytes	1.9.4	View Logs	Dynamic
0.033	2398.5 ms	5	0	0:11:20	140.4 MBytes	1.9.4	View Logs	Dynamic

Resource			Usage	Billable	Price	Cost
Frontend Instance Hours			485.65 Instance Hours	457.65	$0.05/ Hour	$22.89

On Friday, May 9, 2014 6:40:19 AM UTC+3, blackdew wrote:

[cloudpre]

We have two applications on AppEngine. We use F4. You can imagine the costs when they have gone into 450 instances. 

Any idea on how we are going to be compensated for the down time (app out of quota - $100/day) and the lost business?

On Friday, May 9, 2014 9:10:19 AM UTC+5:30, blackdew wrote:

[Mael Caldas]

The same for our app. From 2 hours ago, the same app version raised from 5 to 100 instances, and almost each request is firing a new instance.

<td style="padding: 0.35em 1em 0.25em 0.35em; border-top-width: 1px; border-bottom-width: 1px; border-top-style: solid; border-bottom-style: solid; border-top-color: rgb(197, 215, 239); border-bottom-color:

...

[Krishna]

Same here!

[Sérgio Lopes]

Same problem here. Too many instances without any spike on number of requests. I'm already overquota and offline.

Anyone here knows anything? Any solution? Any Google response?

[Bogdan Nourescu]

There was an issue reported here:

https://code.google.com/p/googleappengine/issues/detail?id=10893

Go star it to receive notifications.

A temporary fix: go to application settings and disable and enable the application. It worked for me

[PR]

Disabling and re-enabling the application seems to work....this fixed the issue for me now...

[Sérgio Lopes]

Fantastic. Disable/enable application worked here too. Thanks.

Hope Google cancel the charges...

[Stefano Ciccarelli]

Disable and after a minute re-enable the application and the problem seems to solve.

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.

--
Nel mondo esistono 10 categorie di persone, quelle che capiscono il binario e quelle che non lo capiscono.

[Yair Eshel]

I seem to have the same problem. I just went over quota, and had add to my daily budget limit (which was over 2 times my regular limit). Is everything allright there?

[husayt]

Over 30 posts here, this issue has been on for hours and no single response from GAE team.

At least acknowledge the problem.

There are some great things about GAE, but definetely customer/developer relations is not one of them.

[Keith Mukai]

Same problem here. Crazy instance numbers since about 4hrs ago. Normally 3-5 instances and then it grew to 60+. Dead in the water now at my billing limit.

[Diego Duclos]

The following was posted on the forums: https://groups.google.com/forum/#!topic/google-appengine-downtime-notify/VmH7s-Eeyso

On Fri, May 9, 2014 at 5:14 PM, Keith Mukai <keith...@essaytagger.com> wrote:

Same problem here. Crazy instance numbers since about 4hrs ago. Normally 3-5 instances and then it grew to 60+. Dead in the water now at my billing limit.

--

[Breno Marques]

I have the same problem, the budget was adjusted twice but is soon exceeded

[Kyle Baley]

This affected us and it's gone back to normal mostly. But for the last two days, our front-end instance usage is still roughly twice what it normally is.

[College]

Is anyone seeing this happening again today?  Our instances are again out of control.  This pattern started eight days ago.