Stop tearing down my instances!

[Cesium]

C'mon,

Let my instances live longer than 3 minutes under no load, would ya'?

Sheeshe,

David

[Barry Hunter]

If you where serious about this, you could actully provide some
details. How do you expect anyone (even google) to be able to take any
action based on what you posted?


At the moment, your post is nothing more than unhelpful rant.


Here endeth my rant!

[Cesium]

Oh, now I get it.

The Application Settings only apply to the Default Version of your app.

Any testing you perform on a specific, developmental deployment like '1-6.myapp.appspot.com' will not be subject

to these settings.

Thus, any performance testing will not reflect said performance if and when it becomes the default version.

Instances of my test deployment are destroyed after about 3 seconds. This was NOT the case for the past few months.

What gives?

David

[Cesium]

Dearest Barry,

Thank you for your helpful comments.

Here's a couple of lines from the log of my test deployment:

(Note x's indicate redacted text.)

2012-05-31 09:06:45.940 /sdtp?t=STATUS_UPLOAD_D8EA 200 5028ms 0kb

xx.xxx.xxx.xxx - - [31/May/2012:08:06:45 -0700] "POST /sdtp?t=STATUS_UPLOAD_D8EA HTTP/1.1" 200 64 - - "1-6.xxx.appspot.com" ms=5028 cpu_ms=3302 api_cpu_ms=128 cpm_usd=0.091769 loading_request=1 instance=00c61b117c3d715046b4d903320e4662386537a5

I 2012-05-31 09:06:45.940

This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application.

2012-05-31 09:06:39.703 /sdtp?t=RTC_SYNC_D8EA 200 42ms 0kb

xx.xxx.xxx.xxx - - [31/May/2012:08:06:39 -0700] "POST /sdtp?t=RTC_SYNC_D8EA HTTP/1.1" 200 72 - - "1-6.xxx.appspot.com" ms=43 cpu_ms=163 api_cpu_ms=0 cpm_usd=0.004579 instance=00c61b117c96f9ab10868c1050194369b27cc1

My uninformed interpretation of these two entries is as follows:

At  09:06:39.703, version 1-6 of my application serviced a request.

A little over 6 seconds later at 09:06:45.940 my application received another request.

Rather than service this request with instance 00c61b117c96f9ab10868c1050194369b27cc1 the

scheduler decided to create a new instance to server it, namely 00c61b117c3d715046b4d903320e4662386537a5.

rant on,

David

[Cesium]

Sigh,

I don't have the energy to be a smart a$$.

A request arrives:

2012-05-31 09:22:25.414 /sdtp?t=APP_DATA_UPLOAD_66AF_via_D8EA 200 738ms 0kb

xx.xxx.xxx.xxx - - [31/May/2012:08:22:25 -0700] "POST /sdtp?t=APP_DATA_UPLOAD_66AF_via_D8EA HTTP/1.1" 200 64 - - "1-6.xxx.appspot.com" ms=739 cpu_ms=1212 api_cpu_ms=746 cpm_usd=0.033736 instance=00c61b117cb82be7b7138b1a1e2ee8b6881669a2

12 seconds later, we get a new instance:

2012-05-31 09:22:37.356 /sdtp?t=APP_DATA_UPLOAD_66AF_via_D8EA 200 7397ms 0kb

xx.xxx.xxx.xxx - - [31/May/2012:08:22:37 -0700] "POST /sdtp?t=APP_DATA_UPLOAD_66AF_via_D8EA HTTP/1.1" 200 64 - - "1-6.xxx.appspot.com" ms=7397 cpu_ms=4330 api_cpu_ms=783 cpm_usd=0.120335 loading_request=1 pending_ms=1485 instance=00c61b117cf948811583949ac99c72597387f8

I 2012-05-31 09:22:37.356

This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application.

This is killing my testing. What changed?

David

[Cesium]

I take it back. Even the default version of an entirely different application

is now showing rapid decay of instances.

(xxx indicates redacted text).

A request comes in, we get a new instance:

2012-05-31 09:52:10.028 /xxx/sdtp?t=ACCESS_POINT_STATUS_UPLOAD 200 7526ms 0kb

xxx.xxx.xxx.xxx - - [31/May/2012:08:52:10 -0700] "POST /xxx/sdtp?t=ACCESS_POINT_STATUS_UPLOAD HTTP/1.1" 200 64 - - "xxx.appspot.com" ms=7526 cpu_ms=3278 api_cpu_ms=128 cpm_usd=0.091121 loading_request=1 instance=00c61b117c81276013340436a0b0e1635cc8de

I 2012-05-31 09:52:10.026

This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application.

7 seconds later, another request, another new instance.

2012-05-31 09:52:17.602 /xxx/sdtp?t=ACCESS_POINT_STATUS_UPLOAD 200 5584ms 0kb

xxx.xxx.xxx.xxx - - [31/May/2012:08:52:17 -0700] "POST /xxx/sdtp?t=ACCESS_POINT_STATUS_UPLOAD HTTP/1.1" 200 64 - - "xxx.appspot.com" ms=5585 cpu_ms=2762 api_cpu_ms=8 cpm_usd=0.076769 loading_request=1 instance=00c61b117c416b1c599f726fee1c70a4bca1d1

I 2012-05-31 09:52:17.601

This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application.

This is really harshing my buzz.

David

[Jeff Schnitzer]

Is it possible you're getting the "Your instance has been terminated"
message? I found that overrunning the heap causes instances to be
killed with this ambiguous message.

It would be handy to have a log message indicating why an instance is
being shut down.

As far as debugging this particular issue (assuming you aren't getting
terminated msgs), are the Application Settings sliders set to
Auto/Auto? Start from there and make incremental changes.

Jeff

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/mMKQ-88fR-IJ.
>
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

[Cesium]

Hey Jeff,

That message would be in the logs right? Don't see it.

Yep, been fiddling with the sliders too. No joy from that.

It's the classic:

Customer: "What did you change?"

Me: "I haven't touched the code in 75 days!"

Customer "You must have done something!"

Me: "It's not my fault!"

Customer: "Your product sucks!"

Me: *sigh"

Check out my Milliseconds per Request plot:

 

This is depressing. I'm going to go pig out on Dim Sum.

David

[Jeff Schnitzer]

I would definitely file a Production Issue Ticket, if you haven't already.

Jeff

David

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/JWBcvo3HrhUJ.

[Cesium]

Is that one of those things where you highlight parts of the screen.

Then it says, "Thanks for your feedback, we probably won't get back to you." ?

Yup, did that.

David

[Cesium]

OK,

I've got 1 resident instance, and 2 dynamic instances.

They are seeing and average of 0.006 Queries per Second (very light load).

There are warm-up requests every 5 minutes or so. But, I am still seeing client transactions that cause an instance to be started.

In the event that an existing instance does indeed serve the request, the time it takes is 10 times higher than it was a couple days

ago.

Here's the milliseconds per request chart:

 

What the frack is happening today? At what point do I tell my boss that we're hosed?

David

[Jeff Schnitzer]

Is it currently "all settings auto" and billing enabled?

I have a production environment and a staging environment (same code).  I find that without billing enabled, the staging environment sees lots of startup requests.  We've actually given up trying to run the staging environment without billing enabled.

Jeff

David

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/g45a4EDAagsJ.

[Cesium]

Smooth sailing for a month and a half. Billing enabled.

Today, its going to hell in a tortilla for my customer facing apps, and my development apps. All have billing enabled.

Is the appengine group at a team building retreat in Nappa? WTF?
David

[Cesium]

A few hours ago, one app started to settle down:

milliseconds per request:

 

But the app that makes the customers call my f%*#ing cell phone shows:

milliseconds per request:

 

It appears both apps showed improvement about 5 hours ago. Yet this one still suffers.

So, Google App Engine Team, whatcha' think?

Did you put fresh gerbils in the UPS?

Antivirals in the bio-neural gel packs?

Could you do it for all my apps?

Don't make me put on my Dire Straits album this early in the day!

David

[stevep]

Your billing experience seems similar to mine (overly long thread link
below). I have not yet tried Joshua's setting for very low transaction/
sec apps, but there is a suggestion here that Auto/Auto slider
settings may be sub-optimtal for very low QPS apps. What I am really
starting to wonder about is the infrastructure costs this Scheduler
behavior might impose. Constantly starting/stopping instances, and --
most certainly -- starting a buffer instance when only one would ever
be needed (based on QPS history) -- well all those needless instance
startup cpu cycles aren't free. Maybe it is less cost, less resource
intensive than having an idle instance stay alive, I am sure we will
never know. -stevep

Link:
http://groups.google.com/group/google-appengine/browse_thread/thread/9c63e9b38c038a35/95c3c4106bd0b340?lnk=gst&q=stevep#95c3c4106bd0b340

[Iein Valdez]

Cesium - please file a production ticket here with all the details and we'll see what's going on. Thanks.

http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue

[Cesium]

Hey Iein,

Thank you for the reply!

I sent you an email with some details.

Cheers,

David

[Warwick Slade]

Well after reading more posts on this subject I have decided it is time to move away from GAE.

I would rather pay $US15 a month for a dedicated tomact that will be always up than have this sort of response times.

I would warn any new adopters to think twice before jumping onto GAE until these issues are resolved.

Unless of course you have a request per second.

On Monday, 4 June 2012 16:18:10 UTC+10, Warwick Slade wrote:

I am very interested in your thread.

My app is unloaded very frequently.

My GWT app issues two ajax calls when it starts up and I see two messages in the log that the instance has been restarted.

/event/url1 15:10:05 servlet Context initialised
/event/url2 15:10:16 servlet Context initialised
/event/url1 15:10:24 This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time.
/event/url2 15:10:36 This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time.

So I what I need to decide is do I
1. Move my rpc ajax calls onto the service and use the memcache to try and speed up start time (Will this give me the 200ms response time that I see when I quickly refresh my browser?)
2. Forget about using gae and go back to my hosted tomcat solution.