No warmup requests are being issued by the Java instance scheduler

[Carl Schroeder]

Has this feature been disabled? Have the requirements for it changed?

The "Configured Services" section of my Application Settings says that "Warmup Requests" are enabled.

I can find no trace of any warmup requests in my logs.

Also, the scheduler is still acting crazy. It is starting up new java instances (with user facing requests) when there are dynamic instances alive and well and idle. 20 second response times to REST calls that normally return in 50ms is simply unacceptable performance. Asking my users to wait 40 seconds while a page that contains multiple REST calls loads is simply not an option.

I am back to deciding whether to port the java to python and stay on GAE, or trying out dynamoDB on AWS.

[Takashi Matsuo]

Hi Carl,

On Mon, Oct 22, 2012 at 8:26 AM, Carl Schroeder <schroede...@gmail.com> wrote:

Has this feature been disabled? Have the requirements for it changed?

The "Configured Services" section of my Application Settings says that "Warmup Requests" are enabled.

I can find no trace of any warmup requests in my logs.

Currently, the warmup requests are issued only when you have some min idle instances settings instead of 'automatic'.

 

Also, the scheduler is still acting crazy. It is starting up new java instances (with user facing requests) when there are dynamic instances alive and well and idle. 20 second response times to REST calls that normally return in 50ms is simply unacceptable performance. Asking my users to wait 40 seconds while a page that contains multiple REST calls loads is simply not an option.

To avoid this, currently the options are

* to have sufficient number of min idle instances

or

* to make your loading requests faster

-- Takashi

 

I am back to deciding whether to port the java to python and stay on GAE, or trying out dynamoDB on AWS.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/U2PyWapvPqkJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Takashi Matsuo | Developers Advocate | tma...@google.com

[Jeff Schnitzer]

Obligatory comment: Yet another user who finds scheduler behavior to
be "broken out of the box".

Carl: The relevant issue to star:
http://code.google.com/p/googleappengine/issues/detail?id=7865

Jeff

[Guenlay]

可以用 google_hk 

2012/10/22 Jeff Schnitzer <je...@infohazard.org>

[Carl Schroeder]

The use of "resident" instances with a low site traffic profile resulted in a different version of the pathological behavior. With periods of low usage, the scheduler decommissioned dynamic instances. When traffic picked up again, it did not use the "resident" instances. Instead, it spun up 3 or 4 dynamics causing the page load to stall out. At least, that was the case 1 month ago when I ditched the entire concept of "min idle instances" (since they appeared to always be idle and never serve traffic). I use wget to issue a single request once every 60 seconds in an attempt to have at least 1 dynamic instance always available.

I will test again to see if "idle instances" actually add any value to a site with my traffic profile. Before, they doubled or tripled the cost and did not improve the user experience at all.

What is the reasoning behind not sending warmup requests?

I have already starred the relevant scheduler enhancement issues.

[Carl Schroeder]

I just tested with my Application Settings configured to have 1 Resident instance.

With 1 resident, and 1 idle instance. I hit a page reload. 7 requests were served by the instances according to the Instances pane in the App Engine Console. The requests were handled in the following manner:

Zero went to the Resident instance. 

3 new instances were spun up. 2 with warmup requests, 1 with the first user request given to a cold start.

4 then went to the existing idle dynamic instance

1 of the new instances handled 3 requests, the other 2 only handled warmups.

The VERY first request (the basic HTML of the page) went to a cold instance. This happened despite the fact that there was an idle Resident instance available AND an idle Dynamic instance available. The user experience is staring at the browser for 20 seconds before anything happens. That is unacceptable.

Even with 4 dynamic instances, the scheduler is still spinning up new ones with user facing requests. This is bizarre, pathological, diabolical, nonsensical behavior. I am running out of adjectives here.

[Kristopher Giesing]

On Monday, October 22, 2012 5:51:38 AM UTC-7, Takashi Matsuo (Google) wrote:

Hi Carl,

On Mon, Oct 22, 2012 at 8:26 AM, Carl Schroeder <schroede...@gmail.com> wrote:

Has this feature been disabled? Have the requirements for it changed?

The "Configured Services" section of my Application Settings says that "Warmup Requests" are enabled.

I can find no trace of any warmup requests in my logs.

Currently, the warmup requests are issued only when you have some min idle instances settings instead of 'automatic'.

This isn't mentioned in the documentation anywhere that I can see.  Did I miss it?  It seems like an incredibly important point to leave out.

 

Also, the scheduler is still acting crazy. It is starting up new java instances (with user facing requests) when there are dynamic instances alive and well and idle. 20 second response times to REST calls that normally return in 50ms is simply unacceptable performance. Asking my users to wait 40 seconds while a page that contains multiple REST calls loads is simply not an option.

To avoid this, currently the options are

* to have sufficient number of min idle instances

or

* to make your loading requests faster

The third option, of course, is to use something other than GAE.  Given that solution #1 pushes GAE costs over paying for an always-on host from another vendor, and #2 doesn't get you to a reasonable SLA, #3 honestly looks like the best choice from where I'm sitting.

I'm trying very hard not to get angry about this.  I actually have a good friend who is a developer on App Engine and I want very much to make this work, but it just seems like the scheduler is fundamentally broken, and Google is refusing to recognize it.  Am I missing something?  For those of you who have apps deployed on GAE, why isn't this behavior killing you?

- Kris

 

[Kristopher Giesing]

Can you post the relevant logs?  Each request is stamped with the ID of the instance that served it.

When I looked into my own logs I found that the request I thought was cold-served was actually going to a recently warmed instance, but that the warmup request didn't fully initialize everything so the first request after warmup was still too long.  But it sounds like you have more and better data here to prove something is actually broken.

- Kris

[Carl Schroeder]

It is actually pretty easy to prove when a request is going to a cold instance vs a warmed up one. Simply output a log line from a place that is only called during initialization of an instance. Make sure that the warmup handler calls this part of the code, and that the code is only called once per instance. I put mine in the abstraction layer that sits between appengine, and my application code. 

My warmup handler shows this log message. So does any user facing request that causes an instance of my app to be initialized.

Also, I did a search for "This request caused a new process" in the log files. Just today, 27 instance starts. My app can serve its user base off a single instance with excellent response times. (If only I had a way to tell the scheduler to piss off)

For the past hour, I had 3 warmups handled and 4 user facing requests routed to cold instances. I verified that the instances spun up by user facing requests had different IDs than the ones spun up by warmup reqeusts. I have an AWS instance that hits my site once every 60s to attempt to insure that there is always a dynamic instance loaded. I have 1 resident instance configured to "enable" warmups.

[Kristopher Giesing]

I'm not asking you to prove it to yourself or me, but to Google :)

- Kris

[Carl Schroeder]

They already have my app id and can look at the logs AFAIK. The issue isn't whether it is or is not happening, the issue is convincing Google that it should not be happening.  I get the vibe that they think it is OK for our users to wait 20 seconds for a page load that should take 800ms.

Most else about app engine has been either good or great. All I need is a button that I can press or checkbox I can check that will guarantee only warmup requests go to cold instances.

[Michael Hermus]

Many of us are equally frustrated with this problem. The relevant issue has already been moved to 'Accepted' status by Google, which I believe means that they have decided to address it. Of course, that doesn't give us an ETA, but I am hopeful that it will not be too long.