Prior to today, it seemed they were not getting spun up as readily
either. Today I'm seeing more instances online again (which is good,
app is much more responsive.
Robert
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/INY70Yliu5YJ.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
Here are the current scheduling rules: (> reads as has priority for
handling the incoming request)
1/ Idle Always On instance > Spawning a new Dynamic instance
2/ Spawning a new Dynamic instance > Busy Always On instance
3/ Idle Dynamic instance > Busy Always On instance
4/ Idle Dynamic instance > Idle Always On instance
I will give you an example to illustrate the behavior you all noticed,
that is Dynamic instance handling request while Always On is idle.
(Always On instance started)
- Incoming request
- Always On instance handle the request
- another Incoming request
(Always On instance busy)
- A new Dynamic instance is spawned
(Dynamic instance idle, Always on instance busy)
- Dynamic instance handle the request
- another Incoming request
(Dynamic instance idle, Always on instance idle)
- Dynamic instance handle the request
- No request for more than idle-dynamic-instance-timeout
- Dynamic instance shut down
- another Incoming request
(Always On instance idle)
- Always On instance handle the request
Hope it makes thing clearer.
As part of the new billing model you will have a scheduler knob called
'max-idle-instances' that you can use if extra idling dynamic
instances are undesired.
The good news is that we are open to suggestion, if you think this
behavior is the wrong default, feel free to comment on that thread and
I will follow up your suggestion to the Engineering team.
> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>
>
--
Johan Euphrosine (proppy)
Developer Programs Engineer
Google Developer Relations
Thanks for the followup,
I think you are experiencing a combinaison fo the two following rules
I was pointing to in my previous email:
(> reads as has priority for handling the incoming request)
2/ Spawning a new Dynamic instance > Busy Always On instance
4/ Idle Dynamic instance > Idle Always On instance
Applied to your example it could means that:
Resident Instance 1: Requests: 49 Age: 1Hr
Resident Instance 2: Requests: 6 Age: 1Hr
Resident Instance 3: Requests: 2 Age: 1Hr
Dynamic Instance 1: Requests: 7 Age: 2min
Dynamic Instance 2: Requests: 291 Age: 1Hr
Dynamic Instance 3: Requests: 322 Age: 1Hr
- 1 Hours ago while all your Always On instance were busy and you had
a burst of incoming requests and the scheduler spawned new Dynamic
instances as per rule 2/ highlighted above.
- After the burst and back to normal traffic the new Dynamic Instances
were handing incoming requests in priority as per rule 4/ highlighted
above.
- 2 Minutes ago all your instances Always On + Dynamic were busy again
and the scheduler spawned a new Dynamic instance that handle 7
incoming requests.
Hope that make more sense for you and Francois, but as I said earlier
we are open to suggestion and I will make sure someone working on the
scheduler team monitor this thread for your input.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Robert
Thanks for showing interest in GAE internals, I'd be happy to answer
those questions directly if I can, or forward them to someone who can
answer them better.
> 1. "1 Hours ago while all your Always On instance were busy and you
> had a burst of incoming requests"
> While this may be true when my Always On instances were "busy" running
> some stuff but what about when 2 Always On instances show only "1"
> request served which is the Warmup request itself. Does this mean
> Warmup requests are considered as traffic? If that is the case then
> Always On instances seem rather useless since they will never ever get
> called in this scenario.
On the admin console capture you included in your previous mail, I
didn't see Always On instances showing only "1" request served but
rather:
Resident Instance 1: Requests: 49 Age: 1Hr
Resident Instance 2: Requests: 6 Age: 1Hr
Resident Instance 3: Requests: 2 Age: 1Hr
Let me know if I missed something.
> 2. As Tom mentioned, what qualifies "busy". When threadsafe option was
> implemented in GAE these 3 Always On instances were able to do most of
> the heavy lifting with occasional spinning of dynamic instances.
> Nothing has changed on our side that should alter this behavior. With
> all these changes happening within GAE I am trying to figure out what
> changed and what we can do to contain this burst of traffic within 3
> (or more ) Always On instances with less frequent spinning of Dynamic
> instances.
There are two scheduler knobs that could help you to affect the way
Dynamic instance are spawned.
"Minimum Pending Latency" and "Max Idle Instances" as described here:
http://code.google.com/appengine/docs/adminconsole/performancesettings.html
> 3. "- 2 Minutes ago all your instances Always On + Dynamic were busy
> again and the scheduler spawned a new Dynamic instance that handle 7
> incoming requests. "
> Again what constitutes "busy" as I do not see any request being served
> by Always On instances 2 and 3 in last 1 hour. Note that number of
> requests served by Always On 2/3 are unchanged since they were
> created ...
> Here's my reading in this scenario:
> a. It kills Dynamic Instance 1 within 2 minutes of serving a request
> b. When traffic comes in it looks only for Dynamic Instances if they
> are busy and completely ignores Always On instances at this point
> c. It recreates Dynamic Instance 1
>
> In other words, what rule is applied in this case?
Sorry, those were mostly specification of mine, I didn't know that the
request served by Always On 2/3 were unchanged according to the
information you provided.
I can investigate deeper into the specific behaviour of your
application, if you open a Production Issue with your application id.
> Also I fail to understand rule 4 as both Rob and Luca mentioned. That
> completely undermines having Always On instances under threadsafe
> mode.
>
> 4. I like Rob's suggestion of better load balancing techniques but
> again with a caveat that an instance needs to be able to serve
> multiple threads before reaching a set capacity (80% or so)
>
> 5. Luca's suggestion also makes sense but again with the same
> caveat ... it should be able to process multiple threads before
> queuing
Thanks a lot for your feedback, I will make sure to forward those
suggestions to the engineering team.
>
> 6. I looked at the new sliders in the Admin console and with those the
> situation is even worse. I set the Max Idle Instances to 3 (that's the
> minimum I could choose) and Min Pending Latency to 15 secs ... Guess
> what our CPU usage has gone up to 15 in 12 hrs because of constant
> creation and killing of 3 dynamic instances. Bare minimum traffic and
> few light weight crons.
> But the good side is now I see requests coming in on the 3 Always On
> instances. Is that enough load they are serving ... I don't know yet
> but something to observe.
Maybe you can open a feature request for having a smaller min for 'Max
Idle Instance' when Always On is activated or having Always On
instances count in Max Idle Instance.
> Two things I suggest would be really helpful for us:
> A. The overall key here is to know the thread handling capacity of an
> instance. Better yet if it can be configured similar to Backends but
> dynamic in nature (and of course Backends pricing is outrageous ...
> but that's another topic)
Are you looking for <max-concurrent-requests> support for Servlet ? If
so I would recommend to open a Feature request.
> B. Able to add more Always On instances but again with a dependency
> explained in point A.
Again, opening a feature request make sense to track this separately.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
I will ask to the engineer team and get back to you in thread.
Hope that helps.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
I confirm the current rules is Idle Dynamic instance > Always On instance
While I agree your proposition make sense in some cases, it sounds
also good to take advantage of the Dynamic instances that have just be
spawned especially if the startup cost is high.
Like Gregd highlighted in his pricing faq, Always On will became a
settings to control the number of idle instances you would like to
have running.
Q: How will Always On work under the new model?
A: When App Engine leaves preview all Paid Apps and Apps in Premier
Accounts will be able to set the number of idle instances they would
like to have running. Always On was designed to allow an app to
always have idle instances running to save on instance start-up
latency. For many Apps a single idle instance should be enough
(especially when using concurrent requests). This means that for many
customers, setting an App to be paid will mean a $9/month minimum
spend, you can then use the 24 free IH/day to keep an instance running
all the time by setting Min Idle Instances to be 1.
> I also think an auxiliary problem is with rule 2/
> If you spawn a new instance just because the others are all busy, you may
> tend to have too many instances.
> The fact of spawning new instances should be configurable, depending on how
> many requests are already queued for the instances you have already active
> -- 0, 1, more, etc.
> More in general, if you allowed users to specify how much it costs to them
> to delay serving a request, it would not be difficult to synthesize for each
> app an optimal decision policy to decide whether to switch another instance
> on or off. This can be done using tools from dynamic optimization / control
> theory. I would be glad to help if the people there need guidance on this
> (I used to be in the research group there till a month ago).
It sounds to me that the scheduler knobs are there precisely to address this:
http://code.google.com/appengine/docs/adminconsole/performancesettings.html
Let me know if I overlooked it.
> Luca
> On Thursday, July 21, 2011 5:56:42 PM UTC-7, Johan Euphrosine (Google)
> wrote:
>>
>> After speaking with Engs, I think I can explain what is going on:
>>
>> Here are the current scheduling rules: (> reads as has priority for
>> handling the incoming request)
>>
>> 1/ Idle Always On instance > Spawning a new Dynamic instance
>> 2/ Spawning a new Dynamic instance > Busy Always On instance
>> 3/ Idle Dynamic instance > Busy Always On instance
>> 4/ Idle Dynamic instance > Idle Always On instance
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/X8gH6jjIK0MJ.
Q: How will Always On work under the new model?
A: When App Engine leaves preview all Paid Apps and Apps in Premier
Accounts will be able to set the number of idle instances they would
like to have running. Always On was designed to allow an app to
always have idle instances running to save on instance start-up
latency. For many Apps a single idle instance should be enough
(especially when using concurrent requests). This means that for many
customers, setting an App to be paid will mean a $9/month minimum
spend, you can then use the 24 free IH/day to keep an instance running
all the time by setting Min Idle Instances to be 1.
Let me know if you need more information.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Feel free to open a Production issue with you appid if you want me to
track this specifically.
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/Qczx6foLplUJ.
I think the help text of the 'Idle Instances' settings is pretty
self-explanatory:
"""
You will not be charged for instances over the specified maximum.
"""
I can investigate on why these instances are created even thought you
setup a high min pending latency, and a a Max idle instances
corresponding to the number of Always On instance.
Feel free to open a Production issue, with your application id if this
is affection your operation.
Thanks in advance.
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/TVr3Ko1-baIJ.
Thanks for investigating those behaviors, feel free to continue to
share your findings with the community.
> 3. In our regular app we are running dummy cron jobs to keep 3 DI up
> at all times. Again, amazingly with many concurrent users they are
> able to handle ALL the load. On the contrary RI are almost untouched.
> Here's the latest statistics:
>
> QPS* Latency* Requests Errors Age Memory
> Availability
> 0.000 0.0 ms 25 0 12:43:22 82.3 MBytes Resident
> Icon Resident
> 0.000 0.0 ms 10 0 12:44:45 80.6 MBytes Resident
> Icon Resident
> 0.000 0.0 ms 5 0 12:44:46 81.9 MBytes Resident
> Icon Resident
> 0.350 19.1 ms 11762 0 9:45:11 118.5 MBytes Dynamic Icon
> Dynamic
> 0.317 17.5 ms 3890 0 3:11:04 97.7 MBytes Dynamic Icon
> Dynamic
> 0.333 67.5 ms 15604 0 12:43:41 104.8 MBytes Dynamic Icon
> Dynamic
>
> But this is just a sample set of users. I don't know yet if we can
> scale on this principle but it seems to work on light load. But this
> is definitely short term fix since this is going to be expensive under
> new billing.
I believe those stats illustrate the rule we discussed before:
Idle DI > Idle RI.
Keep in mind that under the new pricing model Min Idle Instances
should superseed Always On instances, as described by gregd in the
billing faq.
So hopefully you should not have idling DI instance wandering around,
and you will be able to control this with Max Idle Instances anyway.
I think your suggestions would be a great addition to the
admin-console, and let user have more visibility and control over
their instances and thus over billing.
You should definitely fill separate feature requests on the public
issue tracker for them.
In the meantime, I will make sure the engineering team is aware of
your suggestion.
> Let me know if I missed something that is already there or I am going
> off tangent here.
You're completely on the topic of this discussion :)
This is not a typo:
4/ Idle Always On instance > Idle Dynamic instance
Under the new pricing model, as gregd highlighted in the pricing faq
Always On will be superseeded with Min Idle Instances.
So we shouldn't see that "weird" behaviour of Always On instance
getting idle, but rather a nice distribution of Dynamic Instance
matching the scheduler knobs you set in admin-console.
Thanks a lot for those suggestions, I will make sure to forward them
to the engineering team.
In the meantime feel free to open feature request for them if you want
to the community to be able to track their progress.
A few suggestions coming out of this thread that I forwarded the
scheduler engineering team:
1/ Show the current CPU% usage of an instance.
2/ Show the current number of threads of an instance.
3/ Allow to override the 30 second deadline for the warm up requests
(for loading frameworks).
4/ New idle_instance_timeout knob: how long an idle instance can live
before being reclaimed.
5/ Allow to configure the capacity for Dynamic Instance (in a similar
way to Backends).
6/ Add rules for changing Max Idle Instances / Instance capacity
during different times of a day and week.
7/ Multiple high availability preset regarding instance spawning: (
safe + unpredicatable cost ) versus ( unsafe + predictable cost ).
8/ Allow user to specify a budget related to request serving delay so
the scheduler can synthesize a policy about instance spawning.
9/ Alternative load balancing strategy: load 1 instance > 50% before
using the second one and so on (instead of round-robin).
Hope that helps.
The only knobs Google provide so far are:
- maximum number of idle instances
- minimum pending latency
See http://code.google.com/appengine/docs/adminconsole/performancesettings.html
for more details.
- min number of idle instances will come later to supplant Always on.
And all knobs should have automatic default values that make App
Engine handle those "automagically in the background".