--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Let me summarize the case you describe, to make sure I get it right.
First, the app has a period of five minutes, wherein there are four
instances that are active the entire five minutes. Then, the app gets
no more traffic all day. Also, you have max-idle-instances set to 1.
It helps here to refer to the billing formula to understand this.
Below, total-instances refers to the blue line on the graph, and is
computed according to the +15-minutes-since-last-request formula,
whereas active-instances refers to the orange line on the graph, and
is based solely on active requests (and no 15-minute logic).
billable-instances-rate = min(active-instances-rate +
max-idle-instances, total-instances-rate)
So, for time period [0:5], this expression evaluates to min(4+1,4)=4,
for the time period [5:20] the expression evaluates to min(0+1,4)=1,
and then for the time period [20:inf] the expression is min(0+1,0)=0.
As such, what will actually happen is a different outcome altogether
from the ones listed above:
"Phew": Charged for 5 minutes of 4 instances, followed by 15 minutes
of 1 instance, followed by nothing, for a total of 35
instance-minutes.
On Tue, Sep 6, 2011 at 7:14 AM, Simon Knott <knott...@gmail.com> wrote:
> Thanks for the description Jon, that clears up a lot of my misunderstanding!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/Q5cRpciJ6V8J.
I assume you are talking about 'drag-2-share' here? Even if the
instance is still running, it won't matter if it has not received a
request in the last 15 minutes. This is why your Instances graph on
your dashboard is usually 0, occasionally 1 for 15 minutes, and then 0
again. You are not billed for instances that have not done anything in
the last 15 minutes.
It would be reasonable for us to physically kill off such instances
which happen to remain running (because they are on a fortuitous
machine) past 15 minutes of idleness, but please note that if we did
this, it would no impact on your bill.
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/IY3SH_AQ2PMJ.
} there is a startup fee of 15 minutes for each instance.
This only affects the computation of the total-instances variable
in the formula, it does not affect the computation of
active-instances.
} - If I understand your calculation correctly, by setting max-idle-
} instances to 1 I can basically pay the startup fee for one instance
} only, right?
Pretty much.
} Is it guaranteed that instances die after being idle for 15 minutes?
They will usually die before 15 minutes [see other response], but
it is possible they may live past 15 minutes of idleness. But
past 15 minutes of idleness, they are completely ignored by the
billing system.
} - What's the sense of being able to set max-idle-instances then, if
} they die anyway?
Death is based on units of time, but max-idle-instances is
expressed in units of instances. They achieve different things.
} Nothing would change by setting the value higher
} (except for the hours billed).
The max-idle-instances setting has two effects. The first is a
hint to the scheduler to kill off excess idle instances [it does
this now, but approximately], and the second is how it is used in
the billing formula.
So, if you had an app with hundreds of instances, and set
max-idle-instances to 1, it would actually kill off hundreds of
idle instances, leaving you just with enough instances to cover
your active load. This can be problematic as your app receives
more traffic, because there is no spare capacity, so you must do
a lot of loading requests, and you will likely serve errors for a
few minutes. Essentially, you would have no slack, so your
serving latencies and error rates would contain more fragility in
the face of changes in traffic patterns. So, it saves money but
if used to excess can really hurt performance. This is ultimately
why we've made it a sliding scale that the developer can choose,
because the only way to truly set this parameter correctly is to
know how relatively important are cost and reliability.
--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/lPvp44U3uLgJ.
If you have set max-idle-instances=1, then in that example, what would
happen is:
(a) Instance is active 1 second every 5 minutes.
(b) That is 288 active seconds for the day.
(c) active-instances-rate is = 288/60*60*24 = 0.0033 for whole day
(d) total-instances-rate = 1 for whole day
(e) billable-instances-rate = min(active-instances-rate +
max-idle-instances, total-instances-rate) = min(0.0033 + 1, 1) = 1
So the total billable instances for the day would be 24, which happens
to also be the number of free instance hours given per day. The
resulting bill would be $0.
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/lPvp44U3uLgJ.
It is not possible to set max-idle-instances to 0. If you had set
max-idle-instances to 1, then your example is basically identical to
Dani's.
In that example, if max-idle-instances=1, then the billable instance
hours would be 24. If max-idle-instances=automatic, then the billable
instance hours would be 24.
On Wed, Sep 7, 2011 at 2:07 AM, Tim <mee...@gmail.com> wrote:
>
> Jon,
> Thanks for that explanation, but if it's authoritative (not meaning to sound
> rude, but I hope .. I take it you're on the AppEngine team rather than a
> user explaining your understanding of how pricing applies) then it perhaps
> illustrates what many of us may be worried about, and maybe a more detailed
> explanation of the scheduler and 15-minute rule may put some minds at rest
> and reduce some of the hostility to the new billing scheme.
> It's the sort of thing that can be easily explained in person, but can be
> hard to get across on paper, so let me explain how I may have previously got
> the wrong impression and been more alarmed than you may consider seems
> appropriate.
Correct. Correct. Correct.
This characterization of the imprecise nature of the implementation of
max-idle-instances in the scheduler is exactly correct. But the
billing formula is always precise.
> But if scheduling (of queries to instances), and billing (of instance time)
> is closer to explanation #2 than #1, then I could probably live with the "15
> minute" rule, but I'd hope you could also see that if the scheduler works
> more like #1 than #2 then that would be why so many of us are worried about
> "15 minutes" and would like to see that interval dropped to more like "2
> minutes" !
Understood. And again, apologies. We mean well, we messed up, we're
working on it.
> An authoritative explanation of the intent (there's that word again Greg) of
> the scheduler / billing logic would be more than welcome and may reduce the
> gap between why you guys think the pricing is reasonable and many of us in
> here think it's not.
Agreed. The system is large and complex, its hard to understand and
hard to explain. Ultimately I am confident that everyone can find a
reasonable outcome but trying to work through these things at scale
(i.e. helping hundreds of thousands of developers to change something
simultaneously) is quite difficult, practically impossible. We're
working on it.
> Cheers
> --
> Tim
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/bDbsEmTyuFgJ.
Not really, no.
>
>
>> } Is it guaranteed that instances die after being idle for 15 minutes?
>>
>> They will usually die before 15 minutes [see other response], but
>> it is possible they may live past 15 minutes of idleness. But
>> past 15 minutes of idleness, they are completely ignored by the
>> billing system.
>
> Let's say I have 4 active instances for 5 minutes, then no traffic for
> the rest of the day, max-idle-instances set to 1.
> 1) 35 minutes will be billed regardless when the scheduler decides to
> kill idle instances, right?
Yes.
> 2) But 1 instance would be idle for at least 15 minutes, right?
Maybe.
>
>> The max-idle-instances setting has two effects. The first is a
>> hint to the scheduler to kill off excess idle instances [it does
>> this now, but approximately], and the second is how it is used in
>> the billing formula.
>
> Let's say I have 4 active instances for 5 minutes, then no traffic for
> the rest of the day, max-idle-instances set to 4.
> 1) 80 minutes will be billed regardless when the scheduler decides to
> kill idle instances, right?
Yes.
> 2) But 4 instances would be idle for at least 15 minutes, right?
Maybe.
>
>
> Take care and thanks for your help,
>
> Tammo
>
Understood. And again, apologies. We mean well, we messed up, we're
working on it.> An authoritative explanation of the intent (there's that word again Greg) of
> the scheduler / billing logic would be more than welcome and may reduce the
> gap between why you guys think the pricing is reasonable and many of us in
> here think it's not.Agreed. The system is large and complex, its hard to understand and
hard to explain. Ultimately I am confident that everyone can find a
reasonable outcome but trying to work through these things at scale
(i.e. helping hundreds of thousands of developers to change something
simultaneously) is quite difficult, practically impossible. We're
working on it.
I'm really glad to hear that!
> I know how hard it can be to explain such things without leaving room for
> misconceptions to form (had similar experiences designing a system and
> scheduler for a lazy ticking memoising computational grid for tens of
> thousands of machines), but maybe some worked examples of how you intend the
> billing to work for certain scenarios might help ?
> eg "app has busy bursts and then nothing", "app has busy bursts and then
> ticks along normally until next burst", "app has 'swells' (gradual ramp up
> and ramp down of traffic over an hour or two)", "asymmetric bursts (like a
> bell curve with skews)"
I think that is an excellent idea.
> I'm willing to accept that the scheduler is always going to making guesses,
> but things like the fact that you're basing billing of "Instance hours" on a
> notional level of service which should be less than the "actual" instance
> hours suggests to me that the intent is to let you tweak the parameters of
> the opportunistic apsects of the scheduler without changing the user
> experience of billing for every such tweak (say deciding to keep the true
> idle instances active for different periods in excess of 15 minutes
> depending on time of day to avoid repeated reloads, but such changes won't
> affect the bills), which sounds more in line with your PaaS aspirations.
Exactly. The other thing that happens is that this imprecision allows
us to paper-over internal infrastructure details. As an example, when
we automatically move an app across datacenters (because of weather,
power, maintenance), the right thing to do is actually run instances
in both the source and destination datacenters at the same time during
the transition. But we don't want to charge developers for this
doubling, that would be wrong. By having the billing formula respect
max-idle-instances as a hard ceiling, we do the right thing here and
eat these costs ourselves.
> So, based on that understanding of what you're doing, I'm going to take the
> time I was spending looking at options elsewhere and put it into moving my
> data onto HR so I can then switch to python 2.7 with concurrent requests,
> and then see how the out-of-preview works in practice for my app.
Glad to hear!
> Cheers
> --
> Tim
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/guF1P8yyofkJ.
Right. The way the total-instances-rate (the blue line) is computed is
from all running instances that have received a request at any point
in the last 15 minutes. This however is not equal to the line you
proposed, the line which is the maximum value of active-instances-rate
over the last 15 minutes. Look at any app's instances graph and you
will be able to see this visually right away.
>
>
>> > Let's say I have 4 active instances for 5 minutes, then no traffic for
>> > the rest of the day, max-idle-instances set to 1.
>> > 1) 35 minutes will be billed regardless when the scheduler decides to
>> > kill idle instances, right?
>>
>> Yes.
>>
>> > 2) But 1 instance would be idle for at least 15 minutes, right?
>>
>> Maybe.
>
> [...]
>
>> > Let's say I have 4 active instances for 5 minutes, then no traffic for
>> > the rest of the day, max-idle-instances set to 4.
>> > 1) 80 minutes will be billed regardless when the scheduler decides to
>> > kill idle instances, right?
>>
>> Yes.
>>
>> > 2) But 4 instances would be idle for at least 15 minutes, right?
>>
>> Maybe.
>
> So setting max-idle-instances to 4 leads to being reliably billed
> more, but not having any reliable way of measuring the benefit?
The benefit should be visible in terms of average serving latency and
reliability. There should be 4 idle instances running over the time
period you proposed, but there is not a hard guarantee of this.
>
>
> Thanks,
Missing in the last post:
(2) http://groups.google.com/group/google-appengine/msg/1a2d27d15781ddb7
--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
1
1
1
12
So the average should be 4x mode.
As I see it there is no reason to need a "Max Idle Instances" only a "max
queue time".
You know that most browsers will give up if they haven't gotten any data in
30s so 30,000 ms is easily the ceiling.
The interface only allows for 15s, so using that as a ceiling.
Spin Up: If Queue is > Max Que Spin up new instance.
Variant "Economy" Spin-up: If x% of requests have Queue time > max Queue
Spin up new instance.
SpinDown:
Billing is in 15 minute intervals, so you can look at the Load over the last
5 minutes and determine how many instances to kill.
This is how every load balancer / Cloud balancer on the planet works (except
GAE)
If CPU % On Current Instance > X
{
Goto Next Instance
If Last instance in Chain
{
Spin New Instance
}
}
Since App Engine isn't parallel Rather than CPU Load, you look at Serial
Task queue, and how many customers are in line.
A good scheduler would be multi-pass, if you have a "problem customer" (one
that is taking 5s when the average is 500ms) you would take other requests
from the line and drop them in to other instances. (This is really what
Idle instances should be for)
I still see the problem with the scheduler being that we are not specifying
a QoS we are specifying at throttle. (and the throttle is tied to some very
weird math, that isn't documented anywhere)
-----Original Message-----
From: google-a...@googlegroups.com
[mailto:google-a...@googlegroups.com] On Behalf Of Jon McAlister
Sent: Wednesday, September 07, 2011 7:02 PM
To: google-a...@googlegroups.com
Subject: Re: [google-appengine] Re: the min instace time is 24 hours?
Hi Vivek,
It's a good point. The most simple advice I'd have for you is to follow this
rubric:
(1) If you're unhappy with the cost, reduce max-idle-instances.
(2) If you're unhappy with the serving latency and reliability, increase
max-idle-instances.
I think that's really all that most developers should need to do.
There are also lots of optimizations that could made to any app (e.g.
concurrency of the runtime or the task queue or the api calls, use memcache,
use http caching, use high-replication datastore), and these have always
been good things to do, but at a certain point there are diminishing returns
on how much can be achieved with these.
On Wed, Sep 7, 2011 at 6:21 PM, Vivek Puri <v...@vivekpuri.com> wrote:
> Folks, reading all the posts(to be true, i read only the first 10 of
> this thread), it seems everyone where is expected to be a GAE billing
> expert to survive. Although all the discussions sound interesting, i
> dont really have the time to understand all the intricacies, and i
> believe same is the case for other developers. Personally, sole reason
> for choosing GAE was that i can focus on my business model and not
> about balancing set of servers and the billing. It would have been way
> better and easier if you had increased the CPU, bandwidth, storage
> price to cover up for revenues. If you look at AWS, they have never
> done this drastic change in the their service and pricing model. Every
> new product comes with its pricing and stackable in some fashion onto
> existing products.
>
> GAE team still has the chance to cancel this change and let everyone
> get back to their core job, their application. Please dont make it an
> ego issue, if there is still a chance to cancel this change. Just for
> comparison, Facebook had cancelled its Facebook Beacon product in 2007
> after putting in tons and tons of effort and partnerships into it. And
> as far as i can recall, the amount of resistance to Beacon product was
> much less than what i have seen with GAE pricing change.
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/aK7utTL6nOQJ.