New Pricing for high traffic apps - in our case 2x == ~$20'000 per month

[Andrin von Rechenberg]

Hi there

We are running a Python GAE service called MiuMeet.

The current cost is about $296 per day. In November it will be $600 per day.

Most of our cost comes from the Frontend Instance hours. We apparently need

4677h per day, thats roughly 200 instances. We peak at around 240 qps. The

average latency is 800ms / request. 240 * 0.8 => 192. So it seems 200 instances

is exactly what we need at peak.

However our instances are only 25% active, see here: http://bit.ly/pXZH4t

We need someone to talk to about the new pricing model and what can be done.

Idle instances will cost us an extra $10k per month. We can't afford that.

The biggest problem we see is that it seems that AppEngine instances are idle

often by design: The datastore is slow you have to wait and idle around, that's

a fact for us.

We would be really grateful for any advice.

-Andrin, Founder of MiuMeet

PS: Here is our billing comparison

Resource	Used	Free	Billable	Charge
CPU Time:  $0.10/CPU hour	2,797.15	6.50	2,790.65	$279.07
Bandwidth Out:  $0.12/GByte	109.32	1.00	108.32	$13.00
Bandwidth In:  $0.10/GByte	30.55	1.00	29.55	$2.96
Stored Data:  $0.005/GByte-day	101.13	1.00	100.13	$0.51
Recipients Emailed:  $0.10/1000 Emails	0.00	2.00	0.00	$0.00
High Replication Storage:  $0.008/GByte-day	109.92	0.50	109.42	$0.88
Backend Usage:  Prices	$0.00	$0.72	$0.00	$0.00
Always On:  $0.30/Day	No	-	-	$0.00
Total:				$296.42

Estimated Charges Under New Pricing

The charges below are estimates of what you would be paying once App Engine's new pricing model goes live. The amounts shown below are for your information only, they are not being charged and therefore do not affect your balance.

If you would like to optmize your application to reduce your costs in the future, make sure to read our Optimization Article. If you have any additional questions or concerns, please contact us at: appengine_up...@google.com.

Frontend Instance Hour costs reflect a 50% price reduction active until November 20th, 2011.

Resource	Used	Free	Billable	Charge
Frontend Instance Hours:  $0.04/Hour	4,677.04	24.00	4,653.04	$186.13
Backend Instance Hours: $0.08/Hour	0.00	9.00	0.00	$0.00
Datastore Storage:  $0.008/GByte-day	109.92	1.00	108.92	$0.88
Blobstore Storage:  $0.0057/GByte-day	101.13	5.00	96.13	$0.55
Datastore Writes:  $1.00/Million Ops	55.69	0.05	55.64	$55.64
Datastore Reads:  $0.70/Million Ops	87.12	0.05	87.07	$60.95
Small Datastore Operations:  $0.10/Million Ops	192.77	0.05	192.72	$19.28
Bandwidth In:  $0.10/GByte	30.55	1.00	29.55	$2.96
Bandwidth Out:  $0.15/GByte	109.32	1.00	108.32	$16.25
Emails:  $0.01/100 Messages	0.00	1.00	0.00	$0.00
XMPP Stanzas:  $0.01/1000 Stanzas	0.00	1.00	0.00	$0.00
Opened Channels:  $0.01/100 Opens	7,180.91	1.00	7,179.91	$71.80
Total*: (before clipping to daily budget)				$414.44

* Note this total does not take into account the minimum per-application charge in the new pricing model.

[Philip]

Hi Andrin,

have you tried the Max Idle Instances knob? If you set it to 1 for
example you will only be billed for the instances that are used + 1
idle instance.

On Sep 5, 10:11 am, Andrin von Rechenberg <andri...@gmail.com> wrote:
> Hi there
>
> We are running a Python GAE service called MiuMeet.
>
> The current cost is about $296 per day. In November it will be $600 per day.
>
> Most of our cost comes from the Frontend Instance hours. We apparently need
> 4677h per day, thats roughly 200 instances. We peak at around 240 qps. The
> average latency is 800ms / request. 240 * 0.8 => 192. So it seems 200
> instances
> is exactly what we need at peak.
>
> However our instances are only 25% active, see here:http://bit.ly/pXZH4t
>
> We need someone to talk to about the new pricing model and what can be done.
> Idle instances will cost us an extra $10k per month. We can't afford that.
>
> The biggest problem we see is that it seems that AppEngine instances are
> idle
> often by design: The datastore is slow you have to wait and idle around,
> that's
> a fact for us.
>
> We would be really grateful for any advice.
>
> -Andrin, Founder of MiuMeet
>
> PS: Here is our billing comparison
>

>   Resource Used Free Billable Charge *CPU Time:*
> $0.10/CPU hour 2,797.15 6.50 2,790.65 $279.07 *Bandwidth Out:*
> $0.12/GByte 109.32 1.00 108.32 $13.00 *Bandwidth In:*
> $0.10/GByte 30.55 1.00 29.55 $2.96 *Stored Data:*
> $0.005/GByte-day 101.13 1.00 100.13 $0.51 *Recipients Emailed:*
> $0.10/1000 Emails 0.00 2.00 0.00 $0.00 *High Replication Storage:*
> $0.008/GByte-day 109.92 0.50 109.42 $0.88 *Backend Usage:*
> Prices<http://code.google.com/appengine/docs/billing.html#Billable_Quota_Uni...>
> $0.00 $0.72 $0.00 $0.00 *Always On:*
> $0.30/Day No - - $0.00 *Total:* $296.42

>  Estimated Charges Under New Pricing
>
> The charges below are estimates of what you would be paying once App
> Engine's new pricing

> model<http://www.google.com/enterprise/cloud/appengine/pricing.html>

> goes
> live. The amounts shown below are for your information only, they are not
> being charged and therefore do not affect your balance.
>
> If you would like to optmize your application to reduce your costs in the
> future, make sure to read our Optimization

> Article<http://code.google.com/appengine/articles/managing-resources.html>.

> If you have any additional questions or concerns, please contact us at:

> appengine_updated_pric...@google.com.

>  Frontend Instance Hour costs reflect a 50% price reduction active until
> November 20th, 2011.

>  Resource Used Free Billable Charge *Frontend Instance Hours:*
> $0.04/Hour 4,677.04 24.00 4,653.04 $186.13 *Backend Instance Hours:*
> $0.08/Hour 0.00 9.00 0.00 $0.00 *Datastore Storage:*
> $0.008/GByte-day 109.92 1.00 108.92 $0.88 *Blobstore Storage:*
> $0.0057/GByte-day 101.13 5.00 96.13 $0.55 *Datastore Writes:*
> $1.00/Million Ops 55.69 0.05 55.64 $55.64 *Datastore Reads:*
> $0.70/Million Ops 87.12 0.05 87.07 $60.95 *Small Datastore Operations:*
> $0.10/Million Ops 192.77 0.05 192.72 $19.28 *Bandwidth In:*
> $0.10/GByte 30.55 1.00 29.55 $2.96 *Bandwidth Out:*
> $0.15/GByte 109.32 1.00 108.32 $16.25 *Emails:*
> $0.01/100 Messages 0.00 1.00 0.00 $0.00 *XMPP Stanzas:*
> $0.01/1000 Stanzas 0.00 1.00 0.00 $0.00 *Opened Channels:*
> $0.01/100 Opens 7,180.91 1.00 7,179.91 $71.80 *Total*:* (before clipping to

> daily budget) $414.44
>
> * Note this total does not take into account the minimum per-application
> charge in the new pricing

> model<http://www.google.com/enterprise/appengine/appengine_pricing.html>
> .

[Andrin von Rechenberg]

That wont help in our case. As I tried to explain before, we need 200 instances

if we serve 240 qps at 800 ms. (240qps * 0.8s = ~200 instances).

75% of the time they are idle, waiting for rpc responses (mostly datastore) in

order to handle the current response. So they need to be running and can't be shut

down. At least that's our understanding. Or am I wrong?

-Andrin

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Philip]

If you take a look to you dashboards instance chart: You will be
billed for Active Instances + Max Idle Instances.
See this post: http://groups.google.com/group/google-appengine/browse_thread/thread/727f420db5b111cc

[renderpaz]

The graph of instances is very confusing to me as neither line seems to correspond to the number of instances running, or running with > 0 qps, try playing with those instance "knobs" they can dramatically effect instance count, but for a high qps app like yours, I don't think you will see a dramatic change.

Are you running python?

it sounds like your will benefit heavily from Python 2.7 with concurrent request handling per instance.  If your requests spend the majority of their time waiting, should be no problem to do 4 up, which could really drop your instance costs.

I'd then try and tackle your datastore reads.  While I have no idea about your app, it seems a bit odd to me that you have more reads that writes, in my app I have a 5:1 write to read ratio as I cache just about everything I write.  If you aren't confident you will soon read what you write, cache on read every time.  Memcache is fast and free, something to think about.

[Johan Euphrosine]

HI Andrin,

The default scheduler settings (Automatic) are very aggressive for achieving the instant scalability that you are used to on GAE:

- it keeps a lot of idle instances around to be able to scale instantly on traffic spikes

- it enforce the minimum pending latency possible for your incoming request

You can alter that behavior by tweaking the Max-Idle-Instances and Min-Pending-Latency in "Application Settings > Performance" as described here:

http://code.google.com/appengine/articles/managing-resources.html

Min Pending Latency: Xms will enforce that the scheduler wait at least Xms if all instances are busy before taking the decision to spawn a new instance to handle an incoming request:

-> It will maximize usage of existing instances over new instance creation.

Max Idle Instance: N will instruct the scheduler not to keep more that N idle instances around to handle traffic spikes.

-> It will enforce that you are not billed for more than (Active Instances + Max Idle Instances).

Note that instances waiting for a RPC are not considered "Idle" but "Active" since, from the point of view of the scheduler, they are currently processing a request.

Once multi-threading is implemented with Python 2.7 the scheduler will be able to make use of these active-waiting instances for handling incoming requests, in the meantime you can use the various async APIs availables to be able to make a better usage of the time waiting for RPCs to complete.

Hope that helps.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Johan Euphrosine (proppy)
Developer Programs Engineer
Google Developer Relations

[Johan Euphrosine]

The time needed to spin up an instance is mostly dependent on how long it takes for your application to warm up.

*You* have to choose what are the reasonable values of Max-Idle-Instances and Minimum pending latency for your application:

- according to the number of idle instance (currently 150) you are willing GAE to keep for handling loading spikes (Max-Idle-Instances)

- the amount of user latency you are willing to trade (currently none/very low) for waiting for an existing instance to be available instead spinning a new one immediately (Min-Pending-Latency).

Those are entrepreneurial choices involving cost versus risk strategy that could not be resolved only with technical advises.

Hope that helps.

On Mon, Sep 5, 2011 at 3:38 PM, Andrin von Rechenberg <and...@miumeet.com> wrote:

Proppy, how long does it take to spin up an instance?

What would you say are reasonable values for

Max-Idle-Instances and minimum pending latency?

-Andrin

[Kenneth]

It looks like you're getting hammered in the database and channel area since your instance charges are actually less than your old cpu charges, although maybe that's due to channels.  I don't know anything about channels myself, maybe there's an optimization there or with the datastore access?

[Andrin von Rechenberg]

Thanks for your help.

that sounds promising.

One question I have is how crashing instances are handled:

If I get a warning message like this:

After handling this request, the process that handled this request was found to be using too much memory and was terminated. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you may have a memory leak in your application.

will the instance spin up a new process within milliseconds?

Yes I have a lot of datastore reads, but this is mostly because all queries are different and caching the responses wont give me much. I use memcache very heavily. I'm also working on a backend for all these different queries to be served out of RAM...

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ASjp-0R-u8wJ.

[Jean-Marc Truillet]

When are the created idle instances deleted when the traffic is going
down? After an amount of time? When the latency is below a certain
duration?

Thanks,

JM

> >>>   Resource Used Free Billable Charge *CPU Time:*
> >>> $0.10/CPU hour 2,797.15 6.50 2,790.65 $279.07 *Bandwidth Out:*
> >>> $0.12/GByte 109.32 1.00 108.32 $13.00 *Bandwidth In:*
> >>> $0.10/GByte 30.55 1.00 29.55 $2.96 *Stored Data:*
> >>> $0.005/GByte-day 101.13 1.00 100.13 $0.51 *Recipients Emailed:*
> >>> $0.10/1000 Emails 0.00 2.00 0.00 $0.00 *High Replication Storage:*
> >>> $0.008/GByte-day 109.92 0.50 109.42 $0.88 *Backend Usage:*
> >>> Prices<http://code.google.com/appengine/docs/billing.html#Billable_Quota_Uni...>
> >>> $0.00 $0.72 $0.00 $0.00 *Always On:*

> >>> $0.30/Day No - - $0.00 *Total:* $296.42

> >>>  Estimated Charges Under New Pricing
>
> >>> The charges below are estimates of what you would be paying once App

> >>> Engine's new pricing model<http://www.google.com/enterprise/cloud/appengine/pricing.html> goes

> >>> live. The amounts shown below are for your information only, they are not
> >>> being charged and therefore do not affect your balance.
>
> >>> If you would like to optmize your application to reduce your costs in the

> >>> future, make sure to read our Optimization Article<http://code.google.com/appengine/articles/managing-resources.html>.

> >>> If you have any additional questions or concerns, please contact us at:

> >>> appengine_updated_pric...@google.com.

> >>>  Frontend Instance Hour costs reflect a 50% price reduction active until
> >>> November 20th, 2011.

> >>>  Resource Used Free Billable Charge *Frontend Instance Hours:*
> >>> $0.04/Hour 4,677.04 24.00 4,653.04 $186.13 *Backend Instance Hours:*
> >>> $0.08/Hour 0.00 9.00 0.00 $0.00 *Datastore Storage:*
> >>> $0.008/GByte-day 109.92 1.00 108.92 $0.88 *Blobstore Storage:*
> >>> $0.0057/GByte-day 101.13 5.00 96.13 $0.55 *Datastore Writes:*
> >>> $1.00/Million Ops 55.69 0.05 55.64 $55.64 *Datastore Reads:*
> >>> $0.70/Million Ops 87.12 0.05 87.07 $60.95 *Small Datastore Operations:*
> >>> $0.10/Million Ops 192.77 0.05 192.72 $19.28 *Bandwidth In:*
> >>> $0.10/GByte 30.55 1.00 29.55 $2.96 *Bandwidth Out:*
> >>> $0.15/GByte 109.32 1.00 108.32 $16.25 *Emails:*
> >>> $0.01/100 Messages 0.00 1.00 0.00 $0.00 *XMPP Stanzas:*
> >>> $0.01/1000 Stanzas 0.00 1.00 0.00 $0.00 *Opened Channels:*
> >>> $0.01/100 Opens 7,180.91 1.00 7,179.91 $71.80 *Total*:* (before clipping

> >>> to daily budget) $414.44
>
> >>> * Note this total does not take into account the minimum per-application

> >>> charge in the new pricing model<http://www.google.com/enterprise/appengine/appengine_pricing.html>
> >>> .

[Andrin von Rechenberg]

Proppy, how long does it take to spin up an instance?

What would you say are reasonable values for

Max-Idle-Instances and minimum pending latency?

-Andrin

On Mon, Sep 5, 2011 at 2:02 PM, Johan Euphrosine <pro...@google.com> wrote: