The Amazing Story Of Appengine And The Two Orders Of Magnitude

[Emlyn]

Hi all,

I don't think I've posted here before, but I've been an appengine user
for a while now (closing on 2 years? Is that even possible?). And like
many, I had a rude shock with the new pricing (going from $0.50/day to
$50/day).

However, I dug into what I'm actually being charged for, and I think
it's all actually in my control to sort out, and that in itself is
sort of fascinating. I wrote a long blog post on this, which people
might find interesting.

The Amazing Story Of Appengine And The Two Orders Of Magnitude
http://point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine-and-the-two-orders-of-magnitude/

I'd be really grateful for feedback, especially if I've gotten
anything wildly wrong. I haven't actually made any of the changes that
I've foreshadowed in the post, that's for the next day or two, and
I'll write a followup article on how it goes.

Thanks in advance for having a look!

--
Emlyn

http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google
Buzz posts,
comments and all.
http://point7.wordpress.com - My blog
Find me on Facebook and Buzz

[peterk]

Very good read, thanks for posting. Will definitely be curious to see
how your changes improve things or otherwise.

On Sep 3, 11:16 am, Emlyn <emlynore...@gmail.com> wrote:
> Hi all,
>
> I don't think I've posted here before, but I've been an appengine user
> for a while now (closing on 2 years? Is that even possible?). And like
> many, I had a rude shock with the new pricing (going from $0.50/day to
> $50/day).
>
> However, I dug into what I'm actually being charged for, and I think
> it's all actually in my control to sort out, and that in itself is
> sort of fascinating. I wrote a long blog post on this, which people
> might find interesting.
>

> The Amazing Story Of Appengine And The Two Orders Of Magnitudehttp://point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine...

>
> I'd be really grateful for feedback, especially if I've gotten
> anything wildly wrong. I haven't actually made any of the changes that
> I've foreshadowed in the post, that's for the next day or two, and
> I'll write a followup article on how it goes.
>
> Thanks in advance for having a look!
>
> --
> Emlyn
>

> http://my.syyn.cc- Synchonise Google+, Facebook, WordPress and Google
> Buzz posts,
> comments and all.http://point7.wordpress.com- My blog

[Nick Rudnik]

Excellent information. This is very similar to my experience and I suspect explains most of the trouble with the new billing scheme. At least the datastore reads are very much within our control. As for keeping the instance count low, I still don't see that as very easy to do. I'd be very curious if you find a way to do it. I've been messing with the sliders for minimum pending latency to keep it above all my averages and I still end up with 9 instances always hanging around to handle a site that has basically NO traffic. I wish Google would explain more about what is happening within that scheduler and ultimately give us the control of when and how to start and stop instances. Something that can be so costly should not be left to a black box algorithm that is controlled by the side who profits from erring on keeping the instance count high.  

[johnP]

Which leads to a potential constructive suggestion. Maybe Goog can
post a troubleshooting guide that lists different line-items in the
new-style billing, and potential gotchas? An example in your blog is
that sudden parallelism is costly and that making things serial is a
good optimization. (Ironically, the opposite of what was being
promoted earlier - mapreduce). Another factoid is that offset queries
are costly. Using queries as a generator is costly, and you should
fetch items at once.

So it can look like this:

Excessive instances cost? Look at these items:
- sudden parallelism
- idle instance setting
- decrease response time
- other?

Excessive writes:
- decrease unneeded indexes
-

Excessive Reads:
- make sure your fetch() rather than looping through results
- check offset queries
- etc...

On Sep 3, 4:36 am, peterk <peter.ke...@gmail.com> wrote:
> Very good read, thanks for posting. Will definitely be curious to see
> how your changes improve things or otherwise.
>
> On Sep 3, 11:16 am, Emlyn <emlynore...@gmail.com> wrote:
>
>
>
>
>
>
>
> > Hi all,
>
> > I don't think I've posted here before, but I've been an appengine user
> > for a while now (closing on 2 years? Is that even possible?). And like
> > many, I had a rude shock with the new pricing (going from $0.50/day to
> > $50/day).
>
> > However, I dug into what I'm actually being charged for, and I think
> > it's all actually in my control to sort out, and that in itself is
> > sort of fascinating. I wrote a long blog post on this, which people
> > might find interesting.
>
> > The Amazing Story Of Appengine And The Two Orders Of Magnitudehttp://point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine...
>
> > I'd be really grateful for feedback, especially if I've gotten
> > anything wildly wrong. I haven't actually made any of the changes that
> > I've foreshadowed in the post, that's for the next day or two, and
> > I'll write a followup article on how it goes.
>
> > Thanks in advance for having a look!
>
> > --
> > Emlyn
>

> >http://my.syyn.cc-Synchonise Google+, Facebook, WordPress and Google
> > Buzz posts,
> > comments and all.http://point7.wordpress.com-My blog

[Joshua Smith]

I also identified task queues as the source of my excessive instances. I suspect this is a quite common issue, and together with datastore access bugs (sorry, but you never should have written it like that) like the one you found, are conspiring to make a lot of these crazy new billing numbers.

I think a lot of people just need to take a deep breath and look at their apps the way you did. It's certainly a lot less work than migrating to EC2.

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>

[peterk]

You can pull from task queues instead of having them push, right? That
could help a lot where task queues are the source of instance spin up.
I'd even be happy to dedicated a back-end to task queue pull-work if
it was necessary - at least that is totally under your control.

On Sep 3, 5:05 pm, Joshua Smith <JoshuaESm...@charter.net> wrote:
> I also identified task queues as the source of my excessive instances.  I suspect this is a quite common issue, and together with datastore access bugs (sorry, but you never should have written it like that) like the one you found, are conspiring to make a lot of these crazy new billing numbers.
>
> I think a lot of people just need to take a deep breath and look at their apps the way you did.  It's certainly a lot less work than migrating to EC2.
>
> On Sep 3, 2011, at 6:16 AM, Emlyn wrote:
>
>
>
>
>
>
>
> > Hi all,
>
> > I don't think I've posted here before, but I've been an appengine user
> > for a while now (closing on 2 years? Is that even possible?). And like
> > many, I had a rude shock with the new pricing (going from $0.50/day to
> > $50/day).
>
> > However, I dug into what I'm actually being charged for, and I think
> > it's all actually in my control to sort out, and that in itself is
> > sort of fascinating. I wrote a long blog post on this, which people
> > might find interesting.
>
> > The Amazing Story Of Appengine And The Two Orders Of Magnitude

> >http://point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine...

>
> > I'd be really grateful for feedback, especially if I've gotten
> > anything wildly wrong. I haven't actually made any of the changes that
> > I've foreshadowed in the post, that's for the next day or two, and
> > I'll write a followup article on how it goes.
>
> > Thanks in advance for having a look!
>
> > --
> > Emlyn
>

> >http://my.syyn.cc- Synchonise Google+, Facebook, WordPress and Google

> > Buzz posts,
> > comments and all.

> >http://point7.wordpress.com- My blog

[Joshua Smith]

Yes, that certainly seems possible. But it's a lot more trouble than just having a way to say to the scheduler, "run these when you have an idle instance with nothing better to do," or "DO NOT spin up an instance just to handle this task"

[Nick Rudnik]

THAT is a fantastic idea. 

[Joshua Smith]

Thanks :)

I've created this feature request issue, in case you want to star it.

http://code.google.com/p/googleappengine/issues/detail?id=5773

On Sep 3, 2011, at 2:26 PM, Nick Rudnik wrote:

THAT is a fantastic idea. 

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/qa8mqxfJ25MJ.

[stevep]

I had spent most of the night wondering about task queue's apparent
inefficiencies under Scheduler, and had planned to open a topic
related to it, but then read Joshua's post.

THIS point made by Joshua is THE HEART of the issue:'...it's a lot

more trouble than just having a way to say to the scheduler, "run
these when you have an idle instance with nothing better to do," or

"DO NOT spin up an instance just to handle this task"...'

As Joshua suggests, there is likely a great deal of optimization
within an instance obtainable by elegantly utilizing TQs. Scheduler
today, however, seems quite inelegant (and unfortunately may remain
that way because instance optimization is markedly revenue and profit
negative for GAE).

At this point, I would like to suggest that GAE employ a consultant
who is an expert with systems modeling with iThink. The current
opaqueness and complexity of Scheduler are an ideal situation for this
type of systems analysis.** Link:
http://www.iseesystems.com/softwares/Business/ithinkSoftware.aspx

Please Google, can you give us some more help here.

Thanks,
stevep


** not at all affiliated with iThink, but have developed some
financial analyses using iThink back in the day when I worked for HP.
It really is a great tool for situations like this, and you guys will
certainly find that there are some very experienced and capable
consultants who will do great things for you without too much time
needed from GAE personnel.

[Emlyn]

On 4 September 2011 01:35, Joshua Smith <Joshua...@charter.net> wrote:
> I also identified task queues as the source of my excessive instances.  I suspect this is a quite common issue, and together with datastore access bugs (sorry, but you never should have written it like that)

I totally agree with you! I don't clearly remember writing that bit of
code, but it looks like a kludgy workaround for a bug. This app is an
evenings & weekends thing, so "what can I do in 10 mins to fix
critical issue X" is often the driving factor. That doesn't lead to
good code ;-)

> like the one you found, are conspiring to make a lot of these crazy new billing numbers.

I think that's going to end up being true. What Google are doing with
the new billing is surfacing the true costs of the platform, which
were hidden before. It's pain we've got to go through, and it'll be
better for all of us in the long run. That said, it would have been
nice to have had a little longer between getting access to the new
billing tools and actually wearing the new costs. But it does focus
the mind!

>
> I think a lot of people just need to take a deep breath and look at their apps the way you did.  It's certainly a lot less work than migrating to EC2.

Absolutely. We need to not lose sight of just how great a PAAS
platform this is, and how much we gain by using it.

[Emlyn]

+1

--
Emlyn

http://my.syyn.cc - Synchonise Google+, Facebook, WordPress and Google

Buzz posts,
comments and all.

http://point7.wordpress.com - My blog

[Sergey Schetinin]

I made a very similar request back in June -- there was zero reaction from Google.

My suggestion is a little different -- provide a separate slider for "maximum latency for task queue requests".

-Sergey

[Joshua Smith]

Here's another approach, which seems a lot more powerful than either of the things we suggested:

http://code.google.com/p/googleappengine/issues/detail?id=5775

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ZgT491lCdLUJ.

[Sergey Schetinin]

Well,.. I intentionally tried to make a suggestion that requires as little deviation from the existing plan as possible. I have no faith in GAE team making anything but baby steps towards the community -- that request in the linked ticket is way too ambitious.

-Sergey

[Joshua Smith]

Let's give the GAE team some credit, and assume that they can discern the short-term-fix issues from the long-term-solution issues.

I starred this new idea, because, in the long term, it would be a really amazing toolset to have for my enterprise apps (not my free app, where all I want to do is keep idle instances from running).  It also would be a level of load balancing control never before seen in the cloud world, which should get some people at google interested.

And I suspect that Brandon would find ways to use it that just blew us away.

On Sep 4, 2011, at 7:47 AM, Sergey Schetinin wrote:

Well,.. I intentionally tried to make a suggestion that requires as little deviation from the existing plan as possible. I have no faith in GAE team making anything but baby steps towards the community -- that request in the linked ticket is way too ambitious.

-Sergey

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/TIcQn3mpRLgJ.

[Emlyn]

Here's the next post, showing the results of just changing the
performance sliders. Again, hard data. graphs, all the good stuff.

http://point7.wordpress.com/2011/09/04/appengine-tuning-1/

[GAEfan]

+1

Also, the ability to make a spider bot wait, without spawning a new
task. Although that may hurt search engine placement.

It is a conflict of interest for google to charge for instances, and
then send their bots around to run up your costs.

Google should be able to add some logic to check if the new instance
were caused by googlebot or the task queue, and make that instance
free.

[Alfred Fuller]

On Sat, Sep 3, 2011 at 8:23 AM, johnP <jo...@thinkwave.com> wrote:

Which leads to a potential constructive suggestion.  Maybe Goog can
post a troubleshooting guide that lists different line-items in the
new-style billing, and potential gotchas?  An example in your blog is
that sudden parallelism is costly and that making things serial is a
good optimization.  (Ironically, the opposite of what was being
promoted earlier - mapreduce).  Another factoid is that offset queries
are costly.  Using queries as a generator is costly, and you should
fetch items at once.

So it can look like this:

Excessive instances cost?  Look at these items:
 - sudden parallelism
 - idle instance setting
 - decrease response time
 - other?

Excessive writes:
 - decrease unneeded indexes
 -

Excessive Reads:
 - make sure your fetch() rather than looping through results

If you look or use fetch the cost is the same (though fetch might be faster as it will pick larger batch sizes)

 - check offset queries

I am sure this is the culprit in Emlyn's case. Use query cursors (http://code.google.com/appengine/docs/python/datastore/queries.html#Query_Cursors) not offsets! :-)

 - etc...

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

[Alfred Fuller]

BTW, I really like the blog post Emlyn.

On Sun, Sep 4, 2011 at 11:54 PM, Alfred Fuller <arfuller+...@google.com> wrote:

On Sat, Sep 3, 2011 at 8:23 AM, johnP <jo...@thinkwave.com> wrote:

Which leads to a potential constructive suggestion.  Maybe Goog can
post a troubleshooting guide that lists different line-items in the
new-style billing, and potential gotchas?  An example in your blog is
that sudden parallelism is costly and that making things serial is a
good optimization.  (Ironically, the opposite of what was being
promoted earlier - mapreduce).  Another factoid is that offset queries
are costly.  Using queries as a generator is costly, and you should
fetch items at once.

So it can look like this:

Excessive instances cost?  Look at these items:
 - sudden parallelism
 - idle instance setting
 - decrease response time
 - other?

Excessive writes:
 - decrease unneeded indexes
 -

Excessive Reads:
 - make sure your fetch() rather than looping through results

If you look or use fetch the cost is the same (though fetch might be faster as it will pick larger batch sizes)

...if you *loop or... :-)

[Eric Jang]

Fantastic post. Thanks for clarifying what an instance was!

[Sergey Schetinin]

Well, I'd love to just trust the team to just figure it out, but that
doesn't seem to work out too well recently, so I don't think being
optimistic is justified here. I'm not saying they can't do it, but
maybe they are understaffed, or the process is broken, but things
aren't going well. (Whoever works at the datastore backend is amazing
though).

--
http://self.maluke.com/

[Steve Sherrie]

I've create this feature request for a way to create scheduler
profiles and then filter traffic into those profiles.
Star it if you think it would be useful.

http://code.google.com/p/googleappengine/issues/detail?id=5775

Steve

On Sep 3, 4:14 pm, Joshua Smith <JoshuaESm...@charter.net> wrote:
> This point is not so simple:

>
> > (and unfortunately may remain
> > that way because instance optimization is markedly revenue and profit
> > negative for GAE)
>

> It really is not in google's economic interest to have idle instances running.  More idle instances means more infrastructure cost, and, if you charge for it, more user backlash.  Idle instances are a necessary evil when you want responsive, dynamic scaling.  But technically, you only need as many as it takes to satisfy the simple equation (rate of join * max wait) / startup time.  That is, you need to have enough on hand to handle the incoming requests while you spin up some more.
>
> The key problem is that for probably all applications, the numerator in that equation is completely different for tasks than it is for users browsing the site.  And it is often different for different kinds of tasks.  (In fact, it's often different for different users/urls/handlers as well, but sometimes too much configurability is a bad thing.)
>
> The scheduler isn't as complicated as many people seem to believe.  The only real magic in it is the heuristics for computing the three values in that equation when the user sets them to "automatic."  The rest is simple trend averaging.
>
> -Joshua

[Emlyn]

I've just posted the last of what became 4 posts in this series.

http://point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine-and-the-two-orders-of-magnitude/
http://point7.wordpress.com/2011/09/04/appengine-tuning-1/
http://point7.wordpress.com/2011/09/07/appengine-tuning-an-instance-of-success/
http://point7.wordpress.com/2011/09/10/appengine-tuning-schlemiel-youre-fired/

tl;dr is, that my pricing's back down really low, things have worked out.

btw I've had great feedback, tips and techniques from this community.
Thanks! I think that, regarding longevity of a tech, the culture that
builds around a it is just as important as the tech itself. All signs
are that AppEngine is going to be a long term viable platform.

On 3 September 2011 19:46, Emlyn <emlyn...@gmail.com> wrote:

[Rohan Chandiramani]

+1 , very good read!

[Gerald Tan]

Nice blog Emlyn.

The reason why your Frontend Instance hours are lower than you expected is because you assumed that you will be billed for the area under the BLUE line in the Instance graph. It's not. You are being billed for the area under the YELLOW line (Active Instance) PLUS your Max Idle Instance setting. So your Active Instances is hovering at around ~0.72, and I assume you have set your application's Max Idle Instance to 1. Therefore ~1.72 * 24 = ~41.28 Instance Hours

[Emlyn]

Oh wow, you're absolutely right. Going back to the billing on the 4th
of September (after I changed Max Idle Instances and before I made any
code changes), I was already seeing the full price drop. So, I didn't
need to make the changes to spread out my tasks; that dropped the blue
line down, but I don't pay for the blue line.

So optimisation is even easier than I thought. Gerald, I'll quote you
in an update to my post, if you're ok with that.

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-appengine/-/dWoTZKzCy7kJ.

> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

--

[Gerald Tan]

I'm ok with that.

Your next concern would be the Datastore Reads. I think you should be able to bring that down a lot by using memcache and/or combining your monitors into a few entities. I'm not familiar with python, but I believe you can use pickle to serialize an array of monitors into a blob which you can store as one entity. You may need to split your list of monitors into a few groups to stay under the max entity size of 1MB, but this will drastically cut down on the number of reads that you need, especially if you combine with a memcache.

[Emlyn]

On 11 September 2011 18:17, Gerald Tan <woeful...@gmail.com> wrote:
> I'm ok with that.

cool

Oh, I don't need to visit all the monitors, far from it. It's just a
bit of crap code doing that which needs to be refactored out entirely.

>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-appengine/-/3QXRW2UJcB8J.