I have to leave App Engine platform, feeling really bad and sad

[Xin Liu]

Dear Community,

First of all, I loved App Engine since its inception in 2008, I was very exciting and happy to get my hands on App Engine and Python in June 2008.

But now I'm so sorry and sad that I will have to leave.

My site V2EX.com ( v2ex.appspot.com ) is one of the largest communities of developers and designers in China, it's launched on App Engine since 2010.

Now serving about 40000 page-views a day, 13000+ registered users, 165000+ posts and ranked 11800 according to Alexa.

New pricing means 10x change for me, it will go from $0.9 to $12.76 a day (numbers taken from usage history on Nov 1), so it'll be $360 a month.

I'm living in China, $360 can do many meaningful things here, it's almost enough to hire someone.

Since my site is not making enough money now, at least it can't cover new pricing, so I'm afraid that I'll have to leave and port my site to AWS/Linode. Because new pricing not just more expensive, it also prevented me to develop new features. Since my site is a very active community, so even I got it optimized, memcache can't always help because each time when users post something, then pages need to be updated. More active posting, more updates, and that leads to millions datastore reads/writes every day. Btw I was just curious about how much would it cost to run Jaiku.com under new pricing?

In old pricing, datastore reads/writes are not charged, so it was safe for active communities like my site. But now, more interactions mean unaffordable bills. I'm feeling really really bad about this change.

Perhaps it's also a good chance for me to figure out a better way to make more money.

Hasta la vista, Google App Engine.

Xin Liu

@livid

[Jeff Schnitzer]

I think you should re-examine your data model.

40,000 page views per day is not a lot - if *every single one* of
those wrote to an entity, you're still under the free 50k datastore
operations. And even if every one of those page views wrote to
*three* entities, an extra 100k writes will cost you all of $0.10.

If you exceed the $9/mo minimum spend on 40k page views per day, you
are doing something wrong. If I had to guess, your schema is probably
*way* too normalized and you're probably not using memcache
effectively.

Jeff

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/cmUxJ1LeSZYJ.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

[James Gilliam]

I am sad with you. Like you, many of us are struggling with the new
pricing changes. I had to redo lots of code to make it more new
pricing structure friendly. I use to be optimized around cpu. I had
to adjust to instances and now to overall reads and writes. GAE
missed the fact that code was written to optimize around their
original pricing structure.

[hector@ISB]

This is an interesting option, if you don't wish to rewrite your code:  http://code.google.com/p/appscale/  Keep the option open to come back to GAE if your site becomes hugely popular.

Another way to look at it is that $360/month for 13,000 users comes down to about $0.028/user/month.  Maybe advertising revenue could help cover that cost.  Or a premium membership fee.

Sometimes you have to spend money to make money, right?

[Xin Liu]

My site's typical page is like this:

http://www.v2ex.com/t/7577

As you have seen, a topic with dozens replies, each reply carry 3 entities: reply, member and avatar. And there are other supporting dynamic elements on the page (most of them can be cached).

But when someone replied to the topic, then it will trigger a lot memcached objects to be removed. For hot topics, memcached objects can just survive minutes or even seconds.

And there could be 100+ or even 1000+ entities on one page.

If I knew they're going to charge datastore reads/writes, I may think another way to write the code. But now, it's too painful to rewrite the whole site.

[Jeff Schnitzer]

FWIW, GAE has always changed for datastore reads & writes, they just
upped the price.

It sounds like you have a fundamental architectural issue in that your
cache invalidation is too heavy-handed. I would look for ways to
invalidate smaller parts of the cache, or put it on a schedule
(presumably you don't need *realtime* updates of the page content). I
sympathize that GAE's old pricing model led you to an cost-inefficient
architecture, but I'm not sure you're going to do better on AWS -
pulling thousands of items per second out of a clunky EBS-backed MySQL
might be a problem at the bottom end of the budget.

Suerte,
Jeff

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-appengine/-/0dfGjryGNWwJ.

[Paul]

1000+ entities on one page? How did you manage that? It really sounds
like architectural flaw. I am writing an app with huge amount of
constantly changing data (that needs to be persisted) and I don't have
that problem.

[Sun Jing]

I think there must be a way to optimize, but I agree with you that it's not worth redesigning and rewriting your site.

In my case, I spent dozens of hours to reduce about 50% datastore reads of my blog.

I could fetch entities with just one line of code before, and it's automatically memcached.

But now, I have to query keys first, then check memcache to find missing entities, batch get the rest, set them into memcache, merge all the entities in the proper order. Since I have several memcache keys for each entity (like id, name, title, url, etc.), the reality is even worse.

 

It seems you can't reduce the cost as acceptable as before without dropping many features or implementing in a extremely strange way.

With the rework time, it's almost enough to move to other platform and scale better.

Well, GAE may be easy to scale, but money isn't. I seldom see other platforms raise price, but GAE keeps raising price and reducing the free quotas, at least 3 times I known.

----------
keakon

My blog(Chinese): www.keakon.net

Blog source code: https://bitbucket.org/keakon/doodle/

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

[mscwd01]

@Jeff - Saying: "40,000 page views per day is not a lot - if *every

single one* of
those wrote to an entity, you're still under the free 50k datastore

operations" simply isn't true, is it?

One entity with, say, 5 properties would take at least 5 (probably
more) write operations to persist it. This means 40,000 * 5 would be
200,000 datastore operations - way over the free limit.

Please correct me if I'm wrong, this is just my understanding of how
this works.

[Jeff Schnitzer]

On Thu, Nov 3, 2011 at 5:24 PM, mscwd01 <msc...@gmail.com> wrote:
>
> One entity with, say, 5 properties would take at least 5 (probably
> more) write operations to persist it. This means 40,000 * 5 would be
> 200,000 datastore operations - way over the free limit.

It doesn't work this way, but you're right that I'm probably
undercounting. GAE writes once for the entity and then once for each
index - not for each property. You get as many properties as you want
(up to the 1MB entity size limit) "free" as long as they are
unindexed.

Still, each indexed property is two additional writes (one asc, one
desc) and must be factored in to the plan.

On the other hand, only a small percentage of the OP's 40k page views
are likely writes, and "overage" is ten cents per 100k writes. This
is not his/her problem.

Jeff

[Thomas Wiradikusuma]

Typical startup flow is like this:
1. build an app with the most rapid way (forget about best coding
practices etc)
2. get traction
3. get funding
4. pay someone smarter to refine the code and make it scale

Even in the 'right way' scenario, people often defer optimization as
"premature optimization is the root of all evil". But developing in
GAE forces us to optimize from the beginning (e.g. must use index,
must denormalize, must make program startup fast).

That's not all bad, but I think it's making us slow. Partly because
we' accustomed to develop "normal" apps (use SQL with all its bells
and whistles, use Spring for Java, freedom of framework/stack, etc).
What do you guys think?

[Leandro Rezende]

Thomas i totally agree with u... im still learning to denormalize =( ... im missing too much the true SQL commands... inner joins, groups...

2011/11/4 Thomas Wiradikusuma <wiradi...@gmail.com>

[Adrian Scott]

I share your concern, Thomas... It is also worth comparing the performance of an early stage (#1 in your list) non-optimized app at low volumes on an AWS box versus on GAE (e.g. pageload time)... something to think about, no?

-A


--
Adrian Scott, Ph.D.
CEO, Founder
CoderBuddy
http://www.coderbuddy.com/ <-- Create a Google App Engine app in a minute without installing anything

[Gerald Tan]

What you should be doing is to combine all the entities required for one page into a single entity. Each time an entity is added to a page, you update the page. This way, each page visit should only trigger one datastore read. Of course, writing will be slightly more expensive since you will need to write the entity, read the page and update it, but it should be fair assumption that you will be reading a lot more than you are writing.

[de Witte]

Hi Xin,

It seems like you have the exact same problem as Keakon.

http://code.google.com/intl/nl/appengine/forum/?place=topic%2Fgoogle-appengine%2F5XGPbkiGw0Y%2Fdiscussion

I will not copy and paste my writing.

The solution is simple and doesn't cost you that much time to rewrite. It is important that you change your mindset, don't use the old mysql/php style to build a page.

We are currently building a forum application, which is basically the same as your website or any other blog. It has exact the same features and even more. De design stays under the free quota with 500K pageviews, with 3000 new posts and new 500 topics, per day.

The fact is that it can be done, and much cheaper than alt hosting, but you have to invest time in a good design.

Good luck

[hector@ISB]

I don't think denormalizing your data counts as optimization.  I think the problems here have more to do with bad design.  Inserting a layer in your stack to memcache data should not require a re-design, just a wrapper around your data access interface.

[Jeff Schnitzer]

On Thu, Nov 3, 2011 at 7:34 PM, Thomas Wiradikusuma
<wiradi...@gmail.com> wrote:
> Typical startup flow is like this:
> 1. build an app with the most rapid way (forget about best coding
> practices etc)
> 2. get traction
> 3. get funding
> 4. pay someone smarter to refine the code and make it scale

Good luck with that. If you cannot do #4 on your own then I don't
believe you will be able to hire someone else who can. Especially not
in this market.

My philosophy:

1) Understand the architecture of the platform you're working with
2) Bootstrap a viable product that can generate revenue
3) Iterate the business as fast as you can. Every day should be spent
developing new features, not trying to make the old features scale.

> That's not all bad, but I think it's making us slow. Partly because
> we' accustomed to develop "normal" apps (use SQL with all its bells
> and whistles, use Spring for Java, freedom of framework/stack, etc).
> What do you guys think?

There is an acclimatization period to working with GAE. It's much
shorter than the acclimatization period required to work with
Spring/Hibernate/SQL, except that perhaps you've already gone through
the later. After you work with GAE for long enough, most
architectural issues become pretty obvious. I find that I am now
*much* faster developing GAE apps than J2EE apps, and I never find
myself scratching my head wondering why some query isn't performing
the way I expect.

Jeff

[Thomas Wiradikusuma]

> > 1. build an app with the most rapid way (forget about best coding
> > practices etc)

> > 4. pay someone smarter to refine the code and make it scale
>
> Good luck with that.  If you cannot do #4 on your own then I don't
> believe you will be able to hire someone else who can.  Especially not
> in this market.

Hi Jeff, I agree with you completely. But let's see a hypothetical
scenario:

You and Mr. Joe just happen to work on the same idea, at roughly the
same time.
For simplicity sake, both of you are competent Java programmers with
same skills.
He decides to use AWS and you go with GAE.

Since AWS uses good old technologies, he's faster to deliver. Of
course, yours is
more scalable, but probably you won't get to the point where it is
required,
because Mr. Joe already grab the market.

Being scalable is a requirement.. at later stage. But before everyone
sees anything,
being *first* is more important.

Even if you are able to deliver at the same time, as Adrian pointed
out, his unoptimized
version would probably has better performance than yours (and no
deadline exception).

> There is an acclimatization period to working with GAE.  It's much
> shorter than the acclimatization period required to work with
> Spring/Hibernate/SQL, except that perhaps you've already gone through
> the later.  After you work with GAE for long enough, most
> architectural issues become pretty obvious.  I find that I am now
> *much* faster developing GAE apps than J2EE apps, and I never find
> myself scratching my head wondering why some query isn't performing
> the way I expect.

I think many GAE developers are corporate programmers who use GAE on
the
side (it's rare to find someone who quit his job to focus on a
platform
that's still on preview). As such, their acclimatization to GAE is
slow(er than their existing experience). But I agree with you.

Until that happen, I'll still scratch my head when some framework
doesn't work :)

[Brandon Wirtz]

I am sure I'm faster on GAE in python than I would be in any other Python
environment. I don't tweak anything, and I don't fight over which library or
secondary technologies I'm going to use.

I worked with a Ruby team that spent more time fighting over which DataBase
technology and when gems they were going to use than they did doing the
coding.

Give people choices and they will just waste time making decisions.

As a 1 man shop, write for what ever you want. If you have a team, GAE
means you write code within the limits of the platform, and you get things
done fast with need for fewer meetings.

--

[Jeff Schnitzer]

On Sat, Nov 5, 2011 at 12:42 AM, Thomas Wiradikusuma
<wiradi...@gmail.com> wrote:
>
> You and Mr. Joe just happen to work on the same idea, at roughly the
> same time.
> For simplicity sake, both of you are competent Java programmers with
> same skills.
> He decides to use AWS and you go with GAE.

In this case, Joe is doomed.

Given teams with equivalent expertise, the GAE team will be
significantly more productive. There are exceptions to this - some
problem domains (sophisticated geo, analytics) are more difficult to
do without specialized storage systems - but in the general case, GAE
development is faster. I'll be deploying code while Joe will be
working on getting his database configured properly.

Of course, if you're sandbagging the GAE team with coming up to speed
from scratch, then sure it will probably take longer. But this is an
argument for never learning new technology - expertise gets you into a
local maxima of efficiency, and only by climbing other (sometimes
tortuous) learning curves do we learn that the "old way" actually
sucked. The GAE curve is worth it.

Jeff