vijayp on migration of partychat from GAE to EC2/GAE-hybrid; from $20/day expense to < $1/day

[Adrian Scott]

http://www.vijayp.ca/blog/?p=162

Interesting data...

-A


--
Adrian Scott, Ph.D.
CEO, Founder
CoderBuddy
http://www.coderbuddy.com/ <-- Create a Facebook or Google App Engine app in a minute without installing anything

[Jeff Schnitzer]

As someone that uses partychapp every day (it's my "campfire"), I'm
actually really sad about this move. I'm disappointed that some parts
of GAE are wildly overpriced and create annoying "non-features".

Fortunately it's fairly easy (and getting easier all the time) to set
up these extra services elsewhere in the cloud.

Jeff

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

--
We are the 20%

[Raymond C.]

The comment from Dave Peck on that post is interesting as well:

This is quite interesting, thank you. As someone who has worked with and on App Engine since the early days (I gave a talk at Google I|O 2009 about scaling with GAE) I agree for the most part with your final conclusion: “no system that is likely to become product ionized at scale should be written on App Engine.”

This is a sad conclusion to arrive at after all these years, especially when the original promise of App Engine was (essentially) “write your applications against our strange, quirky API, and they’ll scale far more cheaply and reliably than they could otherwise.” I think the pricing changes, coupled with regular performance problems, gives the lie to that promise.

 

I’m porting several of my apps off of App Engine, including a big production app (https://www.getcloak.com/) — luckily, the only “odd” service I make use of is the Task Queue. (Specifically, I make use of Task Queue transactionality — so there isn’t a direct analog in, say, Celery.) GetCloak.com is heading to AWS as well.

http://www.vijayp.ca/blog/?p=162#comment-761

[Shane Elbo]

I've just started to learn about GAE.  Should I go for this strategy instead of full rely on GAE?

[Leandro Rezende]

run as u can =P

2012/1/5 Shane Elbo <shane...@new5s.com>

I've just started to learn about GAE.  Should I go for this strategy instead of full rely on GAE?

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/f0eK3bnZZJYJ.

[Andrin von Rechenberg]

One other interesting thought is that people often justify the higher price with

the stability of GAE and that Google takes care of running your system and

you dont have to carry a pager.

The flaw with this justification is, that if my app would cost $50'000 a month on

AppEngine and $20'000 somewhere else, I would run it somewhere else!

For the extra $30'000 i would save, i could hire 2 really great Site Reliability

Engineers that take care of running the system and make the experience for

my developers just as good as GAE's. Many people have made systems scale

outside of AppEngine, which is for an experienced engineer as hard as adopting

to GAEs limitations. So I'm wondering who Google targets at the moment with

AppEngine. For people that really think big, GAE might not be the choice anymore.

And Amazon is really taking the market these days.

I truly hope that one day GAE is so competitive that it takes the market as

much as EC2 is taking it now. From my point of view, the pricing changes were

a huge set back (maybe it was a success for Google's Business side), but I'm

positive that sometime in the future GAE will become more competitive again.

Cheers,

A big GAE-Lover

[Ikai Lan (Google)]

Andrin, that's a totally fair argument, and one that we are always talking about. I think there are ways we can make GAE can provide value, but the key is for us to think about value, not price. If we obsess about price per megabyte of bandwidth, we're losing sight of our goal of enabling developers. We need to demonstrate that the total cost of ownership of GAE is top notch by continuing to execute and improve the overall platform.

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com | twitter.com/ikai

[jon]

Ikai, sure Google has chosen to differentiate on advanced features,
not price. It's OK to charge a little more than your competitors. The
question is, how do you know if you're now charging correctly? Are you
charging too much? Are you charging too little? How do you know?

Based on user feedback, I'd say you've charged too much. I don't mind
most of the price increases, but the datastore operations are very
expensive. It's so expensive that the Relation Index Entity technique
proposed by your own Brett Slatkin has become cost prohibitive. If
Twitter was built on GAE using Relation Index Entity, each tweet made
by a popular user with 1 million followers would cost $1 each. Let's
say there are 1000 such users on Twitter. And if each one tweeted 10
times a day, the cost would be $10,000 a day!

I mean c'mon, Google must be seriously overcharging when best
practices/solutions suggested by Googlers themselves have become
unusable.

> >> 2012/1/5 Shane Elbo <shane.e...@new5s.com>

[Brandon Wirtz]

I can name that tune in a LOT less money. We are actually looking at
building a twitter-esque service on GAE. (actually doing Video with Text and
Audio) and we are estimating that we can push Video,Audio and Text at a
price of .32 cents a gig delivered regard less of number of follows or
followers. Our biggest concern is the 12 cents each way on the bandwidth
more than just about anything else.

We did the math on things, and the biggest limitation is the size of the
memcache, which doesn't scale with the number of instances or the size of
the instances. But we even solved most of that issue. (sorry not sharing
how).

But in all the conversations about price it always comes down to things are
expensive if you do things the way you are used to doing them rather than
doing them the way GAE is optimized to do them. My biggest complaint with
pricing is that what costs money has shifted more than once. Which cause use
to "re-optimize". I realize that I exploit the pricing to get the best
price... but as with all pricing you have to lose money on one or to savvy
guys to make up for the not so savvy guys you take to the cleaners :-)

[jon]

> I can name that tune in a LOT less money.  We are actually looking at
> building a twitter-esque service on GAE. (actually doing Video with Text and
> Audio)  and we are estimating that we can push Video,Audio and Text  at a
> price of .32 cents a gig delivered regard less of number of follows or
> followers.

I was talking about fan out cost only, so content format (video, audio
etc) had no relevance to my Twitter example.

> We did the math on things, and the biggest limitation is the size of the
> memcache, which doesn't scale with the number of instances or the size of
> the instances. But we even solved most of that issue. (sorry not sharing
> how).

We end up using memcache to handle fan out data.

> But in all the conversations about price it always comes down to things are
> expensive if you do things the way you are used to doing them rather than
> doing them the way GAE is optimized to do them.

Argh you didn't read my previous message properly.

Fact 1: Brett Slatkin, a Googler, gave a talk about the use of
Relation Index Entity as a best practice fan out implementation. He
mentioned that this was how Jaiku was implemented.

Fact 2: I went ahead and implemented this for my own app. I assume
many other developers did.

Fact 3: Price change rendered RIE cost prohibitive.

It's unfair to say that expensive bills are due to developers' fault.
Surely some of us are bound to do the right thing.

> My biggest complaint with
> pricing is that what costs money has shifted more than once. Which cause use
> to "re-optimize". I realize that I exploit the pricing to get the best
> price... but as with all pricing you have to lose money on one or to savvy
> guys to make up for the not so savvy guys you take to the cleaners :-)

Yep this sucks. What's also sad is that the price hike forces
developers to re-implement alternative solutions that are clunkier and
more error-prone. The RIE-based solution that we abandoned was so
simple and elegant.

[Rohan Chandiramani]

I would really love to hear an opinion of one of the googlers on jon's post.

[Richard Watson]

Hi Jon,

While I think there are some cases that need to be tweaked, the example you provided doesn't seem too insane to me.  If I had a service with 1000 1-million-follower users and I couldn't make a buck each time they tweeted, I wouldn't think it's Google's fault.  Saving, storing and sending 1 million messages from an obviously high-value customer is going to cost someone money, but must bring opportunities as well.  That's a great asset someone should be able to monetize and I imagine VC's would be all over it.  Besides, long before I got to that scale I'd be optimizing cases like readers who only touch the site once a week, or month, or year. Must be many of those.

Having said that, I don't work at your scale so I'm certainly not feeling the same pain yet!

Regards,

Richard

[stevep]

When achieving competitive pricing demands customers can understand
and apply a secret sauce, something is seriously lacking. Although I
like GAE, it needs a lot of work on non-technical, whole-product
solution elements. stevep

[Stephen]

On Sun, Jan 8, 2012 at 8:12 AM, jon <jonni...@gmail.com> wrote:
>
> It's so expensive that the Relation Index Entity technique
> proposed by your own Brett Slatkin has become cost prohibitive.
>
> If Twitter was built on GAE using Relation Index Entity, each tweet made
> by a popular user with 1 million followers would cost $1 each.

I make it $2, as there is both an ascending and descending index on
the MessageIndex.receivers list property, even though you only need
one for equality matching.

Actually, it's probably more like $3 as you need to get tweets in date
order, which will require a compound index on receivers and date. This
is looking pretty inefficient as $2 of the $3 cost is the overhead of
two unused single property indexes.

> Let's say there are 1000 such users on Twitter. And if each one  tweeted 10
> times a day, the cost would be $10,000 a day!

Apparently people tweet 250,000,000 times a day (2,900/sec), and the
average user has 136 followers. If twitter used the relatiion index
pattern on App Engine they might spend aprox. $105,000/day for write
opperations to store tweets, about $38 million per year (or
$1.20/sec).

[jon]

Stephen thanks for the correction. Obviously I didn't try hard enough
to paint a bleak picture :)

Richard I highly doubt that Twitter would be able to turn a tweet into
something worth, say, $5. Remember that most tweets are rather
mindless and of low quality.

On Jan 10, 5:33 am, Stephen <sdeasey+gro...@gmail.com> wrote:

[Greg]

Appengine is great for some things, and not so great for others. If
you have a Twitter-level service, you are going to be better off
running it on your own optimised platform, end of story. If you have
the royal wedding website to host, Appengine is great because you pay
for those few days and no more.

It's also great for most developer-centric startups. you can get your
service up and running without the considerable expense of hiring good
sysadmins. If you don't hit the big time, you'll pay almost nothing.
If you do, it'll handle it until you have the resources to hire those
sysadmins to build you that optimised platform.

I don't understand why everyone expects Appengine to be the perfect
platform for every possible application. If vijayp feels Partychat is
better on AWS, good for him. Although I do hope he's got the app
distributed over more than one domain though, to prevent the day-long
outages they've had in the last six months. Personally I've chosen
appengine so I don't have to even think about such issues, but that's
because I'm more interested in developing apps than architecting
scalable and highly available systems.

YMMV.

[Brandon Wirtz]

I think your view is far too simplistic. Twitter doesn't run that close to
the metal. Even at their scale the man resources to do some of the
optimizations don't work out because of the migration to those
optimizations.

Life isn't just about the hosting cost. IT Administration is a large
portion of what you are outsourcing. Many of the people on this list have a
tendency to think in terms of "what can I do with just me" not what would
this cost if I needed a team to manage it.

-----Original Message-----
From: google-a...@googlegroups.com
[mailto:google-a...@googlegroups.com] On Behalf Of Greg
Sent: Tuesday, January 10, 2012 1:44 AM
To: Google App Engine
Subject: [google-appengine] Re: vijayp on migration of partychat from GAE to
EC2/GAE-hybrid; from $20/day expense to < $1/day

YMMV.

--