Update on our GAE Experience

[Brandon Wirtz]

I wanted to take a minute to update you on where we are with our GAE Experience since people often tell me I live in my own little world of rainbows and unicorns.

 

GAE Support sucks.  I might as well send messages by carrier pigeons, and that is generous. Filling out a form for support is like putting a note in a bottle, putting that bottle in cement and chucking it in to the ocean and hoping support will see it when they are on a scuba trip. (or made to where cement shoes by the mob) (we have used the quota increase request forms at the two locations we know about, sent emails, nothing)

Uptime is not too bad, but Slowtime is.  Memcache gets slow, datastore gets really slow, chron jobs fire late (or not at all). This is worse on low traffic apps. For some reason if you can get 8 instances running life is happy. The world changes and the unicorns dance.

 

Backends are worthless if you want them to do more than one thing.  Autoscaling backends doesn’t work. Bug just sits there, I’m told things work as expected. Apparently I should just expect that everyone will get a server busy error.

 

Quota limits are really stupid.  We moved a bunch of things off of GAE because GAE has really stupid limits on the amount of data you can download, and the amount of inter-app data you can use.  We have apps that talk to apps, and we routinely bump against quota limits. Doesn’t matter how much traffic we are doing, or what we are spending, you are rate limited on HTTP requests. Well if you consume an API, or use an api you own on another Appengine App, you will just magically explode when you get a traffic spike.

 

So here is where we are at:

We moved a lot of things off of AppEngine. We are going to try and salvage some of this with getting premier support. (though last time we were told it wouldn’t make us happy, this time I’m going to pay the money, find out if I’m unhappy, and if I’m not we will leave, for real, for permanent)

We love that AppEngine saves us from building infrastructure. It is great for rapid prototyping. When the scalability works it is awesome. When the quotas kick in and break us, (and it took 2 weeks to get the quotas lifted last time) it sucks and we look stupid. 

 

I am posting this here because the ONLY way to get support is to post a nasty message in the forums. Anything else is largely ignored.

 

If you want a great product that works and you are ok with that it will work to the limits of what it does, and that your uptime will be good, but you won’t have any insight in to your downtime, Appengine is great. If you need someone to tweak, change, or fix something. Or you want to be able to tell your CEO, “yeah they are fixing that as we speak” and not have it be a lie, this is not the place for you.

-Brandon
650-281-1467

[Brian]

Hey Brandon, I love your analogies! Your post reminds me of a statistic I heard years ago -- that for every complaint a company receives from their customers there are many other customers that never complain but instead silently leave and move on to another supplier. I hope that if Google reads your post they will indeed consider it a gift and remember that there are probably many others experiencing the same issues, but are silently moving on...

[vlad]

You mentioned you have moved things off of GAE. Where are you finding greener pastures? AWS?

[Peter Ondruška]

Yes, AWS, and I am sure I am not alone.

[Brandon Wirtz]

I’m not quite ready to give referrals in Google’s own house.  If you follow me on Facebook/bwirtz you will catch mention from time to time.

[alex]

> I’m not quite ready to give referrals in Google’s own house. If you follow

Oh come on! When did that happen? And where's the BackOps
half-a-screen signature with images and all? This is not Brandon I
knew...

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

[Cesium]

Finally! The seeds of an Alex-Brandon dust-up appear!

This group has been so BORING!

David

Brandon: All good points; well stated.

[Brandon Wirtz]

To a Google competitor. (I don't compete with Google.... Yet) And that was
just my signature.

[Brandon Wirtz]

I can also assure you it was not a move to CloudFlare.

[Brandon Wirtz]

 

>Finally! The seeds of an Alex-Brandon dust-up appear!

>This group has been so BORING!

 

>David

 

Sorry, we built a product that makes SRI’s Siri look like it understands language to the same extent your dog understands “ready to go for walkies?”.

And while we started using the Python NLTK library, (which requires a lot of work to smoosh in to AppEngine) we completely replaced that.  Then we built a politics site using the alpha version that is amazing if a bit more complex than it should be. Then we built the TLDR content summarizer. With all that going on and the stuff we have coming next, is likely to change how Search, CRM, and Enterprise Support are done.

The amount of time I spend in this forum has always been related to making sure I could get my clients the best service from AppEngine, and to foster a community that would help make certain my platform continued to exist, expand, and iterate.

With me spending less time supporting clients, and with me being my only client.  The forum has been less important too me.

 

-Brandon

http://www.Tldrplugin.com

 

 

 

[alex]

Still boring...
Good luck though!

[Nickolas Daskalou]

Thanks for this review Brandon.

Does your assessment apply to all runtime environments (Java, Python and Go)?

Nick

[Brandon Wirtz]

> Does your assessment apply to all runtime environments (Java, Python and Go)?

 

We are primarily Python shop.  Uptime has been pretty close between Py and Java, no clue about go.

 

Support response time I think is the same regardless. So I’m going to say yes.

 

But others might argue that Java sucks more than Python. From the things I have heard, I won’t argue with that.

 

[Nickolas Daskalou]

Cheers Brandon.

Nick

--

[Kristopher Giesing]

On Wednesday, November 21, 2012 7:49:34 AM UTC-8, Brandon Wirtz wrote:

I can also assure you it was not a move to CloudFlare.

Zing! Not boring.

[Mani Doraisamy]

In app engine's defense: We have built a Workflow-as-a-Service and Platform-as-a-Service on top of app engine. The workflow product (http://kissflow.com) runs 1500 appspots & Cloud SQL for our customers on a premier account.

Our workflow engine, rule engine were built grounds up on python for app engine. IMO, backends & quota limits are mostly problems arising out of bigtable data migration (for new application versions) & joins/aggregates (in reporting). Our experience has been great after moving to Cloud SQL.

thanks,

mani

[Brandon Wirtz]

>IMO, backends & quota limits are mostly problems arising out of bigtable data migration (for new application >versions) & joins/aggregates (in reporting).

 

A few “for instances”

We get feeds from Associated Press. When they update the feed and we try to grab a few hundred images at a time, we hit URL Fetch limits.

We have a Language Heuristics Engine that is a core tech of all our product, we hit limits on how many request from other AppEngine Apps can hit that.

 

We were trying to do Large jobs with backends, which are supposed to autoscale. They don’t.

While you are right that those limits hit during migrations, they are also part of our daily life. We spend a lot of time thinking about how we can engineer around the synthetic limits of Appengine.  And often how those limitations are inflicted on us change suddenly without notification.

 

[Jeff Schnitzer]

On Thu, Nov 22, 2012 at 10:57 PM, Brandon Wirtz <dra...@digerat.com> wrote:

 

We were trying to do Large jobs with backends, which are supposed to autoscale. They don’t.

Where did you read that?

Jeff 

[Brandon Wirtz]

We get “busy” errors. There is still an open issue on that.

http://stackoverflow.com/questions/12478321/why-are-dynamic-backends-on-google-appengine-not-auto-scaling

http://code.google.com/p/googleappengine/issues/detail?id=8053

 

 

Or do you mean where did we read that they are supposed to autoscale?

https://developers.google.com/appengine/docs/python/backends/overview

--

[alex]

On Fri, Nov 23, 2012 at 6:53 AM, Brandon Wirtz <dra...@digerat.com> wrote:
> We get “busy” errors. There is still an open issue on that.
>
> http://stackoverflow.com/questions/12478321/why-are-dynamic-backends-on-google-appengine-not-auto-scaling
>
> http://code.google.com/p/googleappengine/issues/detail?id=8053

He didn't specify how many instances he wanted to run.
You should probably read this again:
https://developers.google.com/appengine/docs/python/config/backends

"instances: An integer between 1 and 20 indicating the number of
instances to assign to the given backend. Defaults to 1 if
unspecified."

(note the default).

>
>
>
>
>
> Or do you mean where did we read that they are supposed to autoscale?
>
> https://developers.google.com/appengine/docs/python/backends/overview

3rd paragraph:

"Backends do not automatically scale in response to request volume.
Instead, you specify the number of instances of each backend, and
change this number by performing anupdate or configure command."

[Brandon Wirtz]

We have specified 20 with no change. Sometimes it works, then the next week
it won't. Things changed first week in August.

I used to love backends. I'm the guy famous for saying they were magic.

[alex]

We have too and it always works as expected, unless there was a bug in
our code (which happend a few times).

Backends are really a great addition to the platform, and I keep loving them.
There's also Mapreduce/Pipeline - you might wanna check that if
backend isn't solving your problem.

[Brandon Wirtz]

>There's also Mapreduce/Pipeline - you might wanna check that if backend
isn't solving your problem.

We use that some, but that doesn't help with things that take a long time.
If I manage them rather than choosing dynamic we can spin up and down
backends, but if they are user initiated that is a lot harder.
And there are weeks when backends will work as advertised. Then they won't.

[alex]

> We use that some, but that doesn't help with things that take a long time.

You just probably need to shard it in a different way, so that each
chunk requires less time to get processed, i.e. within the limits.
Some other times, for simpler tasks, we employ a little trick by
enqueueing a simple task which would process just the right amount if
data and then re-enqueue itself with a pointer of some kind (often a
datastore cursor) advanced to the next chunk.

> And there are weeks when backends will work as advertised. Then they won't.

You'd have to back that up with more than just words and links to
SO/issue tracker with missing config params or "more info needed"
label :)

[alex]

And at the end of the day, if nothing else works for you, there's also
Compute Engine. Have you tried that?

[alex]

> but if they are user initiated that is a lot harder.

Then, don't do that. Personally, I prefer not to let users initiate
background tasks/backends directly, but to launch it from a frontend
instance. It's just safer that way (both security and load control).
Plus, it's easier to change my underlying processing logic when
there's something in between users and that data processing
implementation of some kind.

[Brandon Wirtz]

> And at the end of the day, if nothing else works for you, there's also
Compute Engine. Have you tried that?

Yeah, the latency between AppEngine and Compute Engine is often too high for
some of the stuff we want, and without access to the shared datastore we
lose too much along the way.

>> And there are weeks when backends will work as advertised. Then they
won't.
>
> You'd have to back that up with more than just words and links to
> SO/issue tracker with missing config params or "more info needed"
> label :)

Have forwarded in the past. Several apps used to run almost exclusively as
backends. Used the backend for NLTK which by default doesn't fit in an F4.
So it ran in an f8 backend. The front end would proxy requests so that we
could have a pretty URL, and that would then hit the backend. One day
magically it stopped auto-scaling. So we moved to static backends. Due to
cost we stripped NLTK down to fit in an F4. Now we don't use NLTK.

Backends are still useful for load testing, but they 503 a lot. Even when
they are working, because you don't have the control you do of a front end,
if you suddenly need 100 requests it fails.

The queue per backend is 10, and only one new instance at a time will spin
up so you only get a queue of 20 durring the time that you spin from 1 -2
and if your spin up time is 1.5s then you can only handle 16-ish request in
the first second. Not real useful if you want to process 100 articles that
will all take 90s to run.

There is not a good way to use "defer" so you can't do retries either,

[Brandon Wirtz]

> Some other times, for simpler tasks, we employ a little trick by
> enqueueing a simple task which would process just the right amount if
> data and then re-enqueue itself with a pointer of some kind (often a
> datastore cursor) advanced to the next chunk.

When Analyzing a large body of text you can't really "chunk" the task.
While chapter marks kind of work, not really.
Same for image processing, we do shape detection and trim large images you
can't "chunk" the task.

[alex]

> When Analyzing a large body of text you can't really "chunk" the task.
> While chapter marks kind of work, not really.

Store intermediate data in memcache/datastore/task post body and re-enqueue.

> Same for image processing, we do shape detection and trim large images you
> can't "chunk" the task.

Agree but this seems like the right job for a dynamic/static backend.
I'd create a separate app to do just that if I needed all 20 backends,
i.e. a lot of load. But, I wouldn't let users launch it directly, but
enqueue from another app. If it 503, temporary store somewhere to try
enqueue again.

[alex]

What I understand from this is "I want this and I want that, with
sugar". Problems that don't seem to be caused by the platform but
because you probably haven't found the right/better solution yet.
That's it. And it seems to have nothing to do with statements like
"backends don't scale".

[alex]

Also, you should really give Go a try. It's amazing how fast you can
process data (including images) comparing to Python or Java. Plus,
your loading requests will be 10x faster. Obviously, RPC calls to App
Engine services will be the same though.

[Brandon Wirtz]

I want things to work the same every day.

-----Original Message-----
From: google-a...@googlegroups.com
[mailto:google-a...@googlegroups.com] On Behalf Of alex
Sent: Friday, November 23, 2012 2:58 AM
To: google-a...@googlegroups.com
Subject: Re: [google-appengine] Re: Update on our GAE Experience

[Brandon Wirtz]

>Also, you should really give Go a try. It's amazing how fast you can
process data (including images) comparing to Python >or Java. Plus, your
loading requests will be 10x faster. Obviously, RPC calls to App Engine
services will be the same though.

We profiled GO, a lot of our most CPU intensive things are C Libraries and
those things are actually faster in Py than Go.
RegEx for example is a LOT faster in Py than Go.

Because GO lacks these banks of highly optimized code it just can't keep up.
The Proxy I wrote in GO was generally faster. And Exploded less often on
Unicode. But when I put in all the regex code and HTML
normalization/compression it was not.

General string manipulation is faster in Go if you are just trying to
shuffle things, but If you are using Regex, Go is slower. If you are doing
NumPy type functions GO is slower. If you are doing LXML/Beautiful Soup type
things GO is slower.

[alex]

:) that's exactly kind of things I often heard from windows users
coming to mac os or linux: "where's that thing I used to have in
Windows? Damn, I can't work like this!"

I already imagined you've tried everything up to a point to consider
you and your team to be experts in every single subject. I thought I'd
waste time in this thread only because I didn't want too many other
people consider your original post to be a "public opinion", because
things are simply not how you describe them. At least for what
concerns the technical part.

Hope you'll find your way to do whatever you guys are up to. Just
please don't compare apples to oranges or try fitting a square into a
circle. At least, not in public :)

[Mani Doraisamy]

Wow! you guys are so fast that this thread looks like a chat :)

In the past, quota limits were based on 2 conflicting objectives:

• discouraging people from building non-scalable applications by restricting APIs & time limits.
  
• discouraging people from using app engine at an "unfair" cost during the request pricing regime.
  

The synthetic limits problem that you mentioned are the result of the request pricing regime. It was frustrating to work-around the pricing constraints instead of constraints based on the best practices for scaling. With the new instance based pricing, these quota limits exist only as a best practice for scaling, IMO.

For example, urlfetch deadline limit is now configurable. Similarly, Stanford NLP works quite well on the java version or you could very well use Alchemy API, depending on how much control you want to have on your language heuristics.

It is sad that you are leaving. But, I guess, it is a collateral damage of being an early adopter of a technology whose business model & technology were figured out in the process.

regards,

mani

[alex]

Personally, I believe these "synthetic" limits are more than just best
practices: http://www.google.com/patents/US20110302243. Background and
Summary sections of the Description are actually pretty interesting.
If I'm not mistaken, that's where Appengine comes from. At least
partially.

Aside from that, take Twitter for instance. Back in the days when they
were running some rails apps, it all started failing the moment more
people started twitting. Everybody were saying, "just throw more
hardware at it", which Twitter tried. Turned out throwing more
hardware at it was either highly cost-ineffective or simply didn't
scale anyway. They ditched rails at the end, and started doing lots of
other stuff to make things scalable.

What I'm saying is, guys like Brandon can move off/to Rackspace, AWS
and what have you, but it's often not a solution - just delaying
something that comes back later. All we do is running huge and complex
Turing machines after all. Well, unless it's a neural network thing
capable of recognizing cats in Youtube videos :)

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/1v6LD9zshRoJ.

[Cesium]

Alex wrote:

What I'm saying is, guys like Brandon can move off/to Rackspace, AWS
and what have you, but it's often not a solution - just delaying
something that comes back later. All we do is running huge and complex
Turing machines after all. Well, unless it's a neural network thing
capable of recognizing cats in Youtube videos :)

 

Please provide links to this cat-recognizing algorithm.

I  would like to find more videos like Maru!

David

[alex]

http://mashable.com/2012/10/09/google-artificial-intelligence/

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-appengine/-/HqvP1Tp0XHwJ.

[Ray]

Sounds like you are the "expert" instead...

Do you mean we REALLY have to try GO even it doesnt seems to fit our need? Using your example, you are asking us to "give linux a try, even it doesnt have the softwares you want!"

[Kristopher Giesing]

On Friday, November 23, 2012 3:18:22 AM UTC-8, alex wrote:

I already imagined you've tried everything up to a point to consider
you and your team to be experts in every single subject. I thought I'd
waste time in this thread only because I didn't want too many other
people consider your original post to be a "public opinion", because
things are simply not how you describe them. At least for what
concerns the technical part.

I haven't heard a credible refutation of anything Brandon wrote in the original post.  My own (admittedly limited) experience is very consistent with Brandon's description.

[hyperflame]

On Friday, November 23, 2012 4:40:07 PM UTC-6, Kristopher Giesing wrote:

I haven't heard a credible refutation of anything Brandon wrote in the original post.  My own (admittedly limited) experience is very consistent with Brandon's description.

+1, to you and to Brandon's list.

I'll add to Brandon's list: the bug tracker. There are issues there that have been outstanding for literally years, and there is no clear roadmap of when issues will be fixed. Quick examples (that I'm picking because I'd really like to see them fixed):

Issue 2314 ( http://code.google.com/p/googleappengine/issues/detail?id=2314 ): Making inbound email work for custom domains. This shouldn't be too hard, custom domains already get hosted Gmail, why can't App Engine access the inbound mail services of Gmail? Yet this issue was filed in 2009 and acknowledged by Google engineers in 2010. It's almost 2013, and no followup. I'd like some details on this; will this be fixed soon, or do I need to buy a subscription to context.io/other external mail services and use their incoming mail parsing services?

Issue 2145 ( http://code.google.com/p/googleappengine/issues/detail?id=2145 ): Same idea, make custom domains work via the XMPP service. Filed in 2009, acknowledged in 2010. No roadmap for implementation. This is pretty much required for using XMPP professionally; Jabber IDs with @appspot.com simply don't look professional.

Issue 739 ( http://code.google.com/p/googleappengine/issues/detail?id=739 ): URLFetch operations don't follow no-caching headers. This issue has been marked Fixed, but I still get cached and stale urlfetches even after setting cache control headers, setUseCaches(), etc. I'm not the only one having problems, just check the issue.

Issue 1741 ( http://code.google.com/p/googleappengine/issues/detail?id=1741 ): The ability to remove a named task from a queue. Filed in 2009, accepted in 2010, absolutely no feedback since then. More fine grained control over task queues in general would be nice.


Leaving issues open for years is insane, give us a roadmap to when you'll fix these, or close them as Won'tFix.

[Jeff Schnitzer]

Hmmm... I was planning to sit this discussion out, but some of this is unfair.

The support-related complaints are certainly legit.  "Never have humans do well what algorithms can do badly" is woven deep into the fabric of Google, no surprise.

The issue list is pretty reasonable.  Stuff does get fixed, but it's a long list.  Please _don't_ close bugs as WontFix unless someone makes an explicit decision not to fix something.  It's unrealistic to expect a roadmap three years out, which is probably how long it would take to fix all the existing issues.

The edges of GAE need to be treaded upon carefully.  Avoid the email, xmpp, and channel apis.  Avoid backends.  The bread-and-butter apis are pretty effective though:  datastore, task queue, memcache, and urlfetch.  Yes, they do sometimes get slow - and you need to understand that the task queue and cron are best-effort, and design appropriately.  This behavior is well-documented, although many people seem to ignore it.

From what I can tell, Brendan has a particularly exotic architecture that involves decomposing his application into lots of appids and relying heavily on backends.  There has been banter on this list before about whether this is a good or bad idea.  One thing for certain is that this is not mainstream app design on GAE... so when you hit snags you may be out there alone.

Jeff

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/SCXgXmZivIMJ.

[Kristopher Giesing]

On Friday, November 23, 2012 11:45:56 PM UTC-8, Jeff Schnitzer wrote:

Hmmm... I was planning to sit this discussion out, but some of this is unfair.

The support-related complaints are certainly legit.  "Never have humans do well what algorithms can do badly" is woven deep into the fabric of Google, no surprise.

The issue list is pretty reasonable.  Stuff does get fixed, but it's a long list.  Please _don't_ close bugs as WontFix unless someone makes an explicit decision not to fix something.  It's unrealistic to expect a roadmap three years out, which is probably how long it would take to fix all the existing issues.

Yeah, not buying that one either, really. However - and this is a big "but" - it drives me crazy how fast bugs get *introduced* in GAE and then not fixed for a long time.  I'm still on 1.6.3 because that's the last release that didn't have a crippling (for me) bug.

 

The edges of GAE need to be treaded upon carefully.  Avoid the email, xmpp, and channel apis.  Avoid backends.  The bread-and-butter apis are pretty effective though:  datastore, task queue, memcache, and urlfetch.

Don't forget Objectify over JDO/JPA.  I don't know why Objectify isn't the recommended API, other than I guess it's technically not Google's.

[Mani Doraisamy]

We have used enterprise support in the past and premier account now. They have been quite responsive. I understand that it costs more. But, it is worth investing if your business needs it. But, it is wrong to accuse that Google does not provide support at all.

For people starting to use app engine, building most of your datamodel on cloud SQL and moving just the high volume models (usually they are 1 or 2 tables only) to bigtable might be a good idea. It reduces learning curve and unknown to just the frontend runtime which is quite stable, IMO.