Huge number of datastore reads?

[Dave Peck]

My production application went over its daily budget today. Let's just
say that my daily budget is roughly 3x what I've ever actually needed
on my busiest day. Today was not my busiest day by a long shot.

It appears that I serviced a small number of requests today. However,
the dashboard claims that I performed roughly 10,000x the number of
datastore read ops as requests, at a cost of
$LOTS_OF_MONEY_FOR_ONE_DAY

This seems like nonsense. My code hasn't changed in a while. I've
never reached a quota quite like this. And there's no way my average
request requires 10,000 read ops. Just no.

Google team -- is there someone I can speak with? I'd like to
understand in detail what happened and how to prevent it going
forward.

-Dave

[Alfred Fuller]

What is your app id?

Did you perform a lot of queries using remote api?

-Dave

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Greg]

I've noticed a couple of blips in memcache today - if you cache using
memcache, maybe you're having to revert to datastore more often than
usual?

Cheers
Greg.

[Brandon Wirtz]

> The worst of it: we looked like rank amateurs in several ways, but perhaps

I'd say "no offense" but I'd be lying.

"More than doubled"

My Daily budget sits at 50x my biggest recorded day on almost every app I
have. If your model is to make money, and you are pretty sure you don't
have a bug that is going to cost you millions of dollars. There is no reason
to not set your budget sky high. If you have only 30 minutes to get your app
back with in quota, you are rank amateurs. Your Scale is only limited by
your budget. If you aren't going to set you budget to as large as you
imagine you would want to scale you are doing it wrong.

If you are going to run with the training wheels on, don't be surprised when
you hit a pot hole and your back tire just spins with you going nowhere.

-Brandon

[Dave Peck]

Although I was hoping for useful, non-judgmental replies too.

-Dave

[Jeff Schnitzer]

On Wed, Mar 7, 2012 at 1:08 AM, Brandon Wirtz <dra...@digerat.com> wrote:

> The worst of it: we looked like rank amateurs in several ways, but perhaps

If you are going to run with the training wheels on, don't be surprised when
you hit a pot hole and your back tire just spins with you going nowhere.

Sorry Brandon, you're way off-base here.  There's no excuse for buried land mines like this to exist.

Nobody realizes that you get frozen out of the system for 30+ minutes at a time until you actually get screwed by the system.  And no matter what you think you've set your budget to, something can come along and blow it.

For example, every night I process a significant volume of CSV data downloaded from a technically unsavvy organization.  They seem to delight in finding new and interesting ways to break my (popular third-party) CSV parser.  The latest was a name field containing "Blah Blah, Jr" - you can guess what that did.  After a few rounds of this I have hacked my parser enough to consume just about any garbage they can feed me.  Nevertheless, there were quite a few nights early on where this error caused a large task-queue-driven data processing operation to fail.  With the default retry schedule, it quickly ate through my (seemed-to-be-generous-at-the-time) billing quota.

Fortunately my technically unsavvy organization is sufficiently clumsy that these problems came up early.  If they were only slightly more clever, these issues might not surface until long into production, when it really mattered.

As those of us in Javaland were reminded today, GAE fails us every now and then.  It needs to fail gracefully.  The default over-quota message is WAY more embarrassing than "XYZ is experiencing technical difficulties" - how can we override it?

If...you are pretty sure you don't have a bug that is going to cost you millions of dollars...

...then you are delusional :-)

Jeff

[Mos]

Hi Jeff,

+1  on all your statements.

> The default over-quota message is WAY more embarrassing than "XYZ is experiencing technical difficulties" - how can we override it?

Should be possible:
http://code.google.com/appengine/docs/java/config/appconfig.html#Custom_Error_Responses

My application had not any overquota yet, thus can't tell if it works.

Cheers
Mos

-----------
http://www.mosbase.com

--

[Jeff Schnitzer]

On Wed, Mar 7, 2012 at 4:09 AM, Mos <mos...@googlemail.com> wrote:

> The default over-quota message is WAY more embarrassing than "XYZ is experiencing technical difficulties" - how can we override it?

Should be possible:
http://code.google.com/appengine/docs/java/config/appconfig.html#Custom_Error_Responses

My application had not any overquota yet, thus can't tell if it works.

That's great!  I guess I need to check for doc updates more often... I don't remember that being there. 

Thanks,

Jeff

[Dave Peck]

> Should be possible:http://code.google.com/appengine/docs/java/
config/appconfig.html#Cust...

Ah, nice feature. Thanks; curious when this was added.

Cheers,
Dave

[Brandon Wirtz]

I don’t disagree that the Payment Page is horribly broken.  And Google Wallet is horrible, and a “notify me if quota is estimated to be reached in 6 hours” wouldn’t be an awesome feature.

But like your CSV input. We’d put that through a validator before it went “live” and that validator might only be another app that says “Hey did this error” but nothing changes in a Live deployment without humans. For small things… we’d hope that 50x would still do us.

 

If you don’t want to be down have a high budget. If you don’t want to wake up broke one morning because of a bug in your code, set the budget low. 

Assume you can only change your budget between 9 and 5 EST on week days.  Which sucks, but I make the same assumption with a number of my clouds If for no other reason than every so often the limiting factor is I need to call a Credit card company to say “Hey I really do want to give Amazon more than $3000 today. Yes I realize I gave them $2500 every day this week, but today I want to give them $6000.”

 

--

[Chris Ramsdale]

Dave,

Sorry that you had issues with Checkout. We continue to work with them on improvements that they should make to the service (and I've forward this thread on to them).

Setting your budget higher is probably not a bad idea, but totally up to you. I think the pros and cons have been outlined (in one way or another) here :-)

Did you figure out why the DS reads increased? Is this still an issue?

-- Chris

Product Manager, Google App Engine

[Dave Peck]

Hi Chris,

Thanks!

> Sorry that you had issues with Checkout. We continue to work with them on
> improvements that they should make to the service (and I've forward this
> thread on to them).

Good to hear. If I can provide further details, just let me know. In
the final analysis, it seemed to us that the billing update issued
from my iPhone worked... but it didn't. More accurate feedback would
have been helpful.

> Did you figure out why the DS reads increased? Is this still an issue?

The operating theory at the moment is that a change to how our back-
end services report back to App Engine triggered a latent (and really
nasty) performance bug. We're still tracking down the details now;
Alfred has been immensely helpful offline!

> Setting your budget higher is probably not a bad idea, but totally up to
> you. I think the pros and cons have been outlined (in one way or another)
> here :-)

Indeed they were. ;-)

Cheers,
Dave

[Alexander Botov]

Experiencing same problem, 5 minutes normal usage of the application
adds up about 10K datastore reads, is it related to 1.6.3 upgrade?
remote_api is not used ...

Regards,
--alex

On Mar 7, 2:53 am, Alfred Fuller <arfuller+appeng...@google.com>
wrote:

[Alexander Botov]

Hi Chris, 

I would like to add to this thread that we definitely experience increase of DS reads from couple of days without any significant change in the code. We just upgraded to HRD and we still have the issue. App id is spendgatel. 

Best,

--alex

-- Chris

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

[Brian]

Dave, did you get any resolution to this issue? I seem to be having a similar issue -- 5 minutes of read ops on a few objects via remote api consumes my free daily quota of 50K.