Over Quota Errors last night

[Joshua Smith]

One of my apps threw a few hundred “Over Quota” errors last night.

However, the app has billing enabled, I’m not aware of any quotas being enforced, and the bills show negligible usage. Was there some kind of an outage last night that google didn’t report? So far all I can find in the logs are the stack traces of the specific requests that were denied, but I can’t figure out why google thought my app was over quota. However, the new console makes finding pretty much anything pretty much impossible, so it’s entirely possible the information I need is there and I just can’t find it.

Advice on how to proceed?

-Joshua

[Nicholas (Google Cloud Support)]

According to the Status Dashboard, there were no known issues regarding quotas yesterday at all.

As shown here, various quotas may be encountered by a given application.  Even with billing enabled, one can still encounter OverQuotaErrors by reaching spending limits or per-minute safety limits.  If quota error are encountered today, you can view them in the Developers Console by going to the App Engine dashboard and selecting Quotas.  If encountered prior to today, you can select VIEW USAGE HISTORY.  Here, you can view many months worth of daily usage reports.  Though it won't reveal usage over quotas, you can see from the previous page how much usage is considered over quota and compare that way.

In addition, the errors encountering should be searchable in your logs.  The article When a Resource is Depleted demonstrates how to catch such exceptions at runtime.  This can be used in conjunction with admin emails to be made aware of such occurrences as soon as they happen in the future.

Hope this helps!

[Joshua Smith]

I’ve tracked down the thing that threw the first over quota exception. It was a nightly cron that does a lot of data roll-ups. But I don’t understand why that would throw a quota exception. I have no spending limits set. You mention per-minute safety limits. What are those? The specific quota is probably datastore reads.

My app basically does the same thing every night. Why would this suddenly be too much?

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/78f541d0-5ad9-4952-91d3-e0b9c9ed023f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Joshua Smith]

Okay, I figured it out.

It’s that old datastore bug where it throws timeout exceptions when you try to iterate on a .all() with a lot of items.

I’m using the workaround that I only iterate over some of them, then I set a cursor, do the .all() again, move forward to the cursor, and then keep going.

Usually that works, but sometimes it will still throw a timeout exception, so I have it in a task so it gets retried.

Something must have changed in the datastore, so the workaround was consistently NOT working around the problem. So the task ran over and over, eventually hitting some undocumented quota limit.

I just lowered the number of items I’ll go through, and that seems to have gotten the workaround working again.

-Joshua

[Nicholas (Google Cloud Support)]

You mention that this is an old Datastore bug.  Is this a known issue on the App Engine public issue tracker?  If so, which issue is it?  Linking to it here may be helpful for the community.

On Tuesday, June 14, 2016 at 9:32:27 AM UTC-4, Joshua Smith wrote:

[Joshua Smith]

Beats me. Here’s the thread where PK gave me the fix:

https://groups.google.com/forum/#!topic/google-appengine/CGaYQhz4V3k

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/57afa0b8-595e-44a8-a4bc-35f7b9b4a617%40googlegroups.com.