Datastore Read Operations Quota Whacked-Out?

[Brian]

The free-tier level appspot deployment allows 50,000 read operations within a 24 hour period. Somehow, during the last two days I've exceeded this quota within 5 minutes using the remote api. Has anyone else experienced this problem?

My python-based remote api job queries my appspot datastore, finds about 60 objects, then I visit each one to check the value of a few properties before capturing the object's key and writing it to a local file on my PC for later use. I can not imagine how this small job can possibly be counted as 50K read ops. Earlier in the week this job was consuming only 3% of my daily quota. What's going on here?

Has anyone else seen this problem within the last couple of days, or is it just me?

[Brian]

Okay, so with a fresh unused (0%) quota today I tried running my remote api job again, but first I cut the size of the task down to 1/3 of what it was previously. On the first run, it consumed 1% of my daily read ops quota. Encouraged, I ran another batch and within moments it consumed all 100% of my quota again! What's up with that?! I can not trim my job down to any smaller chunks than this -- what can I do now to workaround this problem?

[Thiago Catoto]

Hi,

I remember some one talking about remote api over-consuming cause of his "retry", so if you are getting some timeout and stuffs like that, the remote api will try again and will consume your quota both times (the success and the timeout).

Maybe this is your case... not sure if my information is correct too... are you using Java? If so, your should try using python bulk upload (https://developers.google.com/appengine/docs/python/tools/uploadingdata) if it fit in your needs.

Hope it helps,

cheers.

On Sat, Oct 20, 2012 at 10:33 AM, Brian <blru...@gmail.com> wrote:

Okay, so with a fresh unused (0%) quota today I tried running my remote api job again, but first I cut the size of the task down to 1/3 of what it was previously. On the first run, it consumed 1% of my daily read ops quota. Encouraged, I ran another batch and within moments it consumed all 100% of my quota again! What's up with that?! I can not trim my job down to any smaller chunks than this -- what can I do now to workaround this problem?

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/vbz4pmqDfQUJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Brian]

Hi Thiago, thanks for the suggestions.

I'm using the python remote_api to just query my datastore, I'm not uploading any data at this point. The query fails at myquery.fetch(100) with the error: "OverQuotaError: The API call datastore_v3.RunQuery() required more quota than is available." The query has several filters. (I wonder if filters on a query consumes more read quota?)

As far as I know I'm not getting any timeouts -- would it display as such on the command line (it's not now) or how would I check that?

I'll try again tomorrow when I have some quota available again, and will cut the fetch down to 10 objects to see if that yields any more clues...

If anyone else has any suggestions for this issue, please let me know!

[Brian]

Okay, now this is very strange indeed. Armed with a fresh 50K read quota today, I adjusted my query's fetch to 10 objects each. To examine all the matching objects it had to fetch 6 times to get all ~60 objects. This operation consumed 0% of my read quota!!! While it's good news, it's also very strange and leaves me with several questions:

1) Why didn't the built-in default value of .fetch(20) work? It would have run the query only 3 times, yet it consumed 100% of my read quota.

2) Why didn't the value of .fetch(100) work? It would have run the query only once, yet it consumed 100% of my read quota.

3) Why does the value of .fetch(10) consume no discernible read quota?

This seems to be very mysterious behavior, not to mention frustrating and time consuming.

Can anyone shed any light on these inconsistent results?

[Brian]

This gets weirder by the day...

I used remote api to .fetch(10) objects. After 5 1/2 hours the query failed due to OverQuotaError. Really? After 5 /12 hours it can not find 10 objects to complete the query?

This same query returns within a few short minutes on my local dev_server!

What the heck?!

[Brian]

I changed my query to .fetch(1) object, after waiting 37 minutes with no results I killed the request.

(Keep in mind that that my app works great on my local dev_server, but getting my app uploaded to Datastore has taken several weeks now, with no end in sight based on all the problems I keep running into!)

I decided to try a similar query against a different KIND of object, and the query returned immediately with the objects it found. (In this case, success, but I still need to process my other kind of objects yet.)

Has anyone else had this problem -- where fetching a specific kind of object fails to find even the first result? Is it a bug with Datastore? Has Datastore corrupted my objects somehow? Is there someway to determine this? Are there deeper logs available than what's displayed on the dashboard? Any suggestions anyone?

[Brian]

I think I have narrowed the problem down to differences in query behavior between the local dev SDK and production appspot. (FYI -- I have used SDKs starting at 1.3.1 through 1.7.2.) Here's what I've found: 

Via the local dev_server, if you execute a python api query using a filter for a string property set to an empty string, and the local datastore does not contain a match (because they have string values with lengths > 0) then the query will return with no results, just as one would expect. 

However, if you execute the same query via production appspot on an exact copy of the data, then one of the following results will occur:

1) The query will never finish and after running for several hours will consume all the datastore read quota until it crashes due to a lack of quota. Typically, the error message will be: 

google.appengine.runtime.apiproxy_errors.OverQuotaError: The API call datastore_v3.RunQuery() required more quota than is available.

2) The query will crash due to a time-out error. Typically, the error message will be:

urllib2.URLError: <urlopen error (10060, 'Operation timed out')>

If anyone else runs into this problem, here's the solution that (so-far) is working for me:

Change the query filter from = '' (empty string) to < ' ' (space character). Then order (sort) the query results by that same property. Since using an inequality in the filter will not work with the standard (hidden) entity indexes, you will need to build custom indexes.

If anyone knows of a better solution, let me know!