"Quota exceeded"... although it doesn't seem to be

[ShriJ]

Hi,

I've been having a very hard time since the last few days w.r.t.
deleting existing data and uploading new data in my datastore.

For deleting: I get a row using the key_name and then delete it.
dbRow = modelObject.get_by_key_name(key_name)
if dbRow:
dbRow.delete()
My datastore has roughly 60K rows and I delete them one by one. Many
times when doing this, I get a quota exceeded exception. But when I
login to appspot.com for seeing whats wrong, I don't see any error.

Similarly, when uploading data, I am using the bulkload_client.py
script for uploading the data. There again I see the same issues.
There are times when after inserting 500 rows, I get the error and
thereafter no matter how many times I start again, I see the same
error.

I've almost come to the conclusion that downloading and uploading the
data on the same day is kinda impossible.

Am I doing something fundamentally incorrect? If not, is there any ETA
on when will things get better. My website has been down since the
last 4 days because of these issues :(.

-Shri

[Nick Johnson (Google)]

Hi ShriJ,

On Wed, Sep 2, 2009 at 2:44 AM, ShriJ <shri...@gmail.com> wrote:

Hi,

I've been having a very hard time since the last few days w.r.t.
deleting existing data and uploading new data in my datastore.

For deleting: I get a row using the key_name and then delete it.
       dbRow = modelObject.get_by_key_name(key_name)
       if dbRow:
         dbRow.delete()
My datastore has roughly 60K rows and I delete them one by one. Many
times when doing this, I get a quota exceeded exception. But when I
login to appspot.com for seeing whats wrong, I don't see any error.

It's possible to run out of short-term quota without exhausting your daily quota. This limit exists in order to prevent a burst of traffic taking your site offline for the rest of the day.

 

Is there a reason you're not deleting records in batches? This will be much easier on your quotas.

Similarly, when uploading data, I am using the bulkload_client.py
script for uploading the data. There again I see the same issues.
There are times when after inserting 500 rows, I get the error and
thereafter no matter how many times I start again, I see the same
error.

You need to ratelimit the upload to a slower speed in order to not exhaust your short-term quotas.

-Nick Johnson

 

I've almost come to the conclusion that downloading and uploading the
data on the same day is kinda impossible.

Am I doing something fundamentally incorrect? If not, is there any ETA
on when will things get better. My website has been down since the
last 4 days because of these issues :(.

-Shri

--
Nick Johnson, Developer Programs Engineer, App Engine

[Shrinand Javadekar]

> You need to ratelimit the upload to a slower speed in order to not exhaust
> your short-term quotas.
>
> -Nick Johnson

Thanks a ton Nick. I do delete the data in batches. The problem is not
so much with delete. I can always restart the deletion process and
clear out all the data. The problem is with uploading.

I tried uploading in batches of 100 (lines in the csv file). Then with
50. None of them worked. How do I rate-limit the upload to a slower
speed?

Also, I think the appcfg.py with the upload data option tackles this
problem of restarting the upload process from wherever it crashes
last. I am getting some errors in that too. Can you help me with that?
http://groups.google.com/group/google-appengine/browse_thread/thread/795f1030500440f0#

Thanks in advance.
-Shri

[Nick Johnson (Google)]

Hi Shrinand,

The bulk uploader lets you set a rate at which you want to upload your data - see the command line reference in the documentation: http://code.google.com/appengine/docs/python/tools/uploadingdata.html#Command_Line_Arguments

-Nick Johnson