clear *production* datastore

[José Oliver Segura]

hi all,

in order to do a quick performance test, I uploaded and tested a
little app creating around 2000 objects in the datastore, now I want
to remove them, ideally, without accessing the datastore viewer and
selecting page by page and pressing the "remove" button. Anybody knows
if it's possible to access some kind of "clear_datastore"
functionality via the web dashboard (I have not seen it) or via the
deployment client? I doubt it exists, but just in case...

Best,
Jose

[Canis]

There's currently no simple way to do this (I wish there was).

However, you can use the bulk loader client also as a bulk deleter --
I wrote an article at http://www.wooji-juice.com/blog/appengine-bulkloader.html
explaining how. Note, it may not be fast or reliable (in my
experience, it was very slow and unreliable, but that may have been
due to other factors, eg indexing).

[José Oliver Segura]

On Tue, Jul 1, 2008 at 12:11 AM, Canis <wooji...@googlemail.com> wrote:
>
> There's currently no simple way to do this (I wish there was).
>
> However, you can use the bulk loader client also as a bulk deleter --
> I wrote an article at http://www.wooji-juice.com/blog/appengine-bulkloader.html
> explaining how. Note, it may not be fast or reliable (in my
> experience, it was very slow and unreliable, but that may have been
> due to other factors, eg indexing).

Hi Canis,

thanks for the link, it looks like a pretty good research job
:) At first sight it looks like is not trivial to do it, and I think
that my way by using the bulkloader could be more or less the same
than doing it via REST api (which my app supports also). I'll take a
closer look to your article and I'll then decide what way to use...
obviously, not the one with the "clear datastore" button, since it
doesn't exist :)

thanks
Jose

[Doug Chestnut]

Hi Jose,
I use this (and while true; do wget http://myapp.appspot.com/purge?table=testtable&limit=50 -O /dev/null; sleep 10; done; )

class Purge(webapp.RequestHandler):
    def get(self):
        limit = self.request.get("limit")
        if not limit:
            limit = 10
        table = self.request.get('table')
        if not table:
            table = 'DefaultTable'
        exec('q = '+table+'.all()')
        [db.delete(result) for result in q.fetch(int(limit))]
 

HTH,
--Doug

[pronvit]

go to datastore viewer and modify limit= parameter in url field in
browser, if you have 2000 objects then specifying limit=1000 will give
you only 2 pages so you'll have to press select all and delete only
twice

[José Oliver Segura]

On Tue, Jul 1, 2008 at 1:10 PM, pronvit <pro...@gmail.com> wrote:
>
> go to datastore viewer and modify limit= parameter in url field in
> browser, if you have 2000 objects then specifying limit=1000 will give
> you only 2 pages so you'll have to press select all and delete only
> twice

Nice try ;) but datastore viewer complaints with limits like 1000 or 500 with:

"There were errors:

* Limit

",

it works with 100, but 200 gives a server error. I think the only way
would be bul/self-api way...

Thanks anyway!!!

Best,
Jose

[Charlie]

Has anyone put in a feature request for this? Deleting a few thousand
entities is painful! And I've already had to do it many times. Is it
possible to just remove an entire entity class?

- Charlie.

On Jul 1, 9:51 am, "José Oliver Segura" <primi...@gmail.com> wrote:

[Bryan Waters]

Yeah...i found out the limits the hard way. I wrote the following
function to delete a large table and it worked great for smalling
tables but I shut down my application quickly when I tried doing any
more than that.

def deleteAllEntities(table):
q = db.GqlQuery("SELECT * FROM "+table)
results = q.fetch(100)
while len(results) > 0:
for result in results:
result.delete()
q = db.GqlQuery("SELECT * FROM "+table)
results = q.fetch(100)

On Jul 23, 10:59 am, Charlie <schmi...@gmail.com> wrote:
> Has anyone put in a feature request for this?  Deleting a few thousand
> entities is painful!  And I've already had to do it many times.  Is it
> possible to just remove an entire entity class?
>
> - Charlie.
>
> On Jul 1, 9:51 am, "José Oliver Segura" <primi...@gmail.com> wrote:
>
>
>
> > On Tue, Jul 1, 2008 at 1:10 PM, pronvit <pron...@gmail.com> wrote:
>

> > > go todatastoreviewer and modify limit= parameter in url field in

> > > browser, if you have 2000 objects then specifying limit=1000 will give
> > > you only 2 pages so you'll have to press select all anddeleteonly
> > > twice
>

> > Nice try ;) butdatastoreviewer complaints with limits like 1000 or 500 with:

>
> > "There were errors:
>
> >     * Limit
>
> > ",
>
> > it works with 100, but 200 gives a server error. I think the only way
> > would be bul/self-api way...
>
> > Thanks anyway!!!
>
> > Best,

> > Jose- Hide quoted text -
>
> - Show quoted text -

[thedude]

i did something similar. wrote a /delete handler which would just loop
and fetch(100) objects and delete.

a real pain though since the app engine frequently complains about
"over limit"...so i wrapped a curl request to this url in a bash while
loop script that just hammers away while i get lunch deleteing 1000's
of entities.

there must be a better way...

rich

[Jeff Schnitzer]

Wow, revived a 4-year old thread :-)

For future readers: There is a Datastore Admin page in the admin
console which lets you delete all entities of particular kinds in the
datastore. But if you only have a couple thousand entities, it's
still easier to do the limit trick.

Jeff

On Tue, Sep 11, 2012 at 12:15 AM, Jem <jse...@gmail.com> wrote:
> Fantastic! I just deleted 192 entries in one swoop.

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/Xw6N8RJizz8J.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

[vlad]

Beware that deleting data is expensive. Essentually you should factor data deletion into your cost calculations. Think twice of what data you want to store, especially if you are planning to delete it eventually. From my own experience: I have an app with 5-6Gb of data accumulated. Storing it costs me 5c/day, yet deleting it in bulk would cost me ~$50.