in order to do a quick performance test, I uploaded and tested a little app creating around 2000 objects in the datastore, now I want to remove them, ideally, without accessing the datastore viewer and selecting page by page and pressing the "remove" button. Anybody knows if it's possible to access some kind of "clear_datastore" functionality via the web dashboard (I have not seen it) or via the deployment client? I doubt it exists, but just in case...
There's currently no simple way to do this (I wish there was).
However, you can use the bulk loader client also as a bulk deleter --
I wrote an article at http://www.wooji-juice.com/blog/appengine-bulkloader.html explaining how. Note, it may not be fast or reliable (in my
experience, it was very slow and unreliable, but that may have been
due to other factors, eg indexing).
On Jun 30, 9:32 pm, "José Oliver Segura" <primi...@gmail.com> wrote:
> in order to do a quick performance test, I uploaded and tested a
> little app creating around 2000 objects in the datastore, now I want
> to remove them, ideally, without accessing the datastore viewer and
> selecting page by page and pressing the "remove" button. Anybody knows
> if it's possible to access some kind of "clear_datastore"
> functionality via the web dashboard (I have not seen it) or via the
> deployment client? I doubt it exists, but just in case...
On Tue, Jul 1, 2008 at 12:11 AM, Canis <wooji.ju...@googlemail.com> wrote:
> There's currently no simple way to do this (I wish there was).
> However, you can use the bulk loader client also as a bulk deleter -- > I wrote an article at http://www.wooji-juice.com/blog/appengine-bulkloader.html > explaining how. Note, it may not be fast or reliable (in my > experience, it was very slow and unreliable, but that may have been > due to other factors, eg indexing).
Hi Canis,
thanks for the link, it looks like a pretty good research job :) At first sight it looks like is not trivial to do it, and I think that my way by using the bulkloader could be more or less the same than doing it via REST api (which my app supports also). I'll take a closer look to your article and I'll then decide what way to use... obviously, not the one with the "clear datastore" button, since it doesn't exist :)
class Purge(webapp.RequestHandler): def get(self): limit = self.request.get("limit") if not limit: limit = 10 table = self.request.get('table') if not table: table = 'DefaultTable' exec('q = '+table+'.all()') [db.delete(result) for result in q.fetch(int(limit))]
HTH, --Doug
On Tue, Jul 1, 2008 at 5:07 AM, José Oliver Segura <primi...@gmail.com> wrote:
> On Tue, Jul 1, 2008 at 12:11 AM, Canis <wooji.ju...@googlemail.com> wrote:
> > There's currently no simple way to do this (I wish there was).
> > However, you can use the bulk loader client also as a bulk deleter -- > > I wrote an article at > http://www.wooji-juice.com/blog/appengine-bulkloader.html > > explaining how. Note, it may not be fast or reliable (in my > > experience, it was very slow and unreliable, but that may have been > > due to other factors, eg indexing).
> Hi Canis,
> thanks for the link, it looks like a pretty good research job > :) At first sight it looks like is not trivial to do it, and I think > that my way by using the bulkloader could be more or less the same > than doing it via REST api (which my app supports also). I'll take a > closer look to your article and I'll then decide what way to use... > obviously, not the one with the "clear datastore" button, since it > doesn't exist :)
go to datastore viewer and modify limit= parameter in url field in
browser, if you have 2000 objects then specifying limit=1000 will give
you only 2 pages so you'll have to press select all and delete only
twice
On Jul 1, 12:32 am, "José Oliver Segura" <primi...@gmail.com> wrote:
> in order to do a quick performance test, I uploaded and tested a
> little app creating around 2000 objects in the datastore, now I want
> to remove them, ideally, without accessing the datastore viewer and
> selecting page by page and pressing the "remove" button. Anybody knows
> if it's possible to access some kind of "clear_datastore"
> functionality via the web dashboard (I have not seen it) or via the
> deployment client? I doubt it exists, but just in case...
On Tue, Jul 1, 2008 at 1:10 PM, pronvit <pron...@gmail.com> wrote:
> go to datastore viewer and modify limit= parameter in url field in > browser, if you have 2000 objects then specifying limit=1000 will give > you only 2 pages so you'll have to press select all and delete only > twice
Nice try ;) but datastore viewer complaints with limits like 1000 or 500 with:
"There were errors:
* Limit
",
it works with 100, but 200 gives a server error. I think the only way would be bul/self-api way...
Has anyone put in a feature request for this? Deleting a few thousand
entities is painful! And I've already had to do it many times. Is it
possible to just remove an entire entity class?
- Charlie.
On Jul 1, 9:51 am, "José Oliver Segura" <primi...@gmail.com> wrote:
> On Tue, Jul 1, 2008 at 1:10 PM, pronvit <pron...@gmail.com> wrote:
> > go to datastore viewer and modify limit= parameter in url field in
> > browser, if you have 2000 objects then specifying limit=1000 will give
> > you only 2 pages so you'll have to press select all and delete only
> > twice
> Nice try ;) but datastore viewer complaints with limits like 1000 or 500 with:
> "There were errors:
> * Limit
> ",
> it works with 100, but 200 gives a server error. I think the only way
> would be bul/self-api way...
Yeah...i found out the limits the hard way. I wrote the following
function to delete a large table and it worked great for smalling
tables but I shut down my application quickly when I tried doing any
more than that.
def deleteAllEntities(table):
q = db.GqlQuery("SELECT * FROM "+table)
results = q.fetch(100)
while len(results) > 0:
for result in results:
result.delete()
q = db.GqlQuery("SELECT * FROM "+table)
results = q.fetch(100)
On Jul 23, 10:59 am, Charlie <schmi...@gmail.com> wrote:
> Has anyone put in a feature request for this? Deleting a few thousand
> entities is painful! And I've already had to do it many times. Is it
> possible to just remove an entire entity class?
> - Charlie.
> On Jul 1, 9:51 am, "José Oliver Segura" <primi...@gmail.com> wrote:
> > On Tue, Jul 1, 2008 at 1:10 PM, pronvit <pron...@gmail.com> wrote:
> > > go todatastoreviewer and modify limit= parameter in url field in
> > > browser, if you have 2000 objects then specifying limit=1000 will give
> > > you only 2 pages so you'll have to press select all anddeleteonly
> > > twice
> > Nice try ;) butdatastoreviewer complaints with limits like 1000 or 500 with:
> > "There were errors:
> > * Limit
> > ",
> > it works with 100, but 200 gives a server error. I think the only way
> > would be bul/self-api way...
i did something similar. wrote a /delete handler which would just loop
and fetch(100) objects and delete.
a real pain though since the app engine frequently complains about
"over limit"...so i wrapped a curl request to this url in a bash while
loop script that just hammers away while i get lunch deleteing 1000's
of entities.
there must be a better way...
rich
On Aug 9, 8:59 pm, abwaters <abry...@gmail.com> wrote:
> Yeah...i found out the limits the hard way. I wrote the following
> function to delete a large table and it worked great for smalling
> tables but I shut down my application quickly when I tried doing any
> more than that.
> def deleteAllEntities(table):
> q = db.GqlQuery("SELECT * FROM "+table)
> results = q.fetch(100)
> while len(results) > 0:
> for result in results:
> result.delete()
> q = db.GqlQuery("SELECT * FROM "+table)
> results = q.fetch(100)
> On Jul 23, 10:59 am, Charlie <schmi...@gmail.com> wrote:
> > Has anyone put in a feature request for this? Deleting a few thousand
> > entities is painful! And I've already had to do it many times. Is it
> > possible to just remove an entire entity class?
> > - Charlie.
> > On Jul 1, 9:51 am, "José Oliver Segura" <primi...@gmail.com> wrote:
> > > On Tue, Jul 1, 2008 at 1:10 PM, pronvit <pron...@gmail.com> wrote:
> > > > go todatastoreviewer and modify limit= parameter in url field in
> > > > browser, if you have 2000 objects then specifying limit=1000 will give
> > > > you only 2 pages so you'll have to press select all anddeleteonly
> > > > twice
> > > Nice try ;) butdatastoreviewer complaints with limits like 1000 or 500 with:
> > > "There were errors:
> > > * Limit
> > > ",
> > > it works with 100, but 200 gives a server error. I think the only way
> > > would be bul/self-api way...
|
