how to delete a table/entity?

[ajaxer]

thanks.

[Jorge]

In the production GAE, go to the Datastore/Data Viewer. Delete all the
instances. In the development GAE, as far as I know the only way is
rebuilding the application (Clean and Build in NetBeans, for
instance), which will create a new dev datastore.

Jorge Gonzalez


On Nov 27, 1:29 am, ajaxer <calid...@gmail.com> wrote:
> thanks.

[ajaxer]

but there are thousands of instances in the Table.

it is impossible for me to manually delete all of them in a short time
and it is not very reliable to delete them through program.

and the site is open now.
some of the data i won't delete.

[Tim Hoffman]

Hi

You will need to write a process to delete them, or if you can easily
identify all the items (ie you know the keys or a repeated query will
find the one to delete)
you can just use the remote console.

for instance in the console you could

keys_to_delete = mymodel.all(keys_only=True).fetch(1000)
db.delete(keys_to_delete)

Keep doing this until theres none left, unless you have millions of
entities this will go quickly

T

[ajaxer]

yes, i have tried.
but i alway get timeouts.
and the amount is more than millions maybe
I uploading this data by bulk uploading for more than a day.

I think it is very important to be able delete a table in a single act
on the panel

[Gabriel Robertson]

On Sun, Dec 6, 2009 at 7:36 PM, ajaxer <cali...@gmail.com> wrote:
> yes, i have tried.
> but i alway get timeouts.
> and the amount is more than millions maybe
> I uploading this data by bulk uploading for more than a day.
>
> I think it is very important to be able delete a table in a single act
> on the panel

You are thinking about this wrong.
GQL does not have a relational database design, it is an indexed blob
database, there is no concept of "table" as there is in SQL and such.
Instead you create 'kinds', which are more analogous to a message in
Google Protocol Buffers. The things you specify as 'indexes' are
actually just those things pulled from the 'message'. Each index you
specify creates a whole new 'table' with some duplicated data between
all the 'tables'. So for any given kind you will have multiple
'tables' (this is not a SQL like table, this is more like a map/dict,
with ordering and such...). Each index type can also have multiple
kinds in it (such as when you change the format of a kind, you are
creating a new one, but all still indexed together, meaning if you try
to access an old one from your new one, you get a nice exception
thrown, hence why you should always change your kind name when you
change your kind too).

So, a kind can have multiple indexes(tables), and each index(table)
can have multiple kinds.

GQL is not like SQL, you need to relearn things, starting with getting
the concept of tables out of your mind. :)


P.S. The above description is not perfectly accurate as to how it
works, but close enough to get the idea across. :)

[ajaxer]

thanks for the explanation.
but I have no interest to learn such things as big table or something.
the only reason that i keep an eye on this project is it may bring me
convenience in my web development
not that it will bring me some knowledge of science or technology.

before the data manipulation tool becomes good enough, I will only
keep it as a toy.

On Dec 7, 11:00 am, OvermindDL1 <overmind...@gmail.com> wrote:

[Tim Hoffman]

imho I think you do need to understand any new platform to a certain
degree if you really want to take advantage of it.
It is completely different from SQL/RDBMS which means if you don't
change your thinking and adapt to the platform
it "can" only be a toy for you. If you can't get your head around the
fact that its not a rdbms then possibly it is the wrong
tool for whatever you are trying to do. (That doesn't mean app engine
couldn't use a whole range of improvements/features)

For the rest of us its is a very compelling delivery platform for a
range of applications.

Rgds

T

[ajaxer]

but i only need a way to delete an entity by one command or action.
not by a program(and it doesn't always works),
it is certainly not a matter of how it is implemented,
but a feature of the platform.

also 1000 index limit makes it not possible to fetcher older data on
paging.

for if we need an indexed page more than 10000 items,
it would cost us a lot of cpu time to calculate the base for GQL
to fetch the data with index less than 1000.

how can the platform be scalable,
for currently the platform aches with only 10000 items.

[Stephen]

On Dec 15, 8:04 am, ajaxer <calid...@gmail.com> wrote:
>
> also 1000 index limit makes it not possible to fetcher older data on
> paging.
>
> for if we need an indexed page more than 10000 items,
> it would cost us a lot of cpu time to calculate the base for GQL
> to fetch the data with index less than 1000.

http://code.google.com/appengine/articles/paging.html

[ajaxer]

it is too complicated for most of us.
and it still can result in timout if the data is really big

of no much use to most of us if we really have big data to sort and
page.

[Andy Freeman]

> it still can result in timout if the data is really big

How so? If you don't request "too many" items with a page query, it
won't time out. You will run into runtime.DeadlineExceededErrors if
you try to use too many page queries for a given request, but ....

> of no much use to most of us if we really have big data to sort and
> page.

You do know that the sorting for the page queries is done with the
indexing and not user code, right? Query time is independent of the
total amount of data and depends only on the size of the result set.
(Indexing time is constant per inserted/updated entity.)

> >http://code.google.com/appengine/articles/paging.html- Hide quoted text -
>
> - Show quoted text -

[Sandeep Sathaye]

You can check this product we have developed. It takes care of most of the intricacies you have mentioned transparently.

 

http://www.cloud2db.com

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[ajaxer]

of course the time is related to the type data you are fetching by one
query.

if the type of data is larger than 10000 items, you need reindexing
for this result.
and recount each time for getting the proper item.

it seems you have not encountered such a problem.
on this situation, the indexes on the fields helps nothing for the
bulk of data you have to be sorted is really big.

> > >http://code.google.com/appengine/articles/paging.html-Hide quoted text -

[ajaxer]

and do you know that the offset is limited to 1000?

On Dec 17, 12:20 am, Andy Freeman <ana...@earthlink.net> wrote:

> > >http://code.google.com/appengine/articles/paging.html-Hide quoted text -

[Grant Klopper]

I do bulk operations (including delete) using the queue.

Here is some psuedo code

List itemsToDelete = loadOneHundredItems();
for (Item i in itemsToDelete) {
delete(i);
}

//this will stop it when it gets to the end of the queue
if (itemsTodelete.size() > 0) {
QueueFactory.getDefaultQueue().add("/admin/delete-all");
}

[Andy Freeman]

> > if the type of data is larger than 10000 items, you need reindexing
> for this result.
> and recount each time for getting the proper item.

What kind of reindexing are you talking about.

Global reindexing is only required when you change the indices in
app.yaml. It doesn't occur when you add more entities and or have big
entities.

Of course, when you change an entity, it gets reindexed, but that's a
constant cost.

Surely you're not planning to change all your entities fairly often,
are you? (You're going to have problems if you try to maintain
sequence numbers and do insertions, but that doesn't scale anyway.)

> > it seems you have not encountered such a problem.
> on this situation, the indexes on the fields helps nothing for the
> bulk of data you have to be sorted is really big.

Actually I have. I've even done difference and at-least-#
(intersection and union are special cases - at-least-# also handles
majority), at-most-# (binary xor is the only common case that I came
up with), and combinations thereof on paged queries.

Yes, I know that offset is limited to 1000 but that's irrelevant
because the paging scheme under discussion doesn't use offset. It
keeps track of where it is using __key__ and indexed data values.

> > > >http://code.google.com/appengine/articles/paging.html-Hidequoted text -
>
> > > - Show quoted text -- Hide quoted text -

[ajaxer]

it is simple i know.

but what i am concerning is about statistics

to count the different fields for different usage,
it means we must count all the data, get the statistics info at once
query.
and more over, this statistics info may be more than one fields and
have different orders between fields.

for the time being, appengine can not handler this no only that we can
not count an entity but also
we can not count indexed fields.

so for a real and fairly big website which need statistics info to see
the conditions of the site.
how can it be achieved by using appengine?

[ajaxer]

obviously, if you have to page a data set more than 50000 items which
is not ordered by __key__,

you may find that the __key__ is of no use, because the filtered data
is ordered not by key.
but by the fields value, and for that reason you need to loop query as
you may like to do.

but you will encounter a timeout exception before you really finished
the action.

> > > > >http://code.google.com/appengine/articles/paging.html-Hidequotedtext -

[Andy Freeman]

You misunderstand.

If you have an ordering based on one or more indexed properties, you
can page efficiently wrt that ordering, regardless of the number of
data items. (For the purposes of this discussion, __key__ is an
indexed property, but you don't have to use it or can use it just to
break ties.)

If you're fetching a large number of items and sorting so you can find
a contiguous subset, you're doing it wrong.

[ajaxer]

You misunderstand.
if not show me a site with statistics on many fields.
with more than 1000 pages please.
thanks.

[Andy Freeman]

What statistics are you talking about?

You're claiming that one can't page through an entity type without
fetching all instances and sorting them. That claim is wrong because
the order by constraint does exactly that.

For example, suppose that you want to page through by a date/time
field named "datetime". The query for the first page uses order by
datetime while queries for subsequent pages have a "datetime <="
clause for the last datetime value from the previous page and continue
to order by datetime.

What part of that do you think doesn't work?

Do you think that Nick was wrong when he said that time time to
execute such query depends on the number of entities?

You can even do random access by using markers that are added/
maintained by a sequential process like the above.

[ajaxer]

when even with the "datetime <=" you still get a big set, how you can
handle it?
for example you get 10000 item with the most specific filtering sql.
and on this filtering sql, you should have a statistic info. like how
many item it is .

how do you expect the appengine to handle this problem?
how about at one request with many these actions?

[Andy Freeman]

Any application requires fetching an unbounded amount of data for a
single page view is not scalable, no matter what technology you use to
build it, so this problem is not appengine specific.

If you need aggregations (average, median, total, etc), you have to
compute them incrementally or with an off-line process.

> when even with the "datetime <=" you still get a big set, how you can
> handle it?

We're talking about paging through a dataset, presenting n (for small
n) elements at a time to a user.

If we're paging through by the value of field with distinct values and
we want to present 20 results per page, the query for the first page
is "order by field" with limit 20. That query has a "last" result.
The query for the next page is "field > {last result's field value}
order by field", again with limit 20. That query also has a last
result so the form of subsequent queries should be obvious. (If
you've got other conditions, such as user id or key, you need to add
those as well.)

Suppose that entities can have the same field value. If you don't
care how those entities are ordered, the first query's order by clause
can be "order by field, __key__", again limit 20. The next query
tries to pick up entities with the same field as the last result from
the previous query. It looks like "field = {last result's field's
value} and __key__ > {last result's key} order by __key__" and you
keep using it until it fails. You then use a query like the "next
page" query from the previous case. (I stopped mentioning limit
because the value depends on what you need to fill the current page.)

[ajaxer]

your method is right but the limit is there what ever methods we use.

and you didn't get what i mean.

because the offset is limited to 1000.
i can not sort data by fields in results more than some limited items

with out the offset limit, we can do it easily.

[Andy Freeman]

> > your method is right but the limit is there what ever methods we use.

There's always going to be a limit for scalable applications -
appengine just exposes it.

> because the offset is limited to 1000.
> i can not sort data by fields in results more than some limited items

Don't sort. Use indices. They can handle multiple fields.

Indices are the only way to build scalable applications.