Datastore Stored Data Quota Report Wrong

[James Gilliam]

My Datastore Stored Data is reporting .23 Gb (230 megs). Yet, the
Admin Datastore Status says size of total entities is 73 megs. I
first noticed this disparty about a week ago and it has not changed
much in the week. About a 3 to 1 ratio between what the quota is
reporting and what the admin datastore stat is reporting.

Since this is under the free quota I am not being charged for this but
I expect my data to increase rather dramatically soon and would like
to get this sorted out before then.

My conclusion, is the Datastore Store Data in the quota report is
wrong.

Any Ideas?

[Simon Knott]

The additional storage is being used by your indexed-property indexes, along with any custom indexes.

Cheers,
Simon

[James Gilliam]

That seems shocking. Twice as much storage for indexes as the data.
The data includes sizable statistics stuff as well. Plus, I am very
careful on the number on indexed fields and only have 1 compound
index.

[Max]

by default every property is indexed except Blob / Text, and these indexes are not listed on your index page. Try to unindex them and see how much space you can save. 

With regards to your only composite index, is list property invovled?

[James Gilliam]

no ... my compound index is a combination of a timestamp and a string
and only need desc timestamp, no list property involved ... the
forking of the list property searching has scared me away from them
(fyi)

a while back (after the new pricing was announced), i unindexed lots
of fields and it reduced indexed writes by a factor of 40 ...

is it possible that these former indexes on fields could still be
occupying space and, if so, is there a way to get rid of them since i
no longer index those fields?

e.g. if i copy a namespace and delete the former namespace will that
get rid of field indexes which are no longer indexed ...

thank so much for your help ...

p.s. finally, when i first created my entities i did not realize that
every field was indexed by default but finally figure it out ... :)

On Nov 12, 4:16 am, Max <thebb...@gmail.com> wrote:
> by default every property is indexed except Blob / Text, and these indexes
> are not listed on your index page. Try to unindex them and see how much
> space you can save.
>

> With regards to your *only* composite index, is list property invovled?

[Simon Knott]

Any entities stored before you removed the per-property indexes will still have their index data stored - the per-property indexes are only updated if you rewrite the entities.

Query indexes are rebuilt on deployment.

Cheers,
Simon

[James Gilliam]

thank you ...

i assume when an entity is deleted, its indexes are also deleted. so,
if i copy a namespace, new indexes will be built, and delete the
original, old indexes will be removed. correct?

would it be possible to get some statistics on the size of indexes,
perhaps, when showing namespaces in admin stats? likewise, when
displaying compound indexes?

[James Gilliam]

Can I omit the direction in this index.yaml entry?
- kind: foobars  properties:  - name: tablekey  - name: timestamp   
direction: desc
If not, do I need a second compound indexes if I also want the
direction: asc?
Thanks

[James Gilliam]

also, my Datastore Stored Data has not changed in weeks ... I don't
think it is being calculcated ... is that possible?

[Max]

Datastore statistics are refreshed automatically at least once a day

Do you mean you want to copy all data to new namespace just to unindex it? You don't have to do that, in python you just need to set indexed = false. eg 

age = db.IntegerProperty(indexed=False)

then use mapper api to retrieve all data and save again