why my datastore nearly be full?

9 views
Skip to first unread message

saintthor

unread,
Jan 30, 2010, 3:27:21 AM1/30/10
to Google App Engine
in quota details, i notic my Total Stored Data is 95%. it shouldn' t
be so large.

in Datastore Statistics, Size of all entities is only 9MB.

all that happened from about 4% to 95% is i updated an entity for
about 200 times. the class is:

class ltWords( db.Model ):
WordsDText = db.TextProperty()

after this, there is only one entity of ltWords and the WordsDText is
no longer than 1MB. why does it take more than 90% of my datastore
quota and how to release it?

for saintthor.appspot.com

Sylvain

unread,
Jan 30, 2010, 7:42:03 AM1/30/10
to Google App Engine

saintthor

unread,
Jan 30, 2010, 8:08:01 AM1/30/10
to Google App Engine
like this.

i did nothing about memcache. but i updated all data in taskquene. and
exceptions of textproperty over lengh has occured in updating.

saintthor

unread,
Jan 31, 2010, 3:05:40 AM1/31/10
to Google App Engine
i deleted the only entity of ltWords. the Total Stored Data is still
95%!

who can help me?

On 1月30日, 下午8时42分, Sylvain <sylvain.viv...@gmail.com> wrote:

Wooble

unread,
Feb 1, 2010, 11:27:31 AM2/1/10
to Google App Engine
The stored data quota in the dashboard isn't updated in realtime, so
you wouldn't see the change immediately even if that is the problem.
I'd suspect some buggy indexes or something, though, if that really
was your only entity. For one thing, you can't have a single entity
larger than 1MB, and it would be tough to use 949 MB in datastore
overhead even if you wanted to.

Hugo Visser

unread,
Feb 1, 2010, 11:39:26 AM2/1/10
to Google App Engine
I'm seeing the same thing. I got hardly anything stored in the
datastore, but my quota is at 50% right now. I did have some stuff in
the memcache however.
This morning I flushed memcache and deleted all entities in the
datastore but the quota usage is still going up. I'm waiting to see
what happens after the daily reset.

Hugo

Robert Kluin

unread,
Feb 1, 2010, 2:51:05 PM2/1/10
to google-a...@googlegroups.com
We see about 10x overhead, I assume from indexes, in our apps. One
app is using 30M for storing entities and metadata, the indexes take
another 300M. We see similar usage multipliers in all of our apps.

Robert


2010/2/1 Hugo Visser <bott...@gmail.com>:

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>
>

dmakariev

unread,
Feb 1, 2010, 5:54:24 PM2/1/10
to Google App Engine
Hi to all ,
I'm experiencing the same problem. I've described it also in :
http://groups.google.com/group/google-appengine-java/browse_thread/thread/27d03163d170e6ec/9ad021dee7e82359
Currently my status is :
from the dash board : Total Stored Data : 38% (and definitely
counting..)
Datastore statistics : Size of all entities 414 KBytes
from which the main is _ah_SESSION - 86%
I don't have any indexes defined. the number of sessions which my
application currently keeps are round 40 per hour.

Any help regarding this would be appreciated.
If anybody from Google guys is interested - my app-id is "sandcode"

Best Regards:
Dimitar Makariev

On Feb 1, 11:51 am, Robert Kluin <robert.kl...@gmail.com> wrote:
> We see about 10x overhead, I assume from indexes, in our apps. One
> app is using 30M for storing entities and metadata, the indexes take
> another 300M. We see similar usage multipliers in all of our apps.
>
> Robert
>

> 2010/2/1 Hugo Visser <botte...@gmail.com>:

dmakariev

unread,
Feb 1, 2010, 5:26:51 PM2/1/10
to Google App Engine
Hi to all ,
I have more or less the same problem.
I've described it also here :
http://groups.google.com/group/google-appengine-java/browse_thread/thread/27d03163d170e6ec/9ad021dee7e82359
my current status is :
Total Stored Data : 37%
While at the same time the "Display statistics" is : Size of all
entities 414 KBytes
with 86% used by _ah_SESSION , which is ok

I don't have any indexes defined.

if the guys from Google want to take a look my app-id is "sandcode"

Best Regards:
Dimitar Makariev


On Feb 1, 11:51 am, Robert Kluin <robert.kl...@gmail.com> wrote:

> We see about 10x overhead, I assume from indexes, in our apps. One
> app is using 30M for storing entities and metadata, the indexes take
> another 300M. We see similar usage multipliers in all of our apps.
>
> Robert
>

> 2010/2/1 Hugo Visser <botte...@gmail.com>:

Hugo Visser

unread,
Feb 2, 2010, 3:56:15 AM2/2/10
to Google App Engine
Well I guess what I'm seeing is not "normal". I have 129 entities that
take 1 MB in total according to the statistics. I have 48 MB in
memcache and that's it.
Still my total stored data quota is at 59% and didn't get reset, even
after flushing the memcache and deleting all of my entities from the
datastore.

Besides memcache and datastore I also use task queues, anybody else
using those and seeing a quota rise?

This started somewhere around the weekend so I really think this is a
bug of somekind...I hope Google will look into it soon.
This post seems to be related:
http://groups.google.com/group/google-appengine/browse_thread/thread/e619865eaf584706

Hugo

On Feb 1, 8:51 pm, Robert Kluin <robert.kl...@gmail.com> wrote:
> We see about 10x overhead, I assume from indexes, in our apps. One
> app is using 30M for storing entities and metadata, the indexes take
> another 300M. We see similar usage multipliers in all of our apps.
>
> Robert
>

> 2010/2/1 Hugo Visser <botte...@gmail.com>:

Hugo Visser

unread,
Feb 2, 2010, 6:58:55 AM2/2/10
to Google App Engine
Any suggestions from Google folks such as Nick maybe?

On Feb 2, 9:56 am, Hugo Visser <botte...@gmail.com> wrote:
> Well I guess what I'm seeing is not "normal". I have 129 entities that
> take 1 MB in total according to the statistics. I have 48 MB in
> memcache and that's it.
> Still my total stored data quota is at 59% and didn't get reset, even
> after flushing the memcache and deleting all of my entities from the
> datastore.
>
> Besides memcache and datastore I also use task queues, anybody else
> using those and seeing a quota rise?
>
> This started somewhere around the weekend so I really think this is a
> bug of somekind...I hope Google will look into it soon.

> This post seems to be related:http://groups.google.com/group/google-appengine/browse_thread/thread/...

Hugo Visser

unread,
Feb 2, 2010, 7:04:28 AM2/2/10
to Google App Engine
One other observation is that in the billing summary all seems fine:

Stored Data:
$0.005/GByte-day 0.00 1.00 0.00 $0.00

Maybe a dashboard/statistics bug?

Hugo Visser

unread,
Feb 2, 2010, 3:51:07 PM2/2/10
to Google App Engine
OK, searched through the groups again and all of the threads
mentioning this are referring to the size of the index. Initially I
thought I was imune to this issue because I have no indexes defined :)
But turned out that I every property of my entity was indexed, while
they weren't supposed to. I changed my code, deleted the old data and
the quota is down now.

Hope this helps others finding this thread.

saintthor

unread,
Feb 2, 2010, 11:06:57 PM2/2/10
to Google App Engine
there are at most 20 minutes from the updatings to i notice the quota
increase to 95%.

do you mean if i put an entity larger than 1MB, it will take 949MB in
my datastore?

when i do the updatings, some exception occured like

BadRequestError: string property WordsDText is too long. It cannot
exceed 1000000 characters.

the entity will still take 949MB even after the exception?

above all, how to solve it? i want to know how to release the 949MB
more than why it happen.

please help me.

On 2月2日, 上午12时27分, Wooble <geoffsp...@gmail.com> wrote:
> The stored data quota in the dashboard isn't updated in realtime, so
> you wouldn't see the change immediately even if that is the problem.
> I'd suspect some buggy indexes or something, though, if that really
> was your only entity. For one thing, you can't have a single entity
> larger than 1MB, and it would be tough to use 949 MB in datastore
> overhead even if you wanted to.
>

> On Jan 31, 3:05 am,saintthor<saintt...@gmail.com> wrote:
>
> > i deleted the only entity of ltWords. the Total Stored Data is still
> > 95%!
>
> > who can help me?
>
> > On 1月30日, 下午8时42分, Sylvain <sylvain.viv...@gmail.com> wrote:
>
> > > maybe this bug ?
>
> > >http://code.google.com/p/googleappengine/issues/detail?id=631
>

saintthor

unread,
Feb 2, 2010, 11:18:25 PM2/2/10
to Google App Engine
GREAT!!!!

my Total Stored Data resumed to 4%.

who did that? thank you. you are my god.

Reply all
Reply to author
Forward
0 new messages