Datastore quota madness

[Константин Сабуров]

Hello!

i have an application on master/slave datastore and recently noticed really strange things: datastore size on the dashboard is much bigger than expected. in my app i proactively deleting all unused info to stay below 1Gb quota (and there is not many users) but datastore size do not going down, it going up instead and for now it is over 4Gb. 

and this is not only mine point of view: datastore admin page and datastore statistics both shows numbers which much easier to believe

i attached screenshots, may be i overlooked something... i just whant to delete "invisible entities" which take all of my billable space

please help! :)

[João Campos]

I'm experiencing the same. Datastore statistics tell me the application's using 15MB but on Dashboard stored data's on 1GB. It's seems to have increased really fast because I don't remember seeing the stored data size ever near that value. I've also tried to delete data but to no avail.

Does anyone have an explanation for this? What can be done to solve it?

[de Witte]

Indexes and logs also consume data storage.

Op donderdag 22 maart 2012 11:46:11 UTC+1 schreef Константин Сабуров het volgende:

[Константин Сабуров]

Sorry, i decided the same and deleted all indexes. i also run vacuum_indexes, by guess what?

Datastore size *razed* over 5Gb!

This is definitely something wrong with statistics and the worst part that i should pay for it :(((

See screenshots: i deleted indexes, datastore admin shows small amount of data but datastore size is enourmous. It`s not fair :(

https://picasaweb.google.com/113319821637049481863/GaeStrange#5723102521840496978

https://picasaweb.google.com/113319821637049481863/GaeStrange#5723102549743214066

https://picasaweb.google.com/113319821637049481863/GaeStrange#5723102525560721122

пятница, 23 марта 2012 г. 15:40:53 UTC+4 пользователь de Witte написал:

[Константин Сабуров]

about logs - they are not counted with datastore as far as i know... this is separate space for logs which is separately billed if it is going over 1Gb, see app settings

пятница, 23 марта 2012 г. 15:40:53 UTC+4 пользователь de Witte написал:

Indexes and logs also consume data storage.

[João Campos]

Thanks for your reply.

Even if logs do account for data storage, I've got a little less than 115MB of those. With the 15MB of application data that would make up for a 130MB total.

I've also ran a vacuum_indexes and I've got not custom indexes and a very simple data model (only 3 entities).

With such a simple data model and such small amount of data, can the indexes really be responsible for the missing 894MB (1024-130)?

[Jeff Schnitzer]

If you have not optimized your indexes, they will almost always
account for the bulk of your data storage.

Most of the datastore APIs (python db, ndb, java low-level, jdo/jpa,
etc) default to creating two indexes (asc and desc) on every property.
Each of these index entries represents another row in a BigTable
tablet; for small entities, each of these rows are only slightly
smaller than the master blob. So 5 indexed properties could easily
cause a 5-10X expansion in your total stored data volume.

You can't vacuum these indexes; unlike multi-property indexes defined
in datastore-indexes.xml/yaml, single-property indexes are established
when you save an entity and are only deleted by resaving the entity.

If you are concerned about space (or datastore write operations), you
should go through your data model and flag any properties you aren't
querying on as unindexed. Then you will need to resave your entities
to reclaim index space.

Jeff

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/s00-DD85r4cJ.
>
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

[João Campos]

It still sounds somewhat unlikely since with 15MB we're talking of a more than 60x total expansion. The application has about 15 unindexed properties (I had already set a lot of properties to unindexed, precisely to save storage quota) and according to the datastore statistics, most of the data is on the large properties.

Is there anywhere where you can see the space effectively being consumed by indexes?

One thing I noticed -- when I'm "500 server errors" from the dashboard -- is that on the Logs section it says that the app's using 100% of logs storage (which is not true, because I'm using only ~115MB and the first 1GB are supposed to be free -- http://googleappengine.blogspot.pt/2012/02/app-engine-163-released.html). I can't find the logs storage size anywhere else on the dashboard, quota details or billing history.

Could this be the reason?

[Константин Сабуров]

Sorry, this is not my case. I do not use default indexes ("indexed = true") on models properties from the start. Anyway default indexes cannot be responsible for "exploding" since they are for one property only, so their size can account for 2x 3x ... 10x size of entities only. 4 custom indexes with more than one property in it (i really needed in development) that i have, i already deleted to check is this is a key of the problem. besides that there was a long time when nothing changed inside app, user base grouth was not big and total size was <1Gb

i started migration, when it will be completed i will try to delete all data completely. i already tryed this (making backup before) and datastore size did not drop at all, but maybe there is time needed for stats to be re-counted, so i`ll try later and wait day or two before making new assumptions

пятница, 23 марта 2012 г. 19:45:17 UTC+3 пользователь Jeff Schnitzer написал:

> To post to this group, send email to google-appengine@googlegroups.​com.

> To unsubscribe from this group, send email to

> google-appengine+unsubscribe@​googlegroups.com.

> For more options, visit this group at
> http://groups.google.com/​group/google-appengine?hl=en.

пятница, 23 марта 2012 г. 19:45:17 UTC+3 пользователь Jeff Schnitzer написал:

> To post to this group, send email to google-appengine@googlegroups.​com.

> To unsubscribe from this group, send email to

> google-appengine+unsubscribe@​googlegroups.com.

[David Hardwick]

Is it really two indexes for both asc and desc for each simple
property that is indexed? If so, then it would be great if we could
specify if we only wanted ASC or only DESC to sames some unwanted
writes.

With HRD and indexing, the writes add up real quick (especially using
the Pipeline library[1])

rock on,
-hardwick

[1] - http://code.google.com/p/appengine-pipeline/

[João Campos]

@Константин, you don't need to set "indexed = true" to get default indexes, if you don't set "indexed = false" you get the a default index on the property anyway.

@David, You can specify index direction on custom indexes (http://code.google.com/appengine/docs/python/config/indexconfig.html) just not on default ones, I think.

My datastore is M/S (not HDR) and I don't use the Pipeline library.

The Logs Storage doesn't seem to be related either -- I've got another app with an even smaller log storage size that also shows as 100% of the Total Logs Storage for Version on the Logs section and its datastore size value on the Quota Details is within the expected bounds.

In the meantime I've updated the application with the 1.6.4 pre-release SDK (https://groups.google.com/d/topic/google-appengine/JKC07lp8q5g/discussion), since it's supposed to show the indexes size on the Statistics. Can't find them there for now but their last update might not be recent enough.

I'm also waiting for the Billing History report for the 23rd because that's when, apparently, the datastore size jumped to 1GB (or at least when I noticed it). For the previous days the datastore storage value on the Billing History usage reports is always 0.02GB.

Thanks everyone for the support.

On Saturday, March 24, 2012 12:11:47 PM UTC, David Hardwick wrote:

Is it really two indexes for both asc and desc for each simple
property that is indexed?  If so, then it would be great if we could
specify if we only wanted ASC or only DESC to sames some unwanted
writes.

With HRD and indexing, the writes add up real quick (especially using
the Pipeline library[1])

rock on,
   -hardwick

[1] - http://code.google.com/p/​appengine-pipeline/

On Mar 23, 12:45 pm, Jeff Schnitzer wrote:

[Константин Сабуров]

thanks, of course i set  "indexed = false" on props i don` t need to be indexed

But as i said, after migrating my data away i made experiment: deleted ALL data, ALL indexes and wait more than day.

Datastore size is STILL >5Gb. In empty project (backups are counte into blobstore size, not datastore size if anyone think they are big... they are not)

Something really broken here :( I can give "guest" access to project for any developer to show this is not a joke. Proofpics:

https://picasaweb.google.com/113319821637049481863/GaeStrange#5723740881320880066

https://picasaweb.google.com/113319821637049481863/GaeStrange#5723740848497312226

https://picasaweb.google.com/113319821637049481863/GaeStrange#5723740842315244834

https://picasaweb.google.com/113319821637049481863/GaeStrange#5723740855735209986

суббота, 24 марта 2012 г. 19:40:19 UTC+3 пользователь João Campos написал:

[Jeff Schnitzer]

File a production ticket?

Jeff

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-appengine/-/OKEYKwowuZUJ.

[Константин Сабуров]

Not yet, i though this group is somewhat like bug-notifications system. Can you give me directions where there is a form to GAE tickets system?

On dashboard there is no such link (maybe i overlooked it... but cant find anything support-related except link this group)

Or you mean this page: http://code.google.com/p/googleappengine/issues/list ?

воскресенье, 25 марта 2012 г. 16:30:05 UTC+4 пользователь Jeff Schnitzer написал:

> To post to this group, send email to google-appengine@googlegroups.​com.

> To unsubscribe from this group, send email to

> google-appengine+unsubscribe@​googlegroups.com.

[RollingCircle]

Same issue here - application 'i-notam', using about 1MB of M/S
datastore, suddenly showing 1GB of quota used, raising quota-exceeded
errors, killing the app.

What's going on - are there any App Engine team people here?

>  Scr20120322_1436_#1_DatastoreAdminOfficods1GoogleChrome.jpg
> 157KViewDownload
>
>  Scr20120322_1436_#2_DashboardOfficods1GoogleChrome.jpg
> 333KViewDownload
>
>  Scr20120322_1444_#3_DatastoreStatisticsOfficods1GoogleChrome.jpg
> 190KViewDownload

[Dmitry Gusev]

Same issue in my app -- 'ping-service'

807 entities, 14MBytes total size.

1GB Limit exceeded, all this happened less than a week ago.

[João Campos]

A bit of an update from my application:

• I still can't see the indexes size on the Datastore Statistics, even after the stable release of the SDK 1.6.4. I guess it shows only composite indexes (not the default) and I have none of those;
  
• My Billing History reports keep saying that I'm using only 0.01 GB (rather than the 1GB+ that is shown on the Dashboard).

In the meantime, I've created a new application and copied all my data there. The new application has been working fine since, but then again the original had been working fine for well more than a year and moving to a new application might not be a solution at all for some.

Has anyone filed a production ticket yet?

On Thursday, March 22, 2012 10:46:11 AM UTC, Константин Сабуров wrote:

On Thursday, March 22, 2012 10:46:11 AM UTC, Константин Сабуров wrote:

On Thursday, March 22, 2012 10:46:11 AM UTC, Константин Сабуров wrote:

[Dmitry Gusev]

My app has been fixed two days ago and I can see I'm using only 0.01GB, thought this number grows up to 70% during the day and at the beginning of next day it becomes 1% again.

What I did is marked every field in my entities (which not used by queries) as unindexed.

According to doc I should update each entity, but I couldn't because DB quota size was exceeded.

Not sure how I this been fixed though,  but now it works and looks like unindexed properties doesn't consume any Datastore Write quotas right now (and this was the bottleneck before).

[Dmitry Gusev]

My app is over quotas again.

According to DB stats it only uses 17MB (16 for data, 1MB for indexes), but dashboard shows that 1GB of storage used.

https://sites.google.com/site/dmitrygusev/gae-dbstats-2012-04-08-15.43.59.png?attredirects=0&d=0

https://sites.google.com/site/dmitrygusev/gae-overquota-2012-04-08-15.44.25.png?attredirects=0&d=0