AppEngine datastore disk space usage - entities & index size calculation
Sharp-Developer.Net
I wonder how to calculate estimated disk space usage by entities and
indexes?
It would be nice to have some formula the same way as Amaxon have for
SimpleDB.
Also I think it would be a good idea if we have a visibility of space
usage by entity and index at control panel/dashboad.
Is there any guidance on formula and plans to provide the stats on
disk space usage by datastore API?
As I personally have few options how to organize my data schema and
would prefer to make a decision based on facts rather than a guess.
Folks, please rate this message if you have the same question becouse
I haven't found anything by searhing through docs or Internet.
Thanks.
--
Alexander Trakhimenok
http://sharp-developer.net/
javaDinosaur
will not be charged for because they want AppEngine developers write
queries that take best advantage of the DataStore api rather than code
their own cost avoidance query/sort routines in the middle tier.
When sizing DataStore entities you need to consider that entities are
just serialized object blobs, so space usage is going to be less
efficient that a conventional SQL database with separate schema.
nchauvat (Logilab)
> Folks on the Google AppEngine team have said that index space usage
> will not be charged for because they want AppEngine developers write
> queries that take best advantage of the DataStore api rather than code
> their own cost avoidance query/sort routines in the middle tier.
the reason the free storage quota was lower than for other Google
services (Gmail is over the GigaBytes for example) is that the space
used is actually larger due to indexes, logs, etc.
David Symonds
Don't forget that Gmail also started with a "measley" 1 GB, and has
grown over time. Don't be too surprised if the free storage quota of
App Engine grows over time, too.
Dave.
joh...@easypublisher.com
Make me wonder, is the requirement for indexes and logs lower
for GMail then an average GAE app? Why is that so?
(very hypothetically speaking)
/Johan
--
Johan Carlsson
Colliberty Easy Publisher
http://www.easypublisher.com
Derek Upham
> Folks on the Google AppEngine team have said thatindexspace usage
> queries that take best advantage of the DataStore api rather than code
> their own cost avoidance query/sort routines in the middle tier.
Sjors
> Folks on the Google AppEngine team have said that index space usage
> will not be charged for because they want AppEngine developers write
> queries that take best advantage of the DataStore api rather than code
> When sizing DataStore entities you need to consider that entities are
> just serialized object blobs, so space usage is going to be less
> efficient that a conventional SQL database with separate schema.
1 - Does the *current* dashboard already subtract the index size?
2 - How do you calculate the size of an object blob?
Example:
I have 1.44 million entities. Each entity consists of a key and an
integer. This eats 120 MB according the dashboard. That's almost 90
bytes for an integer. I actually only need to store a smallint (2
bytes).
Since I am writing an application that wants to store 21 billion
smallints, this problem represents the difference between 400 dollars
or 10 dollars a month on data storage: the difference between a hobby
and needing a business plan.
Sjors
Brett Morgan
Any particular reason you need to store the integers seperately? Remember, queries only return a thousand objects. You would probably be advised to store chunks of these integers in entities, aportioned by some sane metric, be it time, or geo location, or...
Sjors
--
Brett Morgan http://brett.morgan.googlepages.com/
Sjors Provoost
>> integer. This eats 120 MB according the dashboard. That's almost 90
>> bytes for an integer. I actually only need to store a smallint (2
>> bytes).
>>
>> Since I am writing an application that wants to store 21 billion
>> smallints, this problem represents the difference between 400 dollars
>> or 10 dollars a month on data storage: the difference between a hobby
>> and needing a business plan.
>
> Any particular reason you need to store the integers seperately? Remember,
> queries only return a thousand objects. You would probably be advised to
> store chunks of these integers in entities, aportioned by some sane metric,
> be it time, or geo location, or...
>
>>
>> Sjors
Good question. I am not sure, but I think the answer is that yes, I need that.
Each points represents an altitude (from the NASA SRTM elevation data
of the whole world). My application displays an altitude profile of a
route. Let's say routes are typically between 1 km (walking) and 100
km (driving) and the altitude profile consists of 100 points (good
enough for mobile phone display). The elevation data resolution is 100
meters.
In that case the typical distance between the queried points is
between 100 (less makes no sense) and 1000 meters, or 1 and 10 data
points. So if I want to grab even just 2 points in one query, that
query would probably take another +- Pi * 10^2 points with it.
In other words, from the point of view of the database, the points
that I query are random and far between, so there is no point in
grouping them.
Sjors
Charlie
about 95% of that space going to the entity key. This seems to make
sense since each key is a string. I remember hearing a limit
associated with each key at Google I/O. Does anyone know what it is?
I think I might have been as much as 1,000bytes.
It is extremely tempting to use the data store like a python
dictionary type. Has anyone found a good design pattern to get around
this?
- Charlie.
P.S. Sjors, I suspect NASA SRTM comes in raster form, you could
probably tile this data and store the smaller raster a packed arrays
in the Blob type. Type array.array().tostring() This will give you
good space efficiency since you can store the Short Ints in only 2
bytes.
On Jul 4, 5:20 am, "Sjors Provoost" <provoostena...@gmail.com> wrote:
> >> I have 1.44 millionentities. Each entity consists of a key and an
> >> bytes for an integer. I actually only need to store a smallint (2
> >> bytes).
>
> >> Since I am writing an application that wants to store 21 billion
> >> smallints, this problem represents the difference between 400 dollars
> >> or 10 dollars a month on data storage: the difference between a hobby
> >> and needing a business plan.
>
> > Any particular reason you need to store the integers seperately? Remember,
> > queries only return a thousand objects. You would probably be advised to
Sjors Provoost
>
> In Sjors' example each entity seems to take around 90bytes. With
> about 95% of that space going to the entity key. This seems to make
> sense since each key is a string. I remember hearing a limit
> associated with each key at Google I/O. Does anyone know what it is?
> I think I might have been as much as 1,000bytes.
I am still not sure if that 95% indeed belongs to the entity key, or
if the key is already subtracted from my usage. But I hope you are
right.
> P.S. Sjors, I suspect NASA SRTM comes in raster form, you could
> probably tile this data and store the smaller raster a packed arrays
> in the Blob type. Type array.array().tostring() This will give you
> good space efficiency since you can store the Short Ints in only 2
> bytes.
I am thinking that, in the current situation, the most efficient way
to store the SRTM data is in a file (or multiple files), in stead of
in the data store.
The most important thing is that I need to minimize disk reads so that
I only grab the 2 bytes that I need. My guess is that excludes .tar.gz
compression. That is a pity because this data just screams for
compression.
But I would much rather use the data store, for the sake of elegance.
Sjors
Charlie
Try tiling the original raster. And store each tile as a blob in an
entity. You can use the zlib module to compress/decompress each tile
and if you make them small enough this should be efficient. I assume
you'll be using Google maps, or some other map, to determine the
route. If you match google's tiling system, you can look at which
tiles the user's route crosses and pull only those tiles out of data
store. (Using gets rather than queries.)
I look forward to seeing your app,
Charlie.
On Jul 12, 12:49 am, "Sjors Provoost" <sj...@sprovoost.nl> wrote:
Sjors Provoost
Thanks for your suggestion! You can see my prototype app in action at:
http://dudarev.com/webmaps/profiledemo/
And I wrote a post about the current issues:
http://sprovoost.nl/2008/07/13/restful/
Kind regards,
Sjors