I'm looking for some opinions on to what degree I should aggregate my now small entities together into larger entities. Presently I have 50,000 "Day" entities where each entity is represents a day.
They are relatively small with 6 float, 2 bool, 1 int, and 1 string property. No property indexes. Datastore statistics says they average 161 bytes of data and 80 bytes of metadata (241b total).
80% of my user requests are GETs in which I read:
70% of the time, 10 day entities
25% of the time, 30 day entities
05% of the time, 365 day entities
20 of my user requests are POSTs in which I read & write:
75% of the time, 1 day entity
15% of the time, 7 day entities
10% of the time, ~15 day entities
Since the new pricing is going to charge me per entity read and per entity write (and thankfully no property indexes here), I think I should look at reducing how many reads and writes are involved. I could very easily chunk these individual day entities into groups of 10, or groups of 30. That would (by my rough guess on metadata savings) put the entity size around 2k or 6k respectively.
I am wondering where the line is between retrieving fewer entities and each entity becoming too big because the overhead of unwanted days. With a 10 day chunk, my most frequent GET request would usually need 2 entities (4k) where only half the data was in the needed range. At a 30 day chunk, usually 1 entity would suffice (6k) but 4k of that would be unwanted overhead.
I'm having a hard time getting some internal model for what the impact of serializing & deserializing the overhead days would be. I wish appstats wasn't just for RPCs. I'm guessing the extra time to transfer the larger entities to/from the datastore is relatively minimal with Google's network infrastructure. But now that CPU is throttled down to 600Mhz I don't know what kind of latencies I'd be adding in with serialization.
Right now my most common POST operation is to put 1 entity of 241b. With a 30 day chunk, that would be still a single entity put but 6k in size.
Any opinions, ideas, gut feelings, etc?
Cheers,
Steve