Datastore Write Operations is 16x more of other operations

[Tom Fishman]

Hi,

Our data-store operation statistics is strange, writing operation is 16x more than other operations ( Read and small ).

I checked with appstat, it is usually 1:1, at most 2:1. I don't know where this 16:1 come from?

What's the best way to investigate?

-Tom

[Simon Knott]

How many properties do you have indexed in your entities, and how many custom indexes do you have?

Alfred gives a good example of how writes are calculated for new and updated entities here - https://groups.google.com/d/msg/google-appengine/mjnSqQWOfqU/cgPVeHbrR8oJ

[James Gilliam]

The default is to indexed all properties in an Entity. You have to say
indexed = False or you will get crazy high indexed writes. IMO,
default should be False.

[Kenneth]

Default should not be false. turning back on an index is much much harder than turning it off.  If you turn it back on you have to rewrite all your entities, with the potential for not retrieving some data when executing a query if you don't.  That's bad.

Having said that Google needs to be much more transparent about indexes and writes in general.  There's still no indication on the datastore stats page about the size of your indexes.  There's also no way to know how much your writes are costing you.

[Jeff Schnitzer]

On Sat, Nov 19, 2011 at 8:06 AM, Kenneth <kenn...@aladdinschools.com> wrote:

Default should not be false. turning back on an index is much much harder than turning it off.  If you turn it back on you have to rewrite all your entities, with the potential for not retrieving some data when executing a query if you don't.  That's bad.

FWIW, we're changing the default to false in Objectify 4.0 (among a number of other major changes).  It's universally regarded by our users (and twig's users) that defaulting to true was a mistake.  Defaulting to false forces you to think about what to index from the start - a mandatory step now that indexes are expensive.

Jeff

[Kenneth]

I can certainly see forcing you to decide by not providing a default, that makes you to think about it. Defaulting to off doesn't force you to do anything, it just swings the balance to cost is more important than functionality.  I suspect you're going to be getting a lot of novices who are wondering why their data isn't coming back, or worse who don't realise their data isn't coming back.

Actually, thinking about it some more, if you never create an index on a property, ie as you said it start off at false, does that give you an immediate error if you filter on that property?  That would change my opinion.

[James Gilliam]

One related thing. If you create a bunch of entities with indexes =
True and later turn it to False. Those indexes still exist for
existing entities, until you rewrite the records.