Large number of datastore puts

[Richard Druce]

With the appengine pricing changes, we've been paying attention to our datastore puts. According to the pricing comparison chart we're making 2.18 million puts a day. This seems a lot higher than expected. We receive about 0.6 queries per second which means that each request is making about 60 puts!!

Using the sample code for db profiling http://code.google.com/appengine/articles/hooks.html we measured this for a day and the most we counted was ~14,000 which seems more reasonable. Does anyone have experience with something similar on their site?

Thanks,
Richard

[Jason Collins]

We are seeing a lot more datastore write operations than we can
account for (375M / day). Still trying to get to the bottom of it
because it makes for a scary line item on the new sample bills. We
haven't looked closely at read operations yet because the writes
dwarfs it.

This blog post outlines some unexpected read statistics:
http://point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine-and-the-two-orders-of-magnitude/
. I think that using offset with your queries (instead of cursors)
could lead to an inflated number of reads.

Aside, I'm really surprised that datastore reads cost 70% of a write.
I would have expected perhaps an order of magnitude cheaper.

j

On Sep 6, 7:57 am, Richard Druce <contactd...@gmail.com> wrote:
> With the appengine pricing changes, we've been paying attention to our
> datastore puts. According to the pricing comparison chart we're making 2.18
> million puts a day. This seems a lot higher than expected. We receive about
> 0.6 queries per second which means that each request is making about 60
> puts!!
>

> Using the sample code for db profilinghttp://code.google.com/appengine/articles/hooks.htmlwe measured this for a

[Jason Collins]

Also, not setting indexed=False on your attributes leads to an
unexpectedly high number of datastore writes. Each attribute without
indexed=False will yield two more datastore write operations (asc,
desc), I believe.

Even still, we're trying to account for the large number of datastore
write ops....

j

On Sep 6, 3:05 pm, Jason Collins <jason.a.coll...@gmail.com> wrote:
> We are seeing a lot more datastore write operations than we can
> account for (375M / day). Still trying to get to the bottom of it
> because it makes for a scary line item on the new sample bills. We
> haven't looked closely at read operations yet because the writes
> dwarfs it.
>

> This blog post outlines some unexpected read statistics:http://point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine...

> . I think that using offset with your queries (instead of cursors)
> could lead to an inflated number of reads.
>
> Aside, I'm really surprised that datastore reads cost 70% of a write.
> I would have expected perhaps an order of magnitude cheaper.
>
> j
>
> On Sep 6, 7:57 am, Richard Druce <contactd...@gmail.com> wrote:
>
>
>
>
>
>
>
> > With the appengine pricing changes, we've been paying attention to our
> > datastore puts. According to the pricing comparison chart we're making 2.18
> > million puts a day. This seems a lot higher than expected. We receive about
> > 0.6 queries per second which means that each request is making about 60
> > puts!!
>

> > Using the sample code for db profilinghttp://code.google.com/appengine/articles/hooks.htmlwemeasured this for a

[Alfred Fuller]

A datastore write op != an entity put/delete. It is actually (entity + |index deltas|).

For example if you have a Kind with 3 indexed properties and 1 composite index the number of write ops to put a new entity would be:

1 entity + 1 kind index + (1 ascending + 1 descending) * 3 indexed properties + 1 composite index value = 9 write ops

if you change a single property on an existing entity it will be:

1 entity + (2 ascending changes + 2 descending changes) * 1 indexed property changed + 2 composite index values changed = 7 write ops

(the change must remove the old value and add the new value for each index)

deleting the entity will cost the same a creating it (9 write ops)

It is important to note that these costs have not changed in the new pricing model, they were just obfuscated in the old model in the form of cpu hours. Hope this clears it up.

 - Alfred

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Jason Collins]

Thanks Alfred. That's the clearest definition of datastore write
operations I've seen to-date.

The delta aspect is super-cool and I'm glad to hear it, however, it
makes my accounting for our large number of datastore write operations
even more out of whack - while we might be updating entities, we're
certainly not changing many/all of the indexed properties.

The search continues....

j

On Sep 6, 4:18 pm, Alfred Fuller <arfuller+appeng...@google.com>
wrote:

> A datastore write op != an entity put/delete. It is actually (entity +
> |index deltas|).
>
> For example if you have a Kind with 3 indexed properties and 1 composite
> index the number of write ops to put a new entity would be:
> 1 entity + 1 kind index + (1 ascending + 1 descending) * 3 indexed
> properties + 1 composite index value = 9 write ops
>
> if you change a single property on an existing entity it will be:
> 1 entity + (2 ascending changes + 2 descending changes) * 1 indexed property
> changed + 2 composite index values changed = 7 write ops
> (the change must remove the old value and add the new value for each index)
>
> deleting the entity will cost the same a creating it (9 write ops)
>
> It is important to note that these costs have not changed in the new pricing
> model, they were just obfuscated in the old model in the form of cpu hours.
> Hope this clears it up.
>
>  - Alfred
>

[Matt Jibson]

from http://code.google.com/appengine/articles/managing-resources.html:

"Whenever possible, replace indexed properties (which are the default)
with unindexed properties, which reduces the number of datastore write
operations when you put an entity."

[Stephen]

On Tue, Sep 6, 2011 at 10:05 PM, Jason Collins
<jason.a...@gmail.com> wrote:
>
> We are seeing a lot more datastore write operations than we can
> account for (375M / day). Still trying to get to the bottom of it
> because it makes for a scary line item on the new sample bills.

Divide 375M writes by number of request to get the average writes per
request. Use code similar to the following to log requests with more
than average writes:

https://gist.github.com/715284

Should home in on the write-happy code pretty quickly.

[Alexis]

We also have a large number of Datastore Write ops in our sample bill,
and we figured out that it was likely due to properties with
indexed=True
thanks to the Quota Details admin page that gives more detail:

Datastore CPU Time 0% 431.12 of Unlimited CPU hours
Datastore Entity Fetch Ops 0% 8,288,728 of Unlimited
Datastore Entity Put Ops 0% 4,159,838 of Unlimited
Datastore Entity Delete Ops 0% 1,072,478 of Unlimited
Datastore Index Write Ops 0% 37,251,177 of Unlimited
Datastore Query Ops 0% 642,110 of Unlimited
Datastore Key Fetch Ops 0% 4,403,176 of Unlimited

Here we clearly see that it's "Datastore Index Write Ops" that is
responsible for an expensive bill.
We will see soon the impact of having indexed=False on most
properties.

Hope it helps

On 7 sep, 13:23, Stephen <sdeasey+gro...@gmail.com> wrote:
> On Tue, Sep 6, 2011 at 10:05 PM, Jason Collins
>

[aditya tiwari]

Thanks a lot for the reply