Ancestor write performance with lots of manys

[stevep]

In summary: Is it a bad idea to have one has an ancestor with many
hundreds of children, and a high rate of child write updates.

I am not clear whether there are potential performance issues when
using an ancestor structure where the one ancestor has many hundreds
of children. I have a setup where a Project Kind has up to a thousand
Task Kinds. The Task has a few dozen serialized activities that are
checked out by users. As users complete each activity the Task gets
updated, so there will be many thousands of Task updates. These may be
clustered at times yielding a fairly high rate of transactions per
minute

I have a couple of Task composite indices right now supporting
ancillary Project logic. I could get rid of these, and use ancestor
queries, but am worried about those Task updates which for various
reasons need to be fairly quick -- which I know is already a problem
given the composite indices.

Thanks in advance,
stevep

[Simon Knott]

If you only have a single parent, then you will be limited to the 1 write per second limit on individual entity groups (see bottom of  http://code.google.com/appengine/docs/python/datastore/hr/overview.html where this is noted as a limitation of this kind of structure).

If you attempt many updates to the children, I'd expect to get a lot of concurrent-modification exceptions.

[Jeff Schnitzer]

Just to clarify this slightly, you need to engineer for a throughput of one write transaction per second per entity group.  You can update lots of entities in a single entity group per transaction.

Jeff

On Wed, Nov 23, 2011 at 12:44 PM, Simon Knott <knott...@gmail.com> wrote:

If you only have a single parent, then you will be limited to the 1 write per second limit on individual entity groups (see bottom of  http://code.google.com/appengine/docs/python/datastore/hr/overview.html where this is noted as a limitation of this kind of structure).

If you attempt many updates to the children, I'd expect to get a lot of concurrent-modification exceptions.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/HkDXRsugLd0J.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--

I am the 20%

[Simon Knott]

Oh, thanks for the clarification Jeff. I wasn't aware of that!

[stevep]

Thanks Simon and Jeff. Feel bad that I missed this point, as I had
read it before.

To summarize: A parent with many children comprises an group. Updates
to a group are limited to one per second. If I were to use this
ancestor setup, I would need to provide logic that would in effect
batch a group's children updates, and apply them not more than once
per second.

stevep