high replication datastore limited to 1 write per second?!

[Philip]

I am concerned about the statement:

"This allows queries on a single guestbook to be strongly consistent,
but also limits changes to the guestbook to * 1 write per second *
(the supported limit for entity groups)."
http://code.google.com/appengine/docs/python/datastore/hr/overview.html.

Is it true that writes are limited to 1 per second when using the high
replication datastore or is this an old limitation?

[Steve Sherrie]

This just refers to the entity group write limit that is the same in
both MS and HR datastores.

[Philip]

So that's what seems odd to me. The HR datastore is best used with
entity groups and yet there is such a huge limit on its throughput.

Since we can't rely on memcached values staying around (since they
might be purged at any time), there seems to be no decent workaround
for having immediately consistent reads (in case memcache is purged)
without being limited to 1 write per second.

Do I have that write (pun intended)?

[Joshua Smith]

Yes, you do.

Welcome to the wonderful awful world of creating applications that will scale.

Go learn about sharded counters next. :)

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>

[Jeff Schnitzer]

You have a mild conceptual misunderstanding - you are always using entity groups whenever you use the datastore, both for M/S and HR.  If you do nothing special, every entity will go into its own entity group so you can write to each one at 1/sec.  But this involves tradeoffs WRT transactions and eventual consistency.

Another alternative is to put your entire database in a single entity group, which gives you perfect consistency and transactionality but limits you to 1 write per second.  Obviously this is pretty preposterous, so in reality you will need to create a balance of entity grouping.  The rule of thumb of "one user's data" is not a bad one, but it's not a hard and fast rule.

Jeff

[Philip]

Knowing about shards, I like them to be transparent when possible.
Using defined entity groups everywhere I write data gives me control,
but also seems to be an extra step in the model design that would have
major implications if I wanted to change them. With the limited
capabilities of GQL, reorganizing data appears to be a major chore
with the datastore.

In the world of shared data, having a design of one entity group per
user seems too limiting. I will try to design the entity groups around
the sharing model (what is shared) and hopefully not the opposite
(where technology drives the business decisions rather than the other
way around).

Thank you all for the good answers.

[Scott Lewis]

I'm writing a multiplayer game using App Engine. What are ways I can
work around the 1 write per second within an entity group? I don't
know when a message acknowledgement will return from a client, and
with the unreliable nature of the memcache, I'm at a loss at where I
can store my data to make sure it is processed.

Thank you!

[Scott Lewis]

Also, is there any way to know that I'm updating an entity group
sooner than once a second?

Thanks!

[Gerald Tan]

There are two methods I can think of.
1) Copy on Write Entities - persist a new copy of the entity each time you want to update it, together with the timestamp it is written. When you want to retrieve the entity, query for the latest version.
2) Use a backend instance to store the entities into the backend instance memory.

[Jeff Schnitzer]

On Mon, Oct 24, 2011 at 10:34 AM, Gerald Tan <woeful...@gmail.com> wrote:

There are two methods I can think of.
1) Copy on Write Entities - persist a new copy of the entity each time you want to update it, together with the timestamp it is written. When you want to retrieve the entity, query for the latest version.

The eventual nature of queries in the HRD will sometimes give you stale versions.

 

2) Use a backend instance to store the entities into the backend instance memory.

This is the best answer if you have anything approaching a realtime game.  Turn-based games can usually be made to work in the datastore, but if you expect a lot of rapidly changing shared state, cache it in memory and sync it to the datastore at intervals.

Jeff

[Nick Johnson]

Hi Scott,

It's impossible to say without more details - what are you trying to achieve, what is your model design, and why do you think this will be a problem? It'd probably be better if you started a new thread - here or on Stack Overflow - with more details.

-Nick Johnson

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Nick Johnson, Developer Programs Engineer, App Engine

[Scott Lewis]

Thanks for the advice, Nick, I will start a new thread.

-Scott-

[Scott Lewis]

For those who care, I posted my question in full in the App Engine
Python group. I would greatly appreciate your help in making my game
work correctly!

http://groups.google.com/group/google-appengine-python/browse_thread/thread/a22c8b2c8d4c67b5

-Scott-