MS-HR Migration Post-Mortem

[Greg]

Hi -

I have previously posted about a MS-HR migration how-to here:

http://neogregious.blogspot.com/2011/04/migrating-app-to-high-replication.html

I've now followed up with some lessons learned from the migration:

http://neogregious.blogspot.com/2011/06/high-replication-migration-lessons.html

Hope you find them useful.

Cheers
Greg.

[Ikai Lan (Google)]

Thank you for this!

A few notes:

1. If you do "ancestor queries" where you set an entity group root, that query will be strongly consistent.

2. Get by ID happens transactionally. Currently, I think we only allow each request to have 5 outstanding transactions, so if you do a batch get by ID of, say, 500 entities, this will run about as fast as if you performed 100 fetches serially. To get around this behavior, set the "eventual consistency" flag to true in your datastore client. Note that this will cause your data to exhibit eventual consistent behavior like in point 1 - clever Memcaching should mostly hide the issue from users. We're working on ways to improve this, but for the time being you'll need to be aware of it.

Ikai Lan 
Developer Programs Engineer, Google App Engine

Blog: http://googleappengine.blogspot.com 

Twitter: http://twitter.com/app_engine

Reddit: http://www.reddit.com/r/appengine

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Robert Kluin]

Ikai,
Just wanted to double check, it is 10 concurrent RPCs but only 5
concurrent transactions?

Robert

[Casey Dwyer]

Hi Greg,

>The extra cost is a small price to pay (literally) to escape the maintenance outages, datastore timeouts and high-latency requests that plague the master-slave datastore.
- TFA

IIRC, in one of the I/O talks ("More 9s Please" I believe), they said that read latency is about the same and write latency is high in HRD. Did I hear that incorrectly?

Casey

[Branko Vukelic]

On Fri, Jun 24, 2011 at 11:39 PM, Casey Dwyer <casey...@gmail.com> wrote:
> IIRC, in one of the I/O talks ("More 9s Please" I believe), they said that
> read latency is about the same and write latency is high in HRD. Did I hear
> that incorrectly?

That's exactly correct.

http://www.youtube.com/watch?v=xO015C3R6dw&feature=player_detailpage#t=795s

Just below that, though, the error rate is supposedly much lower for
HRD compared to MS. That's what's meant by "more 9s".

--
Branko Vukelić
bra...@herdhound.com

Lead Developer
Herd Hound (tm) - Travel that doesn't bite
www.herdhound.com

[Ikai Lan (Google)]

Robert - the limit should be:

- 10 active asynchronous RPCs. Technically, since the synchronous API just makes an async call and blocks, I suppose this translates to 10 concurrent RPCs per request. 

- 5 concurrent transaction jobs per datastore instance per request. We are working on various ways to increase this limit, such as parallel transactions to multiple datastore instances. Is my explanation confusing?

To answer the question about read latency - yes, the read latency should be the same. However, get by key is done inside a transaction unless you set "eventually consistent queries" to true. The reason it needs to be done inside a transaction is because we need to be sure the data you are reading is the freshest data. Fresh = data written that passes milestone A:

http://code.google.com/appengine/articles/transaction_isolation.html

Ikai Lan 
Developer Programs Engineer, Google App Engine

Blog: http://googleappengine.blogspot.com 

Twitter: http://twitter.com/app_engine

Reddit: http://www.reddit.com/r/appengine

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

[Robert Kluin]

Cool Ikai, thanks for clarifying.

Robert

[Sergey Schetinin]

On Saturday, June 25, 2011 6:06:16 AM UTC+3, Ikai L (Google) wrote:

Robert - the limit should be:

- 10 active asynchronous RPCs. Technically, since the synchronous API just makes an async call and blocks, I suppose this translates to 10 concurrent RPCs per request.

What happens, if the app tries to start 11th concurrent RPC?