Announcing the High Replication Datastore for App Engine

[Ikai Lan (Google)]

Hey everyone,

If you don't already subscribe to our blog in your RSS reader, I advise that you do so:

http://googleappengine.blogspot.com/2011/01/announcing-high-replication-datastore.html

I've embedded the post below. This is a feature that I'm very excited about. Feel free to post to this thread if you have any questions.

Announcing the High Replication Datastore for App Engine

When App Engine launched over two years ago, we offered a Datastore that was designed for quick, strongly consistent reads. It was based on a Master/Slave replication topology, designed for fast writes while still allowing applications to see data immediately after it was written. For the past six months, as you are probably aware, we’ve been struggling with some reliability issues with the App Engine Datastore. Over the course of the past few months, we’ve made major strides in fixing these issues. However, our experience with these issues has made us rethink some of our design assumptions. As we promised you in some of our outage reports earlier this year, we wanted to give you a more fundamental solution to the problem.

Today I’m proud to announce the availability of a new Datastore configuration option, the High Replication Datastore. The High Replication Datastore provides the highest level of availability for your reads and writes, at the cost of increased latency for writes and changes in consistency guarantees in the API. The High Replication Datastore increases the number of data centers that maintain replicas of your data by using the Paxos algorithm to synchronize that data across datacenters in real time. One of the most significant benefits is that all functionality of your application will remain fully available during planned maintenance periods, as well as during most unplanned infrastructure issues. A more detailed comparison between these two options is available in our documentation.

From now on, when creating a new application, you will be able to select the Datastore configuration for your application. While the current Datastore configuration default remains Master/Slave, this may change in the future.

Datastore configuration options when creating an app.

The datastore configuration option can not be changed once an application is created, and all existing applications today are using the Master/Slave configuration. To help existing apps migrate their data to an app using the High Replication Datastore, we are providing some migration tools to assist you. First, we have introduced an option in the Admin Console that allows an application toserve in read-only mode so that the data may be reliably copied between apps. Secondly, we are providing a migration tool with the Python SDK that allows you to copy from one app to another. Directions on how to use this tool for Python and Java apps is documented here.

Now, a word on pricing: Because the amount of data replication significantly increases with the High Replication datastore, the price of this datastore configuration is different. But because we believe that this new configuration offers a significantly improved experience for some applications, we wanted to make it available to you as soon as possible, even though we haven’t finalized the pricing details. Thus, we are releasing the High Replication Datastore with introductory pricing of 3x that of the Master/Slave Datastore until the end of July 2011. After July, we expect that pricing of this feature will change. We’ll let you know more about the pricing details as soon as they are available, and remember, you are always protected when pricing changes occur by our Terms of Service. Due to the higher cost, we thus recommend the High Replication Datastore primarily for those developers building critical applications on App Engine who want the highest possible level of availability for their application.

Thank you, everyone, for all the work you’ve put into building applications on App Engine for the past two years. We’re excited to have High Replication Datastore as the first of many exciting launches in the new year, and hope you’re excited about the other things we’ve got in store for App Engine in 2011.

Posted by Kevin Gibbs, The App Engine Team

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

Blogger: http://googleappengine.blogspot.com

Reddit: http://www.reddit.com/r/appengine

Twitter: http://twitter.com/app_engine

[Ikai Lan (Google)]

By the way, I should mention that there is no Java SDK. This is *almost* completely a server side change.

The Python SDK ships with migration tools for migrating an existing application to an HR instance. As of right now, there are no Java tools - please use the Python SDK if you need this functionality.

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

Blogger: http://googleappengine.blogspot.com

Reddit: http://www.reddit.com/r/appengine

Twitter: http://twitter.com/app_engine

[Raymond C.]

sounds like a very good option for some applications.  Thank you so much Google guys~

[James Broberg]

Sounds great. Would it be reasonable to expect that read latencies would be slightly better due to the datastore being more highly replicated?

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Nickolas Daskalou]

Hi Ikai,

Great feature! Some questions:

- Typically, how long does it take for HR query data to become consistent (a ballpark figure)?

- Given that developers now have this HR option - allowing them to avoid planned read-only datastore periods - should we expect the frequency of planned read-only periods for the Master/Slave setup to increase?

- Are we able to create new applications with the same name as an already deleted application? This would allow us to workaround the issue of not being able to change an app's storage options once created. Eg: (Step 1) Create a new Master/Slave app named 'mytempapp', (2) Migrate data from an existing Master/Slave app named 'mycoolappname' to the app 'mytempapp', (3) Delete the 'mycoolappname' app, (4) Create a new HR app named 'mycoolappname', (5) Migrate data from 'mytempapp' to the 'mycoolappname' app, (6) Delete the 'mytempapp' app.

Nick

[Martin Ceperley]

Great news, thanks app engine team!

One question, are there any tools for migrating Blobstore/Blobinfo data?

[Matija]

Will you uplift high cpu quota warning and new instance creation average latency prerequisite for apps with high replication datastore ?

[Fabrizio Accatino]

Ikai,

can you elaborate a bit more? Will it be available a java sdk in the future?

The Datastore Admin will be available also for java apps?

http://code.google.com/appengine/docs/adminconsole/datastoreadmin.html

Thank you

Fabrizio

[Alexander Maslov]

I have tried to post this comment GAE blog, but my message was deleted several times. So let me post it here.

Thanks for this update. The is one think I would like to comment. Why do you limit developers to use only one datastore per app? It mite be much better to include additional parameter in datastore API to allow developers to access data in both datastores. As application developer it's hard to tell before hand what replication method better for any specific application. Personally I prefer more reliable solution for any project. But in some large applications there mite be some critical data and some less important data.

Speaking about current implementation. It's really weird that we do not have any way to migrate to High Replication Datastore without creating new application. It mite look not a big issue from Google's point of view, but that makes a lot of troubles to those developers, who use appspot.com domain to host applications. New application name means new domain name, etc.

[Ice13ill]

Will this release have an impact on the latencies and error rates for
applications that use (and will use) Master/Slave ? (hopefully in a
better way)
I was wondering if the increase of the latencies and errors in the
last few months were caused by the intense work of making the new
option available.

[Jeff Schwartz]

On Thu, Jan 6, 2011 at 8:19 AM, Alexander Maslov <alexm...@gmail.com> wrote:

+1

Well said, Alexander, and I wholeheartedly agree. The migration path should be automated from the back end when the developer selects the HRD from the admin application configuration console. I can see the lack of automation here as being a major impediment to the HRD's adaption for developers with existing applications.The process as it now stands exposes live data to possible erroneous migration processes that we create to migrate live data which could cause integrity issues if we don't get it 100% right the first time. And having to create a new application in order to migrate is a no starter IMO.

Come on, Google guys and gals, you are definitely talented enough to automate the migration which would eliminate any chance of damaging live data and eliminate the need to create a new application.

Sincerely,
Jeff

Speaking about current implementation. It's really weird that we do not have any way to migrate to High Replication Datastore without creating new application. It mite look not a big issue from Google's point of view, but that makes a lot of troubles to those developers, who use appspot.com domain to host applications. New application name means new domain name, etc.

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Jeff Schwartz

[stevep]

I think this should be great for my new "high confidence" put
architecture changes (original thread below).

If I'm reading this right, I can set up a separate High Reliability
application that will handle the put().

These puts are much lower volume than other client activities, but I
really need high reliability for them.

So architecture at a summary level is:

1) Client initiates the put to MS application.
2) MS application launches a task queue task to HR application.
3) MS replies back to client that HR task is launched.
4) Client pings HR application to confirm write (showing appropriate
dialog).
5) HR task queue puts record.
6) Client now gets HR confirmation to ping and shows confirmation
dialog.

If I've missed something please advise. The higher cost and higher cpu
cycles for HR (link below) are not issues as only a subset of
critical, lower volume puts get HR costing.

If I've not missed anything, then pleased to say in advanced (before
refactoring code) "Thanks Google".

Links:
Previous write architecture thread:
http://groups.google.com/group/google-appengine/browse_thread/thread/c0e44d36d54384a3

MS vs. HR performance testing (thanks kaekon):
http://groups.google.com/group/google-appengine/browse_thread/thread/5fc3b6a4366de62f

[Ikai Lan (Google)]

Hey guys, some answers to your questions that we've compiled from various team members (mostly Alfred, presenter of "Next Gen Queries" and the fourth one from the left in this video http://www.youtube.com/watch?v=dg0TEIRQePg):

- Typically, how long does it take for HR query data to become consistent (a ballpark figure)?

100ms is the estimate. However - we have seen cases where this can be higher. If you were to use the eventual consistency read option in the master-slave configuration, we've seen that the average replication lag is about 3 minutes with an upper bound of 10 minutes.

- Given that developers now have this HR option - allowing them to avoid planned read-only datastore periods - should we expect the frequency of planned read-only periods for the Master/Slave setup to increase?

No. We had roughly one maintenance period every 2 months last year. The number of scheduled maintenance periods should be about the same. Note that there will be a scheduled maintenance period on February 7th - I'll be posting the details to the downtime notify list shortly. 

- Will this release have an impact on the latencies and error rates for

applications that use (and will use) Master/Slave?

No. Master/Slave remains unchanged. However we are currently working on updates to Master/Slave that should increase availability.

- Why do you limit developers to use only one datastore per app?

High Replication can serve out of multiple places at the same time while Master/Slave cannot. If we allowed both types of datastores in the same app it would have one of the following problems:

   - we'd be limited to serving out of the single 'Master' location which would greatly reduce the availability of the app as a whole, and you wouldn't see a benefit 

   - serving out of a non-'Master' location which would greatly reduce the availability and performance of the Master/Slave datastore, as it would have to reach across data centers for consistent data

   

In the future we might find ways around these trade offs and offer this feature, but this is not something on our immediate roadmap. 

- The migration path should be automated from the back end when the developer selects the HRD from the admin application configuration console.

Migrating from M/S to High Rep with out loss of data will always require re-writing all of an apps data. To do this while preserving data consistency requires a read-only period. Because of this we decided to give the developers direct control over the migration process. This allows for app specific optimization to reduce the read-only period (X kind needs to be copied perfectly, Y kind doesn't need to be perfect, Z kind is session data and can be ignored completely, etc). Out of the box, the solution we currently provide almost completely automates the actual copying of the data. We thought it was important to get the High Replication datastore to developers as soon as possible and we plan on smoothing out other aspects off the migration process, specifically, issues around creating or using a new app, in future releases.

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

Blogger: http://googleappengine.blogspot.com

Reddit: http://www.reddit.com/r/appengine

Twitter: http://twitter.com/app_engine

[Robert Kluin]

Hi Ikai,
One more question: What about people serving from appspot.com?
Changing the appid means customers need reeducated, and could
generally be really inconvenient for anyone with an established app.
Any thought on that issue?

Robert

[Stephen]

On Thursday, January 6, 2011 11:09:24 PM UTC, Ikai Lan (Google) wrote:

- Typically, how long does it take for HR query data to become consistent (a ballpark figure)?

100ms is the estimate. However - we have seen cases where this can be higher. If you were to use the eventual consistency read option in the master-slave configuration, we've seen that the average replication lag is about 3 minutes with an upper bound of 10 minutes.

What are the timings for the master/slave datastore with the EVENTUALLY_CONSISTENT option?

[Ikai Lan (Google)]

Stephen,

The times I gave earlier were estimates of how much replication delay is introduced in each replication scheme. The "eventually consistent" flag is for reads only and dictates whether or not you care to read from the "Master" datastore - reads will go to a slave if there are issues reaching the master.

Robert, in future releases, we will allow developers to create aliases for their applications. Aliases are routing rules that will override the current routing. Unfortunately, these are not production ready, as we don't have any ACLs in place and there are edge cases. For developers that want to migrate to HR datastore right away and need an alias created (say, for HTTPS) and are willing to weather some weirdness around mail, XMPP and cron, I am happy to create the alias for them given that they can prove they own both applications (source and destination application). These developers can fill out a request for an application ID alias here:

http://code.google.com/support/bin/request.py?contact_type=AppEngineBillingSupport

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

Blogger: http://googleappengine.blogspot.com

Reddit: http://www.reddit.com/r/appengine

Twitter: http://twitter.com/app_engine

--

[Robert Kluin]

Thanks Ikai.

[Stephen]

On Friday, January 7, 2011 12:04:36 AM UTC, Ikai Lan (Google) wrote:

Stephen,

The times I gave earlier were estimates of how much replication delay is introduced in each replication scheme. The "eventually consistent" flag is for reads only and dictates whether or not you care to read from the "Master" datastore - reads will go to a slave if there are issues reaching the master.

Ah, I misread your original answer as relating only to the high replication datastore - thanks.

This is kind of a surprising answer: replication between data centres (hi-rep) takes 100ms; replication within a data centre (M/S) takes 3000-10000ms.  I suppose this is a trade-off: because the default for MS is strong consistency you can sacrifice replication lag for higher throughput...?

After I asked the question I came across the original blog announcement for the EVENTUALLY_CONSISTENT option, and it already contains the answer:

  http://googleappengine.blogspot.com/2010/03/read-consistency-deadlines-more-control.html

  "The secondary location may not have all of the changes made to the primary location at the time the data is read, but it should be very close. In the most common case, it will have all of the changes, and for a small percentage of requests, it may be a few hundred milliseconds to a few seconds behind."

Which is somewhat different than an avg of 3000ms and up to 10000ms. The figures from the blog post suggests to me that a query with the EVENTUALLY_CONSISTENT flag is basically not-quite-transactional, where as the latest figures of 3-10mins sounds more like stale results. Is this a policy change?

Regardless, it would be good to have these figures directly in the docs to help folks decide when they can use this feature and to decide between hi-rep and m/s.

Couple of other questions related to this:

- What percentage of reads block due to unavailability of a primary? That is, how often is setting the EVENTUALLY_CONSISTENT flag likely to make any difference at all?

- For reads which do block, what is the average wait time for successful waits and the failure rate for reads which timeout (without an explicitly set deadline)?

- And, how is unavailability of a primary determined? Is it a time out, and if so, how long? (I would use this figure to help determine a suitable deadline for queries which I want to fail over with eventual consistency.)

Thanks.

[Stephen Johnson]

In Ikai's post he says it takes
  "the average replication lag is about 3 minutes with an upper bound of 10 minutes." He says "minutes" not milliseconds.

   so that's not 3000ms to 10000ms, that is 180000ms to 600000ms

if I'm reading your comments correctly. So is it minutes or milliseconds???

[Stephen]

On Friday, January 7, 2011 5:51:06 PM UTC, Stephen Johnson wrote:

In Ikai's post he says it takes
  "the average replication lag is about 3 minutes with an upper bound of 10 minutes." He says "minutes" not milliseconds.

   so that's not 3000ms to 10000ms, that is 180000ms to 600000ms

if I'm reading your comments correctly. So is it minutes or milliseconds???

Yes, I should have wrote '3-10 minutes'.

[Ikai Lan (Google)]

Thanks for the update, Stephen (I am now realizing that there are 2 Stephens in this thread and that you were not having a dialogue in public with yourself)! I misspoke earlier: the numbers I cited were latency numbers in the event of a failure in the Slave datastore. The actual numbers for the majority (99+%) of requests is still a few hundred milliseconds. In most cases, replication is even synchronous. It's in the error cases where the average replication lag is 3 minutes. 

- The number of errors is less than a fraction of a fraction of a fraction of a percent. I don't think a percent number makes sense here: it's the fact that the errors are not evenly distributed. That is - a request that blocks due to unavailability is likely to follow be several requests and lead to a sudden spike of instability in your application that may depend on many small, fast calls.

To answer your last two questions, it makes more sense to explain how eventually consistent reads work:

1. Fire off an RPC to the datastore service internally. 

2. Wait a grace period for this RPC to return. This grace period is in the tens of milliseconds. Most requests (99+%) will return in a time that is far, far under this grace period

3. If the grace period has been surpassed, we send out an RPC to the slave datastore. Now - we just wait to see who returned first: the original RPC to the master datastore or the second RPC to the slave datastore. Again, in most cases, the master datastore is likely to respond first simply because it was given a head start

I really appreciate you asking for clarification. This is confusing, and your questions have given us some good direction about how to improve the documentation. 

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

Blogger: http://googleappengine.blogspot.com

Reddit: http://www.reddit.com/r/appengine

Twitter: http://twitter.com/app_engine

[Waldemar Kornewald]

Hi Ikai,

On Friday, January 7, 2011 12:09:24 AM UTC+1, Ikai Lan (Google) wrote:

Hey guys, some answers to your questions that we've compiled from various team members (mostly Alfred, presenter of "Next Gen Queries" and the fourth one from the left in this video http://www.youtube.com/watch?v=dg0TEIRQePg):

- Typically, how long does it take for HR query data to become consistent (a ballpark figure)?

100ms is the estimate. However - we have seen cases where this can be higher. If you were to use the eventual consistency read option in the master-slave configuration, we've seen that the average replication lag is about 3 minutes with an upper bound of 10 minutes.

Does the eventual consistency lag only affect DB indexes (i.e. the query results contain up-to-date entities as if fetched via db.get(), but they might not really match the query, anymore)? Or does the lag affect the actual entity data returned by the query (i.e. the results contain *outdated* entities, IOW returning older data than db.get())?

Bye,

Waldemar Kornewald

[Ikai Lan (Google)]

The replication lag affects actual entity data as well as indexes. 

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

Blogger: http://googleappengine.blogspot.com

Reddit: http://www.reddit.com/r/appengine

Twitter: http://twitter.com/app_engine

--

[Waldemar Kornewald]

On Friday, January 7, 2011 9:08:02 PM UTC+1, Ikai Lan (Google) wrote:

The replication lag affects actual entity data as well as indexes. 

According to the docs Get/Delete/Put are strongly consistent, so in order to get up-to-date entity data can I run a keys_only query and then use db.get() to fetch the latest results?

Is get() only consistent for single-key calls or also for a list of keys?

Bye,

Waldemar

[Stephen]

OK. So according to 2 and 3, almost always (99+%) you will hit a
master server, so specifying eventual consistency makes no difference.
So for the MS datastore this flag affects robustness not performance,
right?

Couple more questions...

What about the HR datastore -- are queries across entity groups also
likely to be 99+% consistent, or is the figure different?

As cross-entity group HR queries are eventually consistent by default,
is there ever a reason to use the eventually consistent flag, such as
when querying a single entity group?

Is there any difference between queries, gets, puts or deletes related
to consistency, HR or MS datastore, or do the rules apply equally no
matter the datastore operation?

Thanks for taking the time to explain this so far -- much appreciated.

[Ikai Lan (Google)]

Hey Waldemar, sorry for missing your original question. Okay, let me attempt to answer these questions:

1. According to the docs Get/Delete/Put are strongly consistent, so in order to get up-to-date entity data can I run a keys_only query and then use db.get() to fetch the latest results?

No. The problem is that indexes across multiple entity groups are eventually consistent, so though db.get will return the newest version of that entity (it is within a transaction), the indexes themselves may be out of date and only return a partial or incorrect result set.

2. Is get() only consistent for single-key calls or also for a list of keys?

Both. In another thread, a user demonstrated that a batch get of 500 root entities was significantly more expensive than with master-slave datastore. 500 transactions are needed. If you enable eventual consistency, transactions are not used and performance is much more similar to master-slave.

3. OK. So according to 2 and 3, almost always (99+%) you will hit a

master server, so specifying eventual consistency makes no difference.
So for the MS datastore this flag affects robustness not performance,
right?

I'm not entirely clear what is being asked here. Can you provide an example? If you see my answer to question #2 in this post, it may clarify the benefits of the eventual consistency setting.

4. What about the HR datastore -- are queries across entity groups also

likely to be 99+% consistent, or is the figure different?

We don't provide a % consistency, as this makes no sense. If you just made a massive data update prior to the query, the data will likely not be consistent. If you made a small number of chances, the data likely will be. 

The 99% I quoted earlier refers to the amount of time most queries should be consistent after, say, 100ms - 99.99(9?)% of the time, we will replicate data fast enough so that queries are fairly consistent, but in that fraction of a fraction of a percent of the time, a large number of queries will be inconsistent. 

5. As cross-entity group HR queries are eventually consistent by default,

is there ever a reason to use the eventually consistent flag, such as
when querying a single entity group?

If you don't want transaction overhead when querying by key you may want to enable this flag, but for most intents and purposes you probably don't need the eventually consistent flag.

6. Is there any difference between queries, gets, puts or deletes related

to consistency, HR or MS datastore, or do the rules apply equally no
matter the datastore operation?

I'm going to avoid answering this question since this thread should answer most of your questions. If there's anything you're still unclear about, please post a follow-up.

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

Blogger: http://googleappengine.blogspot.com

Reddit: http://www.reddit.com/r/appengine

Twitter: http://twitter.com/app_engine