Performance comparison between Master/Slave and High Replication

[风笑雪]

I've just run a simple test of performance comparison between Master/Slave and High Replication.

The test is running on 2 apps with 6 bytes length name, using the "Test" kind without any properties:

from time import time

from google.appengine.ext import db

class Test(db.Model):

pass

keys = [db.Key.from_path('Test', i) for i in xrange(1, 501)]

tests = [Test(key=key) for key in keys]

t = time()

db.put(tests)

print time() - t

t = time()

Test.all().fetch(500)

print time() - t

t = time()

db.get(keys)

print time() - t

t = time()

db.delete(keys)

print time() - t

Test result:

Master/Slave:

1.52360796928

0.141813993454

0.147518157959

1.14946293831

ms=3060 cpu_ms=44519 api_cpu_ms=43446 cpm_usd=1.236788

High Replication:

4.77071118355

0.235147953033

3.09301185608

5.15948295593

ms=13304 cpu_ms=114004 api_cpu_ms=112812 cpm_usd=3.166840

As we can see, High Replication uses more quotas and time than Master/Slave (especially for db.get() which is 20x slower).

So I think the only benefit High Replication offers now is lower error rate.

----------
keakon

My blog(Chinese): www.keakon.net

Blog source code: https://bitbucket.org/keakon/doodle/

[Raymond C.]

thanks so much for the info.

[Ikai Lan (Google)]

These numbers look wrong to me. Let me investigate and get back to you guys on this.

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

Blogger: http://googleappengine.blogspot.com

Reddit: http://www.reddit.com/r/appengine

Twitter: http://twitter.com/app_engine

On Thu, Jan 6, 2011 at 3:33 AM, Raymond C. <wind...@gmail.com> wrote:

thanks so much for the info.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Ikai Lan (Google)]

Alright folks - I've gotten to the bottom of this. Here's an explanation below.

Short answer:

The write performance is in line with what we expect. The get performance is to be expected when you do a batch get with keys of entities in different entity groups. 

Long answer:

How writes and reads work in Master-Slave

--------------------

This code:

keys = [db.Key.from_path('Test', i) for i in xrange(1, 501)]

tests = [Test(key=key) for key in keys]

db.put(tests)

Test.all().fetch(500)

1. We write the entity to the datastore by writing to a log. As far as we are concerned, the write has now completed, and we return to the user.

2. In the background, the write is applied to the actual datastore based on data in the log. The unit of write is atomic to an entity group. You can think of this as the commit step.

The time between steps 1 and 2 is tiny. For most intents and purposes, it's almost instantaneous. While it's theoretically possible a read operation can come in between steps 1 and 2, in practice, this never happens, so it appears to be consistent.

How writes and reads work in High Replication

---------------------

1. The new data is written to the log. To ensure consistency, we must write the data to the logs of a majority of data centers (this is how Paxos works). This is what adds the additional write latency required for HR to guarantee consistency. It is possible that 1 or more data centers do not receive the log updates; this is okay because we have consensus.

2. Each individual datastore in each data center is responsible for applying the changes in the log.

When a batch get is issued, it's possible it gets routed to a data center that hasn't yet received the write in its log. For this reason, we need to transact on the log entry across data centers to ensure that we are returning the freshest data for consistency reasons. In the sample code above, there are 500 root entities, so 500 transactions need to occur. This is the key difference between a get in Master-Slave replication versus High Replication: it must happen in a transaction. If you remember how entity group transactions work, you'll remember that they happen by reading the log entry for the root entity. If the code above did a batch get on 500 entities in the same entity group, it would run considerably faster because only one transactional read needs to take place.

You can improve get performance at the expense of consistency by setting the read_policy to eventually consistent. Here's a code sample:

t = time()

config = db.create_config(deadline=5, read_policy=db.EVENTUAL_CONSISTENCY)

db.get(keys, config=config)

print "Time it takes to perform a batch fetch with eventual consistency"              

print time() - t

The performance of the code sample above will be similar to that of Master-Slave datastore. 

In the original benchmark, you might notice that query performance across entity groups is significantly faster - that's because queries assume eventual consistency and are not executed in a transaction like get-by-key operations. See the Usage Notes under the High Replication documentation for more details (http://code.google.com/appengine/docs/python/datastore/hr/overview.html).

We're working on improving the performance of this case by making a batch get read from multiple data centers, but this will likely not be available until one of the next SDK releases.

Feel free to ask any questions you may have about the explanation.

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

Blogger: http://googleappengine.blogspot.com

Reddit: http://www.reddit.com/r/appengine

Twitter: http://twitter.com/app_engine

[Stephen]

On Thursday, January 6, 2011 10:06:03 PM UTC, Ikai Lan (Google) wrote:

In the original benchmark, you might notice that query performance across entity groups is significantly faster - that's because queries assume eventual consistency and are not executed in a transaction like get-by-key operations.

Is this right?  The documentation states that STRONG_CONSISTENCY is the default (and this matches the code in the SDK):

http://code.google.com/appengine/docs/python/datastore/queries.html#Setting_the_Read_Policy_and_Datastore_Call_Deadline


Also, the last paragraph on this page contradicts itself:

"A configuration object can be used any number of times. You must create a separate configuration object for each datastore call that uses it."

And further up the page in the section "Queries and Indexes" the link and explanation of KEY_RESERVED_PROPERTY is Java-specific.

[风笑雪]

Hi Ikai,

Thanks for your explanation.

I've changed my code and retest it, the result is here:

5.79944705963

0.194540023804

0.238615989685

5.65248513222

ms=11965 cpu_ms=114077 api_cpu_ms=112812 cpm_usd=3.168863

I can see the batch db.get() becoming much faster than before.

BTW, db.create_config() doesn't be documented in Datastore functions, I think the document should be updated.

----------
keakon

My blog(Chinese): www.keakon.net

Blog source code: https://bitbucket.org/keakon/doodle/

On Fri, Jan 7, 2011 at 6:06 AM, Ikai Lan (Google) <ikai.l...@google.com> wrote:

config = db.create_config(deadline=5, read_policy=db.EVENTUAL_CONSISTENCY)

[Raymond C.]

Thank Ikai for the information.  But I am curious is it true that we have to have such poor performance on .get() with entities on different entity group if eventual consistent result is not acceptable in our use cases?  If this is so I think it is a killing issue which should be emphasized on the docs together with the other cons of the option.   

 

[Matija]

Hi Keakon,

could you put benchmark results for master/slave and high replication datastore with 20 entities (for get and fetch) and 1 entity for put and delete instead of 500. We present data in 20 entity size batch and save only one entity (or entity group) per request so to us this would be more realistic benchmark. Please use original code without single entity group.

Tnx, MATijA.

[风笑雪]

Hi Matija,

Code:

from time import time

from google.appengine.ext import db

class Test(db.Model):

pass

keys = [db.Key.from_path('Test', i) for i in xrange(1, 21)]

tests = [Test(key=key) for key in keys]

db.put(tests)

t = time()

Test.all().fetch(20)

print 'fetch 20:', time() - t

t = time()

db.get(keys)

print 'get 20:', time() - t

t = time()

config = db.create_config(deadline=5, read_policy=db.EVENTUAL_CONSISTENCY)

db.get(keys, config=config)

print 'get 20 (EVENTUAL_CONSISTENCY):', time() - t

key = db.Key.from_path('Test', 21)

test = Test(key=key)

t = time()

test.put()

print 'put 1:', time() - t

db.delete(keys + [key])

Result:

Master/Slave:

fetch 20: 0.0107522010803

get 20: 0.0108997821808

get 20 (EVENTUAL_CONSISTENCY): 0.0107271671295

put 1: 0.0120298862457

ms=193 cpu_ms=2085 api_cpu_ms=2015 cpm_usd=0.058186

High Replication:

fetch 20: 0.0176739692688

get 20: 0.101014137268

get 20 (EVENTUAL_CONSISTENCY): 0.0169019699097

put 1: 0.0453717708588

ms=690 cpu_ms=5078 api_cpu_ms=4987 cpm_usd=0.141334

I think you can ignore the differences because the dataset is too small, but the time would be longer if you have many properties.

----------
keakon

My blog(Chinese): www.keakon.net

Blog source code: https://bitbucket.org/keakon/doodle/

--

[Matija]

Thank you.

I think that we will stay on master/slave version. Now storing one entity group (two entities) with many indexed properties and several composite indexes in master/slave datastore is at around 1500 cpu ms and average latency 300 ms. With 3 times more cost it will be at often over 1000ms average latency limit, not to mention problem of needed ancestors queries to present customer stored data in previous request. Maybe with next gen queries (I assume around may 2011 right before next IO) I will need smaller composite indexes and cost will be bearable. 

Matija.

[Philip]

Hi Ikai,

you mentioned that queries are faster on HR because they do not
guarantee consistency and therefore aren't using transactions. Is it
possible to get the same behaviour for some queries at Master/Slave
datastore?

Best Regards
Philip