Should Google deprecate Master/Slave data store option?

[Mike Lawrence]

Just burned several days with Google Enterprise Support trying to troubleshoot intermittent long latencies; 

unnecessary dynamic instance starts; and failures during dynamic instance starts on a multi-threaded GAE Java app

running on Master/Slave (MS) data store.   

My typical app latencies on MS were under 1 second per request. Periodically I'd see 5-20 second delays with very little traffic.

My app starts in under 3 seconds (stripes/slim3), so when instances failed to start in 30 seconds, I suspected a GAE issue.

After many attempts to resolve the issue, we finally decided to to try switching to the High Replication (HR) data store.

All of the issues disappeared!!! The long latencies were all gone. My traffic was easily being served with a single instance.

No more phantom dynamic instances. No more instance start crashes.

I just volume tested my app. My median request went from 2.5 seconds on MS to under 1 second on HR with 1000 simultaneous users.

Previously I was paying for 3 always-on instances. With HR enabled, there was no need for any always-on instances.

So if High Replication is faster, and more stable, why should Google spend valuable resources supporting Master/Slave?

Is the cost savings of MS worth all the issues it creates? Not for me.

If you're experiencing similar problems on MS I highly recommend giving HR a shot.

FYI, migration to HR was trivial: 

- created a new app id, 

- deployed my app to the new id, 

- used my app to initialize the data (this will be problematic for some), 

- had google point the old app id to the new app id.

now serving from the original URL.

[Ikai Lan (Google)]

Wow, thanks for the update, Mike!

HR isn't 100% problem free yet. We're still working through some of the issues. Addressing infrastructure issues is one of our top priorities right now, but we can only do so much for master/slave, which has been through a bit of a rough patch lately.

I do want to urge people to look at migrating. We're getting really close to releasing a self-migration tool, but developers are still going to have to do some legwork. Make sure you understand the consistency guarantees of HR datastore! I'm hearing a LOT of misconceptions from people who don't understand get-by-key or ancestor questions. I recently did a talk about this. Check out these slides here (they're in English; don't worry about the title slide):

http://www.slideshare.net/ikailan/2011-auggddmexicocityhighreplicationdatastore

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com | twitter.com/ikai

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/XCUAKZ5E1LYJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Joshua Smith]

I certainly hope they don't deprecate M/S, unless/until the following issues are addressed:

1) Ability to migrate without changing app ID. This is fundamental, and we've never seen any *reason* why this is impossible. I suspect that this is very hard, but not impossible. Changing the app ID causes problems because:

- External references to entity keys (not IDs) are pervasive in many apps (such as google search links to deep pages, user bookmarks, or RSS feed URLs) because the documentation encourages the use of these keys in URLs

- Many users have keys wrapped up in properties that the migration tool cannot recognize [this doesn't impact my app, but it's a common issue on this list]

As an alternative, the infrastructure could be fixed to recognize the case of using the old app ID instead of the new app ID in a key, and quietly make the substitution.

2) Migration of large data sets takes an extremely long time. Given that we regularly see M/S downtimes in the couple of hours range, a couple of hours is probably a reasonable amount of time to migrate. A couple of days is much too much. [Note that I have not personally gotten far enough into a migration to see this duration; I'm basing this on many user complaints I've read on this list.]

3) The problem of transactional consistency needs to be addressed. It is not acceptable for transactions to end in a "maybe this worked, maybe it didn't" state. They should either succeed or fail. (Although the docs not not say whether this problem is exclusive to HR, it seems to be, and certainly it matters much more in HR since every put is implicitly transactional.)

4) As Ikai said, there are some unresolved issues in HR that need to be fixed, such as the current mystery in the 'Serious Problem: Rollback of data on HRD' thread (which is eerily familiar: I think somebody else reported a similar issue a while back).

Even if all these are addressed, it would certainly be nice if M/S remained an option. It makes different trade-offs than HR, which may be better trade-offs for some applications. I will be very disappointed if they decide to make HR a requirement for multi-threading Python, since the two are completely orthogonal choices. From my reading, M/S seems to have better throughput than HR, at the expense of more frequent downtime, and spurious datastore timeouts. I'd like to have that option in a multi-threaded app.

-Joshua

[Renan Mobile]

Thanks Joshua,

I really like M/S, i don't have reasons to migrate my first application to HR. If it's possible i would like stay on M/S.

Although I have a plan to build a new application that will works on HR, because the specific business logic.

Oh! I'm a java developer.

Atenciosamente (thanks),

-----
Presidente Renan Franca
____
Renan Mobile ltda: http://renanmobile.com - Soluções em Dispositivos Móveis (Smartphones) com integração via web.
Soluções em:
        Android (Smartphones);
        Google Web Toolkit (Web);

2011/8/17 Joshua Smith <Joshua...@charter.net>

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

[Ikai Lan (Google)]

Master/Slave doesn't have better throughput. In the 99th percentile, because of datastore latency spikes, high replication far outperforms master/slave.

I don't think there are plans to completely deprecate master/slave, but there's been serious talk of closing off the creation of new master/slave applications. 

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com | twitter.com/ikai

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

[Renan Mobile]

Thanks Ikai,

I totally agree with you about prohibit new M/S applications.

Thanks and let's getting better always.  I really trust on Google.
--
Atenciosamente(thanks),
                 Renan Franca
-----
Presidente
____
- Renan Mobile ltda: http://renanmobile.com - Soluções em Dispositivos Móveis (Smartphones) com integração via web.
- Soluções em:

         Android (Smartphones);
         Google Web Toolkit (Web);

[Darien Caldwell]

On Aug 17, 5:24 am, Joshua Smith <JoshuaESm...@charter.net> wrote:
> I certainly hope they don't deprecate M/S, unless/until the following issues are addressed:

I agree. It would also be nice if there was some documentation
detailing some of the things people tend to gloss over. Like

"- used my app to initialize the data (this will be problematic for
some),
- had google point the old app id to the new app id. "

A bit too much like " ??? Profit" for my taste. Maybe the tool Ikai
mentioned will resolve most of that. Until then, M/S is where I'll
stay.

[Fred Sauer]

On Wednesday, August 17, 2011 5:24:48 AM UTC-7, Joshua Smith wrote:

I certainly hope they don't deprecate M/S, unless/until the following issues are addressed:

1) Ability to migrate without changing app ID.  This is fundamental, and we've never seen any *reason* why this is impossible.  I suspect that this is very hard, but not impossible.

We are able to create an alias for migrated apps, so that traffic sent to oldappid.appspot.com will be transparently routed to your new application. So while you won't be able to keep the same app id, you can keep using the same appspot.com domain, which is probably what you care about most.

 

 Changing the app ID causes problems because: 

- External references to entity keys (not IDs) are pervasive in many apps (such as google search links to deep pages, user bookmarks, or RSS feed URLs) because the documentation encourages the use of these keys in URLs

This remains an issues, but can be easily addressed in a couple of lines of application code in the appropriate request handlers. The string encoded keys that are referenced externally have encoded within them the old app id. If you take the decoded key and extract the parent, kind and id/name to create a new key (again, this is 1-2 lines of code), then your app can use that new key to locate the migrated entity within the new datastore.

As far as reference (list) properties: those reference keys are automatically updated as part of the the datastore migration and should not cause you any issues.

 

- Many users have keys wrapped up in properties that the migration tool cannot recognize [this doesn't impact my app, but it's a common issue on this list]

This does affect some apps. As I mentioned above, a couple of lines of code in the right place should take care of this.

 

As an alternative, the infrastructure could be fixed to recognize the case of using the old app ID instead of the new app ID in a key, and quietly make the substitution.

That's a neat idea.

 

2) Migration of large data sets takes an extremely long time.  Given that we regularly see M/S downtimes in the couple of hours range, a couple of hours is probably a reasonable amount of time to migrate.  A couple of days is much too much.  [Note that I have not personally gotten far enough into a migration to see this duration; I'm basing this on many user complaints I've read on this list.]

Applications with small amounts of data can easily self migrate in a short period of time using datastore_admin. Applications with more data will benefit from the new migration tool Ikai mentioned. We created the tool specifically to simplify and speed up the migration for larger apps, and to reduce the read-only period required to make the final switch.

 

3) The problem of transactional consistency needs to be addressed.  It is not acceptable for transactions to end in a "maybe this worked, maybe it didn't" state.  They should either succeed or fail.  (Although the docs not not say whether this problem is exclusive to HR, it seems to be, and certainly it matters much more in HR since every put is implicitly transactional.)

There are subtle differences between the consistency guarantees offered by M/S and HRD. In some cases, apps need to make a few tweaks in order to work correctly in the new environment. The benefit is much more predictable/consistency latency and increased reliability. A good article which reviews the consistency guarantees can be found here:

  http://code.google.com/appengine/articles/transaction_isolation.html

4) As Ikai said, there are some unresolved issues in HR that need to be fixed, such as the current mystery in the 'Serious Problem: Rollback of data on HRD' thread (which is eerily familiar: I think somebody else reported a similar issue a while back).

I haven't looked at that thread yet, but I suspect that reviewing the above article (http://code.google.com/appengine/articles/transaction_isolation.html) would probably be beneficial in the context of that thread.

Even if all these are addressed, it would certainly be nice if M/S remained an option.  It makes different trade-offs than HR, which may be better trade-offs for some applications.  I will be very disappointed if they decide to make HR a requirement for multi-threading Python, since the two are completely orthogonal choices.  From my reading, M/S seems to have better throughput than HR, at the expense of more frequent downtime, and spurious datastore timeouts.  I'd like to have that option in a multi-threaded app.

Thanks for the input. I hope the above response addresses some of your concerns.

 

-Joshua

[Joshua Smith]

On Aug 18, 2011, at 2:10 PM, Fred Sauer wrote:

On Wednesday, August 17, 2011 5:24:48 AM UTC-7, Joshua Smith wrote:

I certainly hope they don't deprecate M/S, unless/until the following issues are addressed:

1) Ability to migrate without changing app ID.  This is fundamental, and we've never seen any *reason* why this is impossible.  I suspect that this is very hard, but not impossible.

We are able to create an alias for migrated apps, so that traffic sent to oldappid.appspot.com will be transparently routed to your new application. So while you won't be able to keep the same app id, you can keep using the same appspot.com domain, which is probably what you care about most.

Perhaps that what some people care about most, but not me.  What I care about is what I spelled out in the next two bullet points :)

 Changing the app ID causes problems because: 

- External references to entity keys (not IDs) are pervasive in many apps (such as google search links to deep pages, user bookmarks, or RSS feed URLs) because the documentation encourages the use of these keys in URLs

This remains an issues, but can be easily addressed in a couple of lines of application code in the appropriate request handlers. The string encoded keys that are referenced externally have encoded within them the old app id. If you take the decoded key and extract the parent, kind and id/name to create a new key (again, this is 1-2 lines of code), then your app can use that new key to locate the migrated entity within the new datastore.

As far as reference (list) properties: those reference keys are automatically updated as part of the the datastore migration and should not cause you any issues.

- Many users have keys wrapped up in properties that the migration tool cannot recognize [this doesn't impact my app, but it's a common issue on this list]

This does affect some apps. As I mentioned above, a couple of lines of code in the right place should take care of this.

1-2 lines of code, in dozens of handlers is the crux of the problem.  Dealing with this at the application layer is problematic because there are so many code paths that need to be examined.

As an alternative, the infrastructure could be fixed to recognize the case of using the old app ID instead of the new app ID in a key, and quietly make the substitution.

That's a neat idea.

Neat enough that you guys might implement it?

I've suggested this before, but I'm glad that I've got your attention this time!

This would be a pretty huge step toward making the migration practical for me.

2) Migration of large data sets takes an extremely long time.  Given that we regularly see M/S downtimes in the couple of hours range, a couple of hours is probably a reasonable amount of time to migrate.  A couple of days is much too much.  [Note that I have not personally gotten far enough into a migration to see this duration; I'm basing this on many user complaints I've read on this list.]

Applications with small amounts of data can easily self migrate in a short period of time using datastore_admin. Applications with more data will benefit from the new migration tool Ikai mentioned. We created the tool specifically to simplify and speed up the migration for larger apps, and to reduce the read-only period required to make the final switch.

Yes, I saw that in his note, and if it works as advertised, I suspect it will resolve this issue.

3) The problem of transactional consistency needs to be addressed.  It is not acceptable for transactions to end in a "maybe this worked, maybe it didn't" state.  They should either succeed or fail.  (Although the docs not not say whether this problem is exclusive to HR, it seems to be, and certainly it matters much more in HR since every put is implicitly transactional.)

There are subtle differences between the consistency guarantees offered by M/S and HRD. In some cases, apps need to make a few tweaks in order to work correctly in the new environment. The benefit is much more predictable/consistency latency and increased reliability. A good article which reviews the consistency guarantees can be found here:

  http://code.google.com/appengine/articles/transaction_isolation.html

I'm not talking about consistency guarantees.  I get that things change under HR, and it's pretty easy to understand what the implications of that are.

What I'm talking about is the problem where you have to write idempotent transactions because you can never tell whether a transaction that threw certain exceptions succeeded or failed.  That's really messed up, and you really need to fix that.

But, to be fair, I'm not sure this problem will cause me actual problems if I switch to HR.  It just makes me really, really uncomfortable.

4) As Ikai said, there are some unresolved issues in HR that need to be fixed, such as the current mystery in the 'Serious Problem: Rollback of data on HRD' thread (which is eerily familiar: I think somebody else reported a similar issue a while back).

I haven't looked at that thread yet, but I suspect that reviewing the above article (http://code.google.com/appengine/articles/transaction_isolation.html) would probably be beneficial in the context of that thread.

Check out the thread.  It's spooky weird behavior that he is reporting, well beyond the stuff in that article.

Even if all these are addressed, it would certainly be nice if M/S remained an option.  It makes different trade-offs than HR, which may be better trade-offs for some applications.  I will be very disappointed if they decide to make HR a requirement for multi-threading Python, since the two are completely orthogonal choices.  From my reading, M/S seems to have better throughput than HR, at the expense of more frequent downtime, and spurious datastore timeouts.  I'd like to have that option in a multi-threaded app.

Thanks for the input. I hope the above response addresses some of your concerns.

Ikai pointed out that MS doesn't really have higher throughput, so I guess keeping the option is not needed if the other issues are resolved.

So, to boil this down to the bits that are critical:

- Implement the "neat idea" of fixing wrong app IDs within the infrastructure;

- Get the new migration tool out of trusted tester status;

- Resolve the outstanding HR issues, such as the one in the Rollback thread

Solve those, and I can migrate.

Thanks for getting back to me on this!!!

-Joshua

[thstart]

Let me share my experience. 

I was developing an AppEngine app for a year.

It was very fast (extremely fast), with latency > 1 sec.

When High Replication data-store was released I decided to give it

a try and switched to it. Unfortunately the latency got >1 Sec 

sometimes > 2 Sec. The app worked a wile that way and because 

still in beta I could experiment. So I decided to go to Billed option

to see if the reliability and more importantly the latency could improve.

Unfortunately this do not affected latency. Then I decided to go back to 

Master/Slave - our customers mentioned immediately our app is 2 times 

faster - again < 1 Sec latency. Then I needed to add https which is 

very important for our app. This do not added significant latency so our

customers are very happy with this solution. Soon we will go public.

So the latency issue is a show stopper for high replication version 

and we would want to stay with master slave. 

Anybody with similar experience?

Thank you,

[Robert Kluin]

When switching to the HR datastore, you might need to make some
adjustments. For instance, fetching a large list of keys from
multiple entity groups may be slower due to the transaction overhead.
So you might need to evaluate your use of entity groups. It may also
be that you've got a design that is making many datastore RPCs in
serial, that will probably be more of an issue when on HR since there
is a slightly higher average latency.

Robert

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-appengine/-/P1c_gMSRtl0J.

[Fred Sauer]

That's accurate. Performing a batch get to get a N entities from different entity groups is slower or HRD. As N increases the issue is more pronounced. This is because the datastore needs to fetch a consistent copy for each entity.

Placing these entities into a single entity group can help greatly. Fetching fewer entities initially, so that the UI updates quickly, and then fetching more as needed, is always a good idea.

A great tool to help understand your use of the datastore is Appstats, which can be easily enabled and provides negligible overhead to most sites. (If you have a high traffic site, or don't want to enable Appstats in production for other reasons, you can easily deploy a non-default version of your app with Appstats enabled). Appstats will provide you a graphical overview of the datastore (and other RPC) calls you are making.

--

Fred Sauer

Developer Advocate

Google Inc.

1600 Amphitheatre Parkway

Mountain View, CA 94043
fre...@google.com

[Fred Sauer]

I recommend looking at Appstats, and verifying whether you are performing batch gets across entity groups. I have not met an app yet that is unable to get acceptable HRD performance.

I should note that apps built on M/S have (usually unintentionally) been optimized for the M/S performance characteristics. It makes sense that some apps may need to or want to make adjustments before migrating to HRD.

I should also point out that I have yet to meet a developer whose not happier on HRD and migrating their app. You may well be the first (although I'm not trying to imply that you are the only one)... :)

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/P1c_gMSRtl0J.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Fred Sauer]

On Thu, Aug 18, 2011 at 11:48 AM, Joshua Smith <Joshua...@charter.net> wrote:

On Aug 18, 2011, at 2:10 PM, Fred Sauer wrote:

On Wednesday, August 17, 2011 5:24:48 AM UTC-7, Joshua Smith wrote:

I certainly hope they don't deprecate M/S, unless/until the following issues are addressed:

1) Ability to migrate without changing app ID.  This is fundamental, and we've never seen any *reason* why this is impossible.  I suspect that this is very hard, but not impossible.

We are able to create an alias for migrated apps, so that traffic sent to oldappid.appspot.com will be transparently routed to your new application. So while you won't be able to keep the same app id, you can keep using the same appspot.com domain, which is probably what you care about most.

Perhaps that what some people care about most, but not me.  What I care about is what I spelled out in the next two bullet points :)

 Changing the app ID causes problems because: 

- External references to entity keys (not IDs) are pervasive in many apps (such as google search links to deep pages, user bookmarks, or RSS feed URLs) because the documentation encourages the use of these keys in URLs

This remains an issues, but can be easily addressed in a couple of lines of application code in the appropriate request handlers. The string encoded keys that are referenced externally have encoded within them the old app id. If you take the decoded key and extract the parent, kind and id/name to create a new key (again, this is 1-2 lines of code), then your app can use that new key to locate the migrated entity within the new datastore.

As far as reference (list) properties: those reference keys are automatically updated as part of the the datastore migration and should not cause you any issues.

- Many users have keys wrapped up in properties that the migration tool cannot recognize [this doesn't impact my app, but it's a common issue on this list]

This does affect some apps. As I mentioned above, a couple of lines of code in the right place should take care of this.

1-2 lines of code, in dozens of handlers is the crux of the problem.  Dealing with this at the application layer is problematic because there are so many code paths that need to be examined. 

As an alternative, the infrastructure could be fixed to recognize the case of using the old app ID instead of the new app ID in a key, and quietly make the substitution.

That's a neat idea.

Neat enough that you guys might implement it?

I've suggested this before, but I'm glad that I've got your attention this time!

I'll certainly bring up with the team.

 

This would be a pretty huge step toward making the migration practical for me.

2) Migration of large data sets takes an extremely long time.  Given that we regularly see M/S downtimes in the couple of hours range, a couple of hours is probably a reasonable amount of time to migrate.  A couple of days is much too much.  [Note that I have not personally gotten far enough into a migration to see this duration; I'm basing this on many user complaints I've read on this list.]

Applications with small amounts of data can easily self migrate in a short period of time using datastore_admin. Applications with more data will benefit from the new migration tool Ikai mentioned. We created the tool specifically to simplify and speed up the migration for larger apps, and to reduce the read-only period required to make the final switch.

Yes, I saw that in his note, and if it works as advertised, I suspect it will resolve this issue.

3) The problem of transactional consistency needs to be addressed.  It is not acceptable for transactions to end in a "maybe this worked, maybe it didn't" state.  They should either succeed or fail.  (Although the docs not not say whether this problem is exclusive to HR, it seems to be, and certainly it matters much more in HR since every put is implicitly transactional.)

There are subtle differences between the consistency guarantees offered by M/S and HRD. In some cases, apps need to make a few tweaks in order to work correctly in the new environment. The benefit is much more predictable/consistency latency and increased reliability. A good article which reviews the consistency guarantees can be found here:

  http://code.google.com/appengine/articles/transaction_isolation.html

I'm not talking about consistency guarantees.  I get that things change under HR, and it's pretty easy to understand what the implications of that are.

What I'm talking about is the problem where you have to write idempotent transactions because you can never tell whether a transaction that threw certain exceptions succeeded or failed.  That's really messed up, and you really need to fix that.

But, to be fair, I'm not sure this problem will cause me actual problems if I switch to HR.  It just makes me really, really uncomfortable.

4) As Ikai said, there are some unresolved issues in HR that need to be fixed, such as the current mystery in the 'Serious Problem: Rollback of data on HRD' thread (which is eerily familiar: I think somebody else reported a similar issue a while back).

I haven't looked at that thread yet, but I suspect that reviewing the above article (http://code.google.com/appengine/articles/transaction_isolation.html) would probably be beneficial in the context of that thread.

Check out the thread.  It's spooky weird behavior that he is reporting, well beyond the stuff in that article.

Even if all these are addressed, it would certainly be nice if M/S remained an option.  It makes different trade-offs than HR, which may be better trade-offs for some applications.  I will be very disappointed if they decide to make HR a requirement for multi-threading Python, since the two are completely orthogonal choices.  From my reading, M/S seems to have better throughput than HR, at the expense of more frequent downtime, and spurious datastore timeouts.  I'd like to have that option in a multi-threaded app.

Thanks for the input. I hope the above response addresses some of your concerns.

Ikai pointed out that MS doesn't really have higher throughput, so I guess keeping the option is not needed if the other issues are resolved.

So, to boil this down to the bits that are critical:

- Implement the "neat idea" of fixing wrong app IDs within the infrastructure;

- Get the new migration tool out of trusted tester status;

- Resolve the outstanding HR issues, such as the one in the Rollback thread

Solve those, and I can migrate.

Thanks for getting back to me on this!!!

-Joshua

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[thstart]

I am responding here to combined comments:

I followed Brett Slatikn recommendations how to perform most efficient operations on DataStore and it works extremely fast 

in Master / Slave.

+Robert Kluin

> For instance, fetching a large list of keys from multiple entity groups may be slower due to the transaction overhead.

I have list of keys only from one entity.

>So you might need to evaluate your use of entity groups.  It may also

>be that you've got a design that is making many datastore RPCs in

>serial, that will probably be more of an issue when on HR since there

>is a slightly higher average latency.

I store keys in lists. When getting the list 

for example

class MyList(BaseModel):

    My_keys = db.ListProperty(db.Key) 

Brett Slatkin recommends about no more than 3,000 max and so do I.

and this is parallel:

            e = MyList.get_profile(email)

                my_keys = e.my_keys

                my_key = db.get(my_keys)

    #That's it! Super fast!

    @classmethod

    def get_profile(self, email):

        if not email:

            return None

        root = MyRoot.get_profile()

        #cache

        self._cache = get_cache("memcached://")

        skey = email

        key_cache = 'MyList' + skey + str(root.key())

        profile = self._cache.get(key_cache)

        #cache        

        if not profile:

            key = db.Key.from_path('MyList',

                                   skey,

                                   parent=root.key())

            profile = db.get(key)

            # not found - create

            if not profile:

                #def txn():

                root = MyRoot.get_profile_del_cache()

                profile = MyList(key_name=skey,

                                       parent=root.key())

                profile.put()

                return profile

                #return db.run_in_transaction(txn)

            #cache

            self._cache.set(key_cache, profile, timeout=CACHE_TIMEOUT)

            #cache

        return profile

+Fred Sauer

>verifying whether you are performing batch gets across entity groups

I have transactions in which I have an operations on different entity groups but this is by design. 

>I should also point out that I have yet to meet a developer whose not happier on HRD and migrating their app. 

>You may well be the first (although I'm not trying to imply that you are the only one)... :)

The app is in development for more than a year and I followed best practices valid at this time before high replication

datastore was introduced. Redesign is not an option because I already have generated a lot of interest and waiting list

of customers ready to pay for the current functionality. The benefit I would get from going to High Replication does not 

justifies the effort: I have already 20,000 lines of code - redesign is not practical.

Best,

--Constantine

[thstart]

+Joshua Smith

>From my reading, M/S seems to have better throughput than HR, at the expense of more frequent downtime, 

>and spurious datastore timeouts.  I'd like to have that option in a multi-threaded app.

This is more important for public app (site) with many anonymous users. 

My app is 100% user based - e.g. used after log-in, but heavy on data related to each particular user. 

The users are not accessing it very often (say they access it many times a day but this is not much)

but when they access it they need to be as fast as possible.

So  frequent downtime is a non-issue, the chance a user to access it at exactly this time are minimal, 

spurious datastore timeouts is a non-issue because this is rare - so I can live with Master / Slave 

imperfections - it is good enough and I don;t need to rewrite and optimize again.

+Ikai Lan

>Master/Slave doesn't have better throughput. In the 99th percentile, because of datastore latency spikes, 

>high replication far outperforms master/slave.

I think Master/Slave is a lot faster in my case up to 10 times faster. I tested it with only read, no write operations,

no transactions, nothing special for example I have a heavy use of this expression:

            e = MyList.get_profile(email)

                my_keys = e.my_keys  

                my_members = db.get(my_keys)

                for member in my_members:

                        member.<property>, etc. <-- do something with the property

and it is parallel in Master/Slave (according to Brett Slatkin presentation) . In High Replication I believe it is serial,

probably I am wrong (I don't have a way to know) but this operation is very slow in High Replication.

My code uses heavily this construct and it is extremely fast.

Best,

--Constantine

[Ikai Lan (Google)]

If you enable eventually consistent queries, a batch get by key in high replication should perform as fast as in master/slave.

The practice you're citing from Brett is a few years old and predates high replication; would you be willing to revisit it if I roped him into updating it for high replication?

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com | twitter.com/ikai

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/DoYJ0jkPJ7cJ.

[thstart]

>If you enable eventually consistent queries, a batch get by key in high replication should perform as fast as in master/slave.

is this related only to queries like:

q = db.GqlQuery("SELECT * FROM Person " +
                "WHERE last_name = :1 AND height < :2 " +
                "ORDER BY height DESC",
                "Smith", 72)

http://code.google.com/appengine/docs/python/datastore/queries.html

The Master/Slave Datastore and the High Replication Datastore have different guarantees when it comes to query consistency. By default:

• The Master/Slave datastore is strongly consistent for all queries.
• The High Replication datastore is strongly consistent by default for queries within an entity group. With the High Replication Datastore, non-ancestor queries are always eventually consistent.

If that is the case I managed to get rid of queries - I don't use any any queries, just working with lists and keys - 

it is parallel, fast, don't need indexes, etc. Bret Slatkin had very useful presentations and found them to be very simple

and to work in practice.

By the way our customers don't know what technique I am using only thing they care is to be fast and they are telling me

up to now they haven't used a web app with so much functionality to be so fast. 

I would say they to be fast is that I designed the database as separate entity for each user - e.g. it scales with numbers of users

automatically. Very important is that I don;t use any queries, indexes, etc. just lists of keys and for ... in construct for search,

also the keys and entities are composed in as many ways possible in advance so when the user is searching for data it gets

it instantly because search is embedded in the keys. In Master/Slave all this worked in parallel and was extremely fast.

>The practice you're citing from Brett is a few years old and predates high replication; would you be willing to revisit it if I roped him into updating it for high >replication?

Yes this is true, but you have to admit this is the only useful information on designing efficient data-store. There is only one serious book explaining

how AppEngine Datastore is working and in this book you can read the same recommendation like Brett Slatkin recommends. And the book 

is a year old:

Programming Google App Engine

by Dan Sanderson

Copyright © 2010 Dan Sanderson.

Dan Sanderson is working at Google and explains everything extremely well and I believe after so much time and efforts invested to understand how it works and to successfully implement a very useful app with waiting customers - do I have to think a book and technique explained in 2010 is already out-dated?

>would you be willing to revisit it if I roped him into updating it for high replication?

I always want to know new techniques, so the quick answer is yes, definitely. 

You should also keep in mind that it is not a good business idea to constantly revisit 

and rewrite a perfectly working and tested code.

Best,

--Constantine

[Ikai Lan (Google)]

Eventually consistent datastore gets make get by key operations faster. Queries should perform the same, but better in failure scenarios in high replication.

As far as things going out of date goes: that's just the technology world for you. Every API or service provider in the world will do their best to keep things up to date, but unless you're doing legacy COBOL maintenance, there's always going to bits where people will say, "Oh, by the way, don't use that way anymore. Use this new way instead." The high replication datastore didn't exist when Dan wrote that book. Also, you should know how the book writing cycle works: a writer works with a publisher, then writes the book over the next year or two. During that time, the publisher markets the book, and the writer furiously tries to keep it up to date, knowing that chances are good there are bits that will be obsolete and his initial plan has to change. I don't know off the top of my head if Dan is planning on releasing a new version of his book or not. The first edition was printed in November 2009. In general, Dan's advice is correct: use get-by-key as much as possible, and any update to the book will likely include a section about high replication.

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com | twitter.com/ikai

Best,

--Constantine

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/BzQWzYVCk3UJ.

[Stephen]

On Thu, Aug 18, 2011 at 10:58 PM, thstart <ths...@gmail.com> wrote:

I think Master/Slave is a lot faster in my case up to 10 times faster. I tested it with only read, no write operations,

no transactions, nothing special for example I have a heavy use of this expression:

            e = MyList.get_profile(email)

                my_keys = e.my_keys  

                my_members = db.get(my_keys)

                for member in my_members:

                        member.<property>, etc. <-- do something with the property

Try this:

            e = MyList.get_profile(email)

                my_keys = e.my_keys  

                my_members = db.get(my_keys, config=db.create_config(read_policy=db.EVENTUAL_CONSISTENCY))

                for member in my_members:

                        member.<property>, etc. <-- do something with the property

It's a relatively new option:

  http://code.google.com/intl/en/appengine/docs/python/datastore/functions.html#create_config

[Nadun Kulatunge]

wow cool fun stuff << http://j.gs/I9f >> i think you will like it

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Bye TC

[Fred Sauer]

On Fri, Aug 19, 2011 at 1:38 AM, Stephen <sdeasey...@gmail.com> wrote:

On Thu, Aug 18, 2011 at 10:58 PM, thstart <ths...@gmail.com> wrote:

I think Master/Slave is a lot faster in my case up to 10 times faster. I tested it with only read, no write operations,

no transactions, nothing special for example I have a heavy use of this expression:

            e = MyList.get_profile(email)

                my_keys = e.my_keys  

                my_members = db.get(my_keys)

                for member in my_members:

                        member.<property>, etc. <-- do something with the property

Try this:

            e = MyList.get_profile(email)

                my_keys = e.my_keys  

                my_members = db.get(my_keys, config=db.create_config(read_policy=db.EVENTUAL_CONSISTENCY))

If you don't want to give you eventual consistency, try this:

my_members = db.get(my_keys, 

             config=db.create_config(max_entity_groups_per_rpc=1))

Similarly, in Java you can set max entity groups per RPC:

- Set  the appengine.datastore.defaultMaxEntityGroupsPerRpc system property to 1

- Use http://code.google.com/appengine/docs/java/javadoc/com/google/appengine/api/datastore/DatastoreServiceConfig.html#maxEntityGroupsPerRpc(int)

 

                for member in my_members:

                        member.<property>, etc. <-- do something with the property

It's a relatively new option:

  http://code.google.com/intl/en/appengine/docs/python/datastore/functions.html#create_config

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Fred Sauer]

On Fri, Aug 19, 2011 at 10:46 AM, Fred Sauer <fre...@google.com> wrote:

On Fri, Aug 19, 2011 at 1:38 AM, Stephen <sdeasey...@gmail.com> wrote:

On Thu, Aug 18, 2011 at 10:58 PM, thstart <ths...@gmail.com> wrote:

I think Master/Slave is a lot faster in my case up to 10 times faster. I tested it with only read, no write operations,

no transactions, nothing special for example I have a heavy use of this expression:

            e = MyList.get_profile(email)

                my_keys = e.my_keys  

                my_members = db.get(my_keys)

                for member in my_members:

                        member.<property>, etc. <-- do something with the property

Try this:

            e = MyList.get_profile(email)

                my_keys = e.my_keys  

                my_members = db.get(my_keys, config=db.create_config(read_policy=db.EVENTUAL_CONSISTENCY))

If you don't want to give you eventual consistency, try this:

my_members = db.get(my_keys, 

             config=db.create_config(max_entity_groups_per_rpc=1))

Also, for reference, here's what this does:

  @ConfigOption

  def max_entity_groups_per_rpc(value):

    """The maximum number of entity groups that can be represented in one rpc.

    For a non-transactional operation that involves more entity groups than the

    maximum, the operation will be performed by executing multiple, asynchronous

    rpcs to the datastore, each of which has no more entity groups represented

    than the maximum.  So, if a put() operation has 8 entity groups and the

    maximum is 3, we will send 3 rpcs, 2 with 3 entity groups and 1 with 2

    entity groups.  This is a performance optimization - in many cases

    multiple, small, concurrent rpcs will finish faster than a single large

    rpc.  The optimal value for this property will be application-specific, so

    experimentation is encouraged.

[thstart]

+Ikai Lan 

+Fred Sauer

+Stephen

Thank you, I need to rethink again with this information: it is useful. I am fine

with new technologies, in my case the product is very sophisticated and the balance

between new Google technology and what my business requires was tilted to business

side - e.g. less new technology and techniques, more pro business solution. 

Definitely an extension to the book, an update to the excellent Brett Slatkin (he explains very well) 

materials will be of much help and am awaiting them and am wondering how i will know if they are ready?

There is some practical questions: these changes obviously have to be tested, I am fine to set up a second

app just for that, but still it would be not the real thing in real situations. Right now I have a group of trusted people 

who are already using it and looking at the logs I can see real life situations which never pop up in test accounts.

So my question is - I obviously will want to implement the changes one by one and see the effect, being right now

on M/S could this be done in M/S environment and then move my users to HR?

Also I suppose when Google goes full HR and obviously wants we to move out of M/S, do Google has benchmarks 

for this kind of usage as I described in order to have a good motivation to move to HR?

Thanks to all!

[Stephen]

On Fri, Aug 19, 2011 at 6:49 PM, Fred Sauer <fre...@google.com> wrote:
>>
>> If you don't want to give you eventual consistency, try this:
>>
>> my_members = db.get(my_keys,
>>              config=db.create_config(max_entity_groups_per_rpc=1))
>
> Also, for reference, here's what this does:
>
>   @ConfigOption
>   def max_entity_groups_per_rpc(value):
>     """The maximum number of entity groups that can be represented in one
> rpc.
>     For a non-transactional operation that involves more entity groups than
> the
>     maximum, the operation will be performed by executing multiple,
> asynchronous
>     rpcs to the datastore, each of which has no more entity groups
> represented
>     than the maximum.  So, if a put() operation has 8 entity groups and the
>     maximum is 3, we will send 3 rpcs, 2 with 3 entity groups and 1 with 2
>     entity groups.  This is a performance optimization - in many cases
>     multiple, small, concurrent rpcs will finish faster than a single large
>     rpc.  The optimal value for this property will be application-specific,
> so
>     experimentation is encouraged.
>     """

It feels like you'd always want to set this to 1. What situations
might make >1 perform better?

[thstart]

A bit from business side reality - today my partners where on a meeting
with a BIG customer representatives. Functionality and speed especially
(M/S currently) impressed them so much that they wanted to add 500 users immediately

to our app and paid for it to be sure they get an access. 

Our app is with Billing On so I don't see this High Replication vs. Master/ Slave 

thing as an issue for Google -  we are paying customers. 

So now the question is open before me and my partners especially:

1) Things are moving very fast from business side and I will mark the app from Alpha

to Beta soon after looking how it works with more simultaneous users.

2) The users are already feeling the speed and any speed decrease will be mentioned

immediately. That is because the app is actually a mobile HTML5 app and used in 

all kind of devices: Android, iPhone, Blackberry. For such kind of app the speed is of

1st priority and any decision about functionality is rejected if the speed is slowed down.

3) Now the question is what benefit we will get (if any) moving from M/S to HR WITHOUT

disrupting the current functionality and speed? Does the amount of time, effort and cost 

which need to be invested in migration to HR is justified? As a start-up this is critical for us.

Every day we have a NEW list of priorities which are business related - users need more and more -

we get this from the feedback. Nobody cares about underlying technology if it works good enough.

Best,

--Constantine

[Fred Sauer]

Feel free to set the default to 1 for your apps.

Fred on my Android.

[Fred Sauer]

Fred on my Android.

On Aug 19, 2011 11:18 AM, "thstart" <ths...@gmail.com> wrote:
>
> +Ikai Lan 
> +Fred Sauer
> +Stephen
>
> Thank you, I need to rethink again with this information: it is useful. I am fine
> with new technologies, in my case the product is very sophisticated and the balance
> between new Google technology and what my business requires was tilted to business
> side - e.g. less new technology and techniques, more pro business solution. 
>
> Definitely an extension to the book, an update to the excellent Brett Slatkin (he explains very well) 
> materials will be of much help and am awaiting them and am wondering how i will know if they are ready?
>
> There is some practical questions: these changes obviously have to be tested,

You can also deploy another app version for testing.

I am fine to set up a second
> app just for that, but still it would be not the real thing in real situations. Right now I have a group of trusted people 
> who are already using it and looking at the logs I can see real life situations which never pop up in test accounts.
> So my question is - I obviously will want to implement the changes one by one and see the effect, being right now
> on M/S could this be done in M/S environment and then move my users to HR?

The impact of these changes are material on HRD, and may have limited performance impact on M/S.

Test where you like, but test the scenario which you plan to use.

Above all else, migrate to HRD as soon as practical.

>
> Also I suppose when Google goes full HR and obviously wants we to move out of M/S, do Google has benchmarks 
> for this kind of usage as I described in order to have a good motivation to move to HR?

While you should care that your app has acceptable performance, and you should use the suggested tips to get optimal performance out of HRD, your primary concern or reasons for migrating off of M/S are
1 Avoiding scheduled read-only periods
2 Avoiding unscheduled outages and maintenance periods
3 Avoiding datastore latency spikes

Every developer who I've communicated with has been happy with their app on HRD and none want to go back. In particular the 3rd point I mentioned often weighs the heaviest for developers.

To put it another way: I have yet to find a use case, when the whole app is considered, where M/S is preferable to HRD.

Hope that helps.
Fred

>
> Thanks to all!

>
>
>
> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.

> To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/cmDqAoiZDiwJ.

[Fred Sauer]

You should mirate to HRD if you care about consistent, predictable speed, performance and reliability. Your app will be happier. You will be happier. Your customers will be happier.

The advice to migrate to HRD is unqualified: you should migrate.

Fred on my Android.

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.

> To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/iDQzYaVOSioJ.

[Brandon Wirtz]

I have a set of apps that Run very well on MS and unless MS is all the way down (which has happened) they ran just fine during maintenance, and No-Write periods.  They are cheaper and faster on MS. 

 

HR is definitely the best for many situation, but there may be times when MS is better especially if you are fighting costs.

[thstart]

+Fred Sauer

>Test where you like, but test the scenario which you plan to use.

the question is if these changes suggested will work on M/S because I will not change everything

in one time, but gradually on M/S and if it works to move it to HR.

[thstart]

+Fred Sauer

>To put it another way: I have yet to find a use case, when the whole app is considered, where M/S is preferable to HRD.

My only concern if I will get the same speed. I don't want to spend all this effort

costing me money and time and at the end to find it will be slower than M/S.

By the way while I was on HR (about 2 months) I mentioned there was a time I cannot upload

my app, had to try 10 times, sometimes to wait 1 hour. This happened very often.

[Jeff Schnitzer]

On Fri, Aug 19, 2011 at 8:16 PM, thstart <ths...@gmail.com> wrote:
> My only concern if I will get the same speed. I don't want to spend all this
> effort
> costing me money and time and at the end to find it will be slower than M/S.

Did you try the solution Stephen mentioned?

my_members = db.get(my_keys,
config=db.create_config(read_policy=db.EVENTUAL_CONSISTENCY))

I would be curious to know if this eliminated the speed issue. From
all reports (and my own highly unscientific observation), HRD seems to
perform consistently better than M/S. But I use EVENTUAL_CONSISTENCY
as a default - probably others do too. I haven't even thought to try
the alternative.

Jeff

[thstart]

+Jeff Schnitzer

>EVENTUAL_CONSISTENCY

so with this we are 1 step back to M/S using HR? What is the point?

P.S. it is 3 hours and I still can not upload my app on the cloud (M/S). Weekends is the time

for most productive work, and I consistently cannot do that. 

For the record now is 11:30 PM PST, 8/19/2011. Checked system status - it says it is OK.

It is not OK if I cannot upload for 3 hours.

[Sekhar]

We're about to launch a healthcare app, and we recently switched from M/S to HRD (both with reserved instances). I'm really, really happy with HRD so far, most importantly we don't see random hiccups in data access any more. Performance has been really consistent.

We are occasionally getting a new error "too much contention on these datastore entities. please try again." however. Clearly it's not due to traffic because it's like 5 of us testing the site at this point. I'm a bit worried our patients/doctors will see this after launch next month. What's this and is there a way to avoid it?

[Jeff Schnitzer]

On Fri, Aug 19, 2011 at 11:30 PM, thstart <ths...@gmail.com> wrote:
> +Jeff Schnitzer
>>EVENTUAL_CONSISTENCY
> so with this we are 1 step back to M/S using HR? What is the point?

Huh? This seems a complete non-sequitur.

> P.S. it is 3 hours and I still can not upload my app on the cloud (M/S).

I get that you're upset about something being broken. As a
non-googler, let me offer you some objective advice: You sound
awfully whiny. You'd probably get better reception by taking a
different tone.

For example, you might post: "My application, which is fast on M/S,
is slow on HRD. Why is that?" This is something we can all explore
and maybe learn something new - it runs contrary to everyone else's
experience. Instead you post "HRD is slow!" and expect us to pry the
details out of you. It doesn't sound like you're actually interested
in improving your product, you just want to rant. Seriously, get a
therapist.

Jeff

[thstart]

+Jeff Schnitzer

> pry the details out of you.

I posted a code if you care to read, the advice is EVENTUAL_CONSISTENCY which is a compromise and I don't need to do it in M/S.

>get better reception ...

Yes I got a reception, Brett will update his presentations, Dan will update his book - I just purchased an access to his 2nd edition - a work in progress, and yes - I make every effort to do what is needed. 

The point is if you have a project which you really care you will do the same to make sure it works. If you have the time to thinker 24/7 with database experimenting, then good luck.

Just checking - have you looked at the code I posted, made tests both in HR and M/S, performed benchmarks or you just want to talk not having  what else to do?

Do this and then come back ton talk about real world situations.

>get a terapist

If you are at the point to have a waiting list of paying customers, and you invested a year in developing, testing and almost to the final, and at this point you have to completely redesign you code without knowing in advance, if you an in areal world situation probably you will get a terapist, I don't need one.

[thstart]

+Jeff Schnitzer

Just got an answer from Dan on G+: "Compared to M/S, the HRD does indeed trade some speed for reliability." 

That was my point. 

[thstart]

> P.S. it is 3 hours and I still can not upload my app on the cloud (M/S).

>I get that you're upset about something being broken.

Yes there are many times when try to upload my code on the cloud but the uploads are rolled back. This lasted yesterday for 3 hours. 

Is it not clear what I want to say? What is your recommendation?

[Robert Kluin]

This could be for a wide range of reasons. You should start a new thread and give us some more details. For example, are the entities with contention warnings global to all users or user specific?  Is the contention caused by user requests or tasks?  What are you doing, updating a counter?  Be sure to give us enough information to help you.

Robert

On Aug 20, 2011 10:52 AM, "Sekhar" <allu...@gmail.com> wrote:
>
> We're about to launch a healthcare app, and we recently switched from M/S to HRD (both with reserved instances). I'm really, really happy with HRD so far, most importantly we don't see random hiccups in data access any more. Performance has been really consistent.
>
> We are occasionally getting a new error "too much contention on these datastore entities. please try again." however. Clearly it's not due to traffic because it's like 5 of us testing the site at this point. I'm a bit worried our patients/doctors will see this after launch next month. What's this and is there a way to avoid it?
>

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.

> To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/QygaPE8F_dwJ.

[Mike Lawrence]

For the one week off of Master/Slave and on HR:

• up-time has been 100%
  
• max response time has improved... down from 22+ seconds to about 2-6 seconds.
  
• Average response time is still sub-second for the week.

[thstart]

+Mike Lawrence

As you post I have couple of questions to you.

>Average response time is still sub-second for the week.

Do you use queries to fetch data or you use a lists?

Do you modified your app when moving to HR, or just transferred it?

>1000 simultaneous users.

Are you users logged in or just retrieving data anonymously?

[Mike Lawrence]

In general, my app does 3 db gets for every db query.

No modifications were necessary with my app. Using slim3 db for transactions.

1000 simutaneous users were logged in using my own authentication (not federated) with sessions enabled.
I only use session to record the fact that a user has authenticated
I save app state in hidden form html.

hope that helps

Sincerely,  Mike Lawrence

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/rpGj8t4Gy-0J.

[thstart]

Hi Mike,

Thank you, I am collecting the bits needed for eventual switch.

In fact I already switched my app to HR, it worked for 1 month but 

I never got the speed of M/S. So now I have to figure out why.

Best,

--Constantine

[thstart]

+Ikai Lan

>Dan's advice is correct: use get-by-key as much as possible, and any update to the book will likely include a section about high replication.

I just bought an access to his book in progress. Asked him a question on G+ about speed in general, get the following answer:
"Compared to M/S, the HRD does indeed trade some speed for reliability. But that reliability is crucial.". Now if that is true what do you recommend to get same speed? Probably there is a way for better parallelization?

What is most important is how to achieve the maximum read speed, the writes I do in background in deferred tasks and the user can wait till the process finishes - this is not an issue.

>The practice you're citing from Brett is a few years old and predates high replication; would you be willing to revisit it if I roped him into updating it >for high replication?

How quick can we get that?

Because the information about migrating M/S to HR is dispersed in different places, 

where is the best place to start? 

Best,

--Constantine

[thstart]

+Ikai Lan

In my example:

root = MyRoot.get_profile()

skey = email

key = db.Key.from_path('MyList',

                                   skey,

                                   parent=root.key())

profile = db.get(key)

For HR are there a need to change the way these keys are formed? 

Does your data migration tool can transfer my database from M/S to HR automatically?

Do my users need to re-enter their data again?

[Ikai Lan (Google)]

Hi,

You're building the keys correctly for strong consistency.

As far as speed goes, let me give you an example of speed in action.

It takes me ~30 minutes to bike somewhere. I can also take the bus. The bus takes me 20 minutes when there are no disruptions. Unfortunately, there are disruptions several times a month. When there is a disruption, it takes me an hour to get somewhere. Is the bus faster, or is biking faster? Would you rather have a consistent 30 minute commute, or a 20 minute commute, that, several times a month, fluctuates to a one hour commute? Now pretend it randomly fluctuates to a 4 hour commute. You'd pick the bike commute every single time. That's what it's like to be on master/slave. I'd love to say it's not like that, but history will prove me wrong. While the work we've done has gone a long way in improving reliability overall, it doesn't change the fact that a single point of failure is subject to massive swings in performance.

I remember seeing a presentation by Facebook engineering about how they engineer for predictability instead of pure speed. It's the same logic. Slow and steady is better than fast and unpredictable, especially in user facing applications.

Another example is Azul System's design philosophy in pauseless GC. It's actually quite interesting. Their view is that the way GC works in Hotspot JVM (generational) optimizes the 99% use case, but in the 1% use case, GC pauses which pause the entire JVM (sometimes for a second or two, which is a deadly stall in real time applications), will exist, so their philosophy is to build a JVM that focuses on more predictability instead of prioritizing speed in the 99% use case. Disclosure: I still use Hotspot since JVM GC pauses don't affect the applications I write, but if they did, I would seriously look at their solution (http://www.artima.com/lejava/articles/azul_pauseless_gc.html).

A single unpredictable component is easy to use. When you have multiple unpredictable components, you see an exponential increase in complexity in handling wild performance swings, and you want to minimize the number of components whose reliability you cannot predict (URLFetch - you can't predict how someone else's system will perform). The datastore is, very often, at the absolute core of applications you will be building. It's very important for the performance of the datastore to be as predictable as possible.

In addition to predictability, you also get much stronger protection against data loss to the point that it is a near statistical impossibility. There's no reason why you wouldn't want to use high replication. The next time someone says something is faster, ask why, because that's where you'll get the real answers.

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com | twitter.com/ikai

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/CBRAyB0paHwJ.

[thstart]

+Ikai Lan

>As far as speed goes, let me give you an example of speed in action.

I very much appreciate your explanation. I understand we cannot know much of the 

internal working and don't need to. That is good enough for me to decide now to move back

to HRD. So did I - I just reactivated the HRD version of same app with same data and mentioned 

it is much faster now. I did not change anything, probably Google made some improvements.

So let we do some internal testing again to see how it will behave now.

Still I will go deeper and optimize from HRD in mind. Any written tutorials, guides, etc. will be 

of much help.

One more recent development makes the debate about M/S vs. HRD obsolete for Python developers

especially - the today's announcement for trusted beta for Python 2.7 under AppEngine which will work

only on HRD. It is clear any speed disadvantage HRD may have (small or not), we can overcome with

2.7 capabilities - concurrency and threading so I am fully in - just send an application for trusted testers.

Thank you!

[Fred Sauer]

On Sat, Aug 20, 2011 at 7:52 AM, Sekhar <allu...@gmail.com> wrote:

We're about to launch a healthcare app, and we recently switched from M/S to HRD (both with reserved instances). I'm really, really happy with HRD so far, most importantly we don't see random hiccups in data access any more. Performance has been really consistent.

Great to hear. It's also exactly what I'm hearing from others I've talked to who have switched from M/S to HRD.

We are occasionally getting a new error "too much contention on these datastore entities. please try again." however. Clearly it's not due to traffic because it's like 5 of us testing the site at this point. I'm a bit worried our patients/doctors will see this after launch next month. What's this and is there a way to avoid it?

In both cases, you should design your application for at most 1 datastore write/second (=1qps) per entity group. You can write simultaneously to as many entity groups as you like.

The problem you mention usually comes up in one of two scenarios:

1. You have a some shared / global data structure which is updated by multiple users at the same time, ultimately at a rate > 1qps. See http://code.google.com/appengine/articles/sharding_counters.html for a common technique for dealing with this.
2. You are testing with multiple users, but they are all logged in with the same account, or are otherwise artificially updating the same data. In this case you have a usage pattern which doesn't match what production traffic would look like.

If you're wanting to build high throughput pipelines, check out http://www.google.com/events/io/2010/sessions/high-throughput-data-pipelines-appengine.html

 

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/QygaPE8F_dwJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--

Fred Sauer

Developer Advocate

Google Inc.

1600 Amphitheatre Parkway

Mountain View, CA 94043
fre...@google.com

[Fred Sauer]

On Sat, Aug 20, 2011 at 1:38 PM, thstart <ths...@gmail.com> wrote:

+Jeff Schnitzer

> pry the details out of you.

I posted a code if you care to read, the advice is EVENTUAL_CONSISTENCY which is a compromise and I don't need to do it in M/S.

When fetching a list of keys use max_entity_groups_per_rpc=1 for speed.

Use EVENTUAL_CONSISTENCY whenever you can as it has to the potential to speed up your app. I say "whenever you can" because sometimes you won't be able to.

 

>get better reception ...

Yes I got a reception, Brett will update his presentations, Dan will update his book - I just purchased an access to his 2nd edition - a work in progress, and yes - I make every effort to do what is needed. 

The point is if you have a project which you really care you will do the same to make sure it works. If you have the time to thinker 24/7 with database experimenting, then good luck.

Just checking - have you looked at the code I posted, made tests both in HR and M/S, performed benchmarks or you just want to talk not having  what else to do?

Do this and then come back ton talk about real world situations.

>get a terapist

If you are at the point to have a waiting list of paying customers, and you invested a year in developing, testing and almost to the final, and at this point you have to completely redesign you code without knowing in advance, if you an in areal world situation probably you will get a terapist, I don't need one.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Q3CS2U8McpAJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[thstart]

+Fred Sauer 

>EVENTUAL_CONSISTENCY 

Thank you I will explore this possibility. Yesterday I moved back the app to HRD and my tests

show significant speed improvement. No change in my code. I suppose Google improved the 

speed somehow. If this is consistent then my users will be happy.

[loveneet s]

Every API or service provider in the world will do their best to keep things up to date, but unless you're doing legacy COBOL maintenance, there's always going to bits where people will say, "Oh, by the way, don't use that way anymore.

--

Ikai Lan 
Developer Programs Engineer,

G | Small Business - All Things B2B

[thstart]

haha COBOL ;) my COBOL is 1 year old. 

The code was fine. Case closed, moved to HRD, the app on HRD is much faster that 

what used to be 1 month ago. No change in code, just moved to HRD. I suppose Google worked 

very hard to improve the speed. 

Bottom line:

My users are reporting to me it works faster than M/S and faster than HRD 1 month ago. 

And yes my code is up to date and following best practices. It turned out this was a temporary 

speed issue with Google servers after they announced HRD. I published some important code I use

in my app and Google confirmed it is OK, I use keys, lists the right way. So it was not the code.

Let's move on.

[Ikai Lan (Google)]

Hooray! Good to hear =).

High replication makes me happy.

--

Ikai Lan 

Developer Programs Engineer, Google App Engine

plus.ikailan.com | twitter.com/ikai

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/WVosQnUL4VYJ.

[Nirmallya Mukherjee]

Here is a usecase that may support M/S over HRD.

 

I have a form where we capture appointment information such as date, time and some other details. Upon a successful save the resulting page is a calendar view that needs to read all appointments including this newly created appointment.

 

Given this and eventual consistency mode of HRD the calendar at times does not show the newly saved appointment. When I refresh the screen after a few seconds it does; obviously by then the data would have replicated. This is obviously not an issue with M/S.

 

Thoughts?

[Fred Sauer]

The calendar times you are querying are all for one user, you could put the calendar entities in a single entity group and then use an ancestor query, which will return consistent results.

If you need to query across entity groups, but still want the calendar entry you just saved to be included in the results, then the recommended approach is:

- Perform your cross entity group query in the normal way

- In your application code, insert the just inserted entity into the result set if it's not there

- Show the user the merged result set

You can store away or track the "just inserted calender entity" in a cookie, in a url / form parameter, in memcache, or in an HTTP session object (Java).

Fred

 

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/lC6R_999_vQJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Nirmallya Mukherjee]

In case of a wizard based interface with multiple steps where users
provide data at every step we have to ensure that we work off session
data. By and large we can conclude that the following design in the
application will help alleviate the eventual read consistency scenario
in HR which is not a problem in M/S.
1. If the appointment (or any object) is created/updated in step 1 it
also needs to go in the user session (I am a Java developer)
2. Step 2 gets the appointments (or any object collection) from the
data store and checks if the object created in step 1 is present. If
not add it in the collection manually
3. Pass it on to the JSP for display

We have to live with the fact that all appointments will not show if
the calendar is global where appointments created by multiple users
need to be displayed.

I was also wondering if Bigtable/Datanucleus can let the program know
the newly generated PK like a traditional relational database. If we
can get the PK then the process of checking the collection becomes
simple in addition to other benefits of knowing the PK after insert.

Finally how much latency are we really talking about between the last
write and the consistent read?

On Sep 2, 10:11 pm, Fred Sauer <fre...@google.com> wrote:

[Fred Sauer]

For the wizard, using an entity group will give you strong consistency. Use ancestor query or plain old fetch (get) by key to get strongly consistent results.

In general, query public data using regular (eventually consistent) queries. Merge in the user's own, just committed, data.

Most of the time you're probably only dealing with 100s of milliseconds or a few seconds.

Fred on my Android.

[Fred Sauer]

The entity id (or name) is known / available after a put().

Fred on my Android.

On Sep 3, 2011 2:47 AM, "NM" <nirmallya...@gmail.com> wrote: