Datastore contention errors after switching from DB to NDB Cloud

[Remko Tronçon]

Hi,

We recently switched from the old App Engine Datastore DB API to the NDB Cloud API. Since we did this, we are getting much more datastore contention errors (which used to be a rarity). Is this expected? I was not expecting a change of library to have any influence on behavior.

thanks,

Remko

[Pierre-Yves (Google Cloud Support)]

Hello Remko,

Contention errors are usually observed when updating entities too rapidly, or when high read or write rates to lexicographically close documents. Since you recently changed the library to Cloud NDB, you may review this migration guide [1] which outlines key differences. One of these differences is that data caching is now done in a Redis in-memory datastore managed by Memorystore instead of the legacy App Engine Memcache service. You may therefore make sure appropriate changes have been made accordingly.

You may also look at the Datastore Best Practices [2] which discusses contention directly. To make sure that your application follows the limit of the new Firestore in Datastore mode limits, you may review those here [3] to make sure that other potential services are interacting with Datastore according to those limits. Firestore in Datastore mode is a different architecture than the legacy Datastore so understanding those are crucial for a smooth transition.

Finally, you may review the official documentation for the Cloud NDB library [4] to make sure that entities, keys, and properties are properly defined.

[1] https://cloud.google.com/appengine/docs/standard/python3/migrating-to-cloud-ndb 

[2] https://cloud.google.com/datastore/docs/best-practices

[3] https://cloud.google.com/datastore/docs/concepts/limits

[4] https://googleapis.dev/python/python-ndb/latest/index.html 

[Remko Tronçon]

Hi Pierre-Yves,

Thanks for your comments. I'm aware of what causes contention errors, and how to resolve them. However, I'm just surprised that moving from the DB (*not* NDB) library to the Cloud NDB library suddenly causes contention errors. (and it's significant; e.g. a model that never used to have problems with 20 shards now still has problems with 60 shards).

I am also aware about the caching behavior. I have not configured any cache. However, since I'm coming from DB (which isn't cached at all either), and since this is about problems with writing (which would write through the cache anyway AFAIK), I don't see how cache changes can cause this.

Unfortunately, since I'm not coming from NDB, I can't say whether it's the move from DB->NDB or NDB->Cloud NDB that causes this. I did the entire jump in one time.

Could you explain why you bring up 'Firebase in Datastore mode'? Am I suddenly using a different architecture by switching to a different library to access the same data?

thanks,

Remko

[Pierre-Yves (Google Cloud Support)]

Hello Remko,

Just to clear any confusion, the product I was referring to was “Firestore in Datastore mode”. Although the names are similar, “Firestore” and “Firebase” are different in that “Firebase” is a development platform and “Firestore” is the new-generation of NoSQL Cloud Database.

In the past, you were using the “Datastore” product and the DB API library to access the database in that product. On January 31st 2019, “Cloud Firestore” became Generally Available [1]. Afterward, an automatic upgrade to Firestore was executed (see [2] for more details), and we made available the new product available in Datastore mode. Although this allows users of the legacy Datastore still feel at home, the architecture of Firestore is different and this is why we recommend migrating from DB to the NDB client library. Here [3] is the correct documentation I meant to send you in my previous message about this migration. You may use it as reference to compare with your own implementation to target potential issues leading to the contention errors.

So to address your inquiry, switching to a different library is not directly what causes the data to be held in a different architectural product, but the fact that this new library is optimized to work with the new product being Firestore. Therefore, I recommend looking at the limits specific to Firestore as outlined here [4], which may be different than the ones for Datastore, and test how your application queries it resulting in more than 1/sec mutations on a single entity group. Please note that cross entity group contention can cause unavailability, so reading an entity group in a transaction counts as a “write” and thus is limited to 1 per second. For a transaction across entities, all participant entities must be uncontended in order for the transaction to succeed.

Let me know if you need further help.

[1]  https://cloud.google.com/datastore/release-notes#January_31_2019 

[2] https://cloud.google.com/datastore/docs/upgrade-to-firestore#automatic_upgrade_to_cloud_firestore_in_datastore_mode 

[3] https://cloud.google.com/appengine/docs/standard/python/ndb/db_to_ndb

[4] https://cloud.google.com/datastore/docs/concepts/limits