Transaction across Java appengine datastore and cloud sql

[Rajesh Gupta]

Hi,

We are running out of indexes on appengine datastore.  We have 10-20 remaining.

No time for doing refactoring and to regain the indexes. 

One thought is to use Cloud sql or Cloud spanner for further development.  There are other reasons also to look for alternatives to datastore. (finding talent, restrictions of datastore on aggregates queries, no easy way to do migrations or schema changes)

Is there a way to do a transaction across datastore and cloud sql or cloud spanner

What are the recommended methods to further expand dev.

Rajesh

www.GainERP.com

ERP & Field Service on Google Cloud

[Ani Hatzis]

It seems your code-base has grown into a quite huge monolithic app. If you haven't done yet, you might look into this: Microservices Architecture on Google App Engine. You could expand your app by putting new models / handlers into new apps using this pattern, and eventually also use a different storage product for them. It also depends on your existing app and the roadmap, how well you can separate the new parts without refactoring the old app.

You cannot do a transaction across Cloud Datastore and Cloud SQL or Cloud Spanner.

You also should keep in mind that every product comes with different advantages and disadvantages and you need to figure out which trade-offs make most sense for your technical and business requirements. Considering a mature app that has to cover many different use-cases, maybe you will pick more than one storage product to meet all of them.

And although I don't know if that's the case here, worth to mention: I often have seen a (composite) index used in Datastore only for user-facing searches where Search API would have made much more sense. Refactoring such cases can be quite easy if your app is the only client writing into its GCD.

Good luck!

[Nick]

You can’t span different storage services with transaction at this time. Right now, only the task queue and datastore can participate in a transaction.

This will allow you to achieve most of the outcome you’re looking for (ie guaranteed writes), but not advanced transactional controls.

When writing to the data store, you can enqueue a task in a datastore transaction. When the task runs you can write to a persistence service in a transaction, for example cloudsql. Your task will give you retry if the attempt fails.