Datastore replication to Cloud SQL and BigQuery with eventual consistency

[bobby via StackOverflow]

Our company writing an app which should support many users and simply using Cloud SQL is not an option. Because of that we are using Datastore. But we need to be able to perform complex queries on our database for analytical purposes.

We have two kinds of entities:

1. non-changing-only-appending, which we need to insert into BigQuery somewhat asynchronously (e.g. posts in forum engines)

2. changing entities (e.g. users in forum engines, contains data like posts count, etc), which we need to insert/update into Cloud SQL

For BigQuery I think we could simply insert job as soon as there is new data and that'll be fine (correct me if I'm wrong).

But for Cloud SQL I couldn't come up with anything. I thought about using Task Queue API, but I need to somehow enforce order of updates (because it matters) and at the same time to be able to process more than one task concurrently, otherwise unprocessed task count will keep increasing.

Please DO NOT REPLY directly to this email but go to StackOverflow:
http://stackoverflow.com/questions/21774992/datastore-replication-to-cloud-sql-and-bigquery-with-eventual-consistency

[Andromeda via StackOverflow]

Could you elaborate on what you meant by "simply using Cloud SQL is not an option." ? Also could you tell more about your application setting (environment) ?

Please DO NOT REPLY directly to this email but go to StackOverflow:

http://stackoverflow.com/questions/21774992/datastore-replication-to-cloud-sql-and-bigquery-with-eventual-consistency/21790989#21790989