Reg: Tools on Google cloud platform that helps us to analytics

[Manikanta Kondeti]

Hello,

We are just one of the groups that uses app engine for hosting. I have a few questions related to storing logs (logging of various actions that happen in the app) . Currently we are storing all the information in form of entities in datastore. If I want to do a behavioral analytics on a particular group of users I have to fetch all of them from app engine and run queries. It takes lots of time to fetch those data and do analytics. Are there any good tools on google cloud platform that help us to make this entire process better?

Regards,

Manikanta

[troberti]

BigQuery seems to be what you want. It will also be much cheaper (and more powerful) than using the Datastore. You could implement it by queueing a task in the taskqueue each time you want to log an action, and that task then inserts the data into BigQuery using the streaming ingestion api. This is the approach we use to log events in our app and we are very happy with it. 

[Nick (Cloud Platform Support)]

Seconding @troberti here, BigQuery is definitely a better match for logs storage and analysis. Datastore would have an advantage if you wanted to update records, but logs are immutable and write-once, so BigQuery, which doesn't support UPDATE queries, is the best match, also because it can query large amounts of data very quickly (the central design premise of BigQuery).

[Nick (Cloud Platform Support)]

You can also set up streaming of your logs directly into BigQuery. Just wanted to add that link in case you hadn't seen it, Manikanta.

Cheers,

Nick
Cloud Platform Community Support