Cloud ML to Big Query

[Justin Or]

Hey guys, I want to populate Data Studio with predictions from cloud ML for a demo on GCP. Right now, we have a Dataflow pipeline passing all training data to a BigQuery table, which gets exported to a csv file on a GCS bucket for training. 

Since Data Studio has a built-in Big Query connector, I wanted to host all the data on that. However, I realised there seems to be no easy way to get the prediction results from cloud ML to pop up in Big Query. 

At the moment, I am thinking of making a script on python to:

1. Generate a batch request of predictions

2. Run the job on cloud ML engine

3. Push the data to big Query.

I think this script would need to be run in AppEngine, or at least on a compute instance, and have no idea how one would run this script from inside Data Studio. 

Is there a better way to solve this problem? Any help would be greatly appreciated.

[Ali T (Cloud Platform Support)]

Hi,

Assuming the prediction results are getting written to GCS, you can use the GCS to BigQuery transfer service to push the data to BigQuery. This would avoid the need to write a script that does so. 

Additionally, if you constantly expect new data to be written to the GCS bucket at a steady time interval, you can setup a schedule for when the transfer automatically runs to ensure the BigQuery data is constantly up to date. That being said, it doesn’t appear to be possible to trigger the transfer from Data Studio using this method.