How export BiqQuery data to google Storage bucket either using with API or Java SDK automatically ?

[vitaly....@reps.ai]

Hi all,

Working with app engine 1.9.37 with java8 , we are using BiqQuery , is it possibly to export to Storage bucket cloud name "xxxx"  the data/tables either with API CRON.xml or with java SDK ?

Our mission is to do it automatically.

Is it possible ? 

[Ali T (Cloud Platform Support)]

It’s possible to do so. This would be like any other application that does some manipulation on data from BigQuery . Your application can be done using the client libraries[1] or through the rest API using extract property[2]. Once the application setup, you can use Cron jobs[3] to schedule your app to run every “n” time interval.

Moreover, as outlined on the official exporting data from BigQuery documentation[4], this can also be done through Dataflow jobs which can also be scheduled to run automatically at a certain time interval.

[1] https://cloud.google.com/bigquery/docs/exporting-data#exporting_table_data

[2] https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/insert

[3] https://cloud.google.com/appengine/docs/standard/python/config/cron

[4] https://cloud.google.com/bigquery/docs/exporting-data

[Vitaly Tarasiuk]

Hi,

Thanks for the answer , I'm trying to do it using Cloud Data Flow , 

Inside the page/templates  I don't see any mention about BiqQuery export...

What are my next steps (which option to choose according from the templates , or  are there any other options ) ? 

Attached screen below

Thanks.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/fd888e79-8687-4a82-9757-604d535a6919%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ali T (Cloud Platform Support)]

There are currently no predefined templates from BigQuery to Google Cloud Storage. However, Dataflow does allow you to create your own templates to fit your use case. You can find information pertaining to creating your own template at the following documentation[1]. Information regarding the required classes and functions to read from BigQuery and write to Google Cloud Storage for Java are available on the Apache Beam API reference page[2]. Additionally, as your use case is to automate the export, you can do so using App Engine Cron services[3].

[1] https://cloud.google.com/dataflow/docs/templates/creating-templates

[2] https://beam.apache.org/documentation/sdks/javadoc/2.6.0/ 

[3] https://cloud.google.com/blog/products/gcp/scheduling-dataflow-pipelines-using-app-engine-cron-service-or-cloud-functions