How to schedule dataproc jobs

[amerikiwi]

What is the best way to schedule dataproc jobs? I have jobs using GCS, BQ, Pyspark in Dataproc. If anyone has some sample code that would be great.

[Dan Sedov]

At the moment cloud scheduler [1] appears to be the best option available. Here's a blog post talking about using it for Dataproc jobs [2].

[1] https://cloud.google.com/appengine/docs/standard/python/config/cron

[2] https://cloud.google.com/blog/big-data/2016/04/scheduling-dataflow-pipelines-using-app-engine-cron-service-or-cloud-functions

[James Malone]

You can also use a self-installed Apache Airflow instance. Airflow has support for Cloud Dataproc along with other GCP products:

https://github.com/apache/incubator-airflow/tree/master/airflow/contrib/operators