Autoscaling Dataproc Jobs

[Vadim Solovey]

Cloud Dataproc is an amazingly fast to provision, easy-to-use, fully-managed cloud service for running Apache Spark and Apache Hadoop clusters in a simple and very cost-efficient way. While you can resize Google Cloud Dataproc clusters at any time but it doesn't have built-in autoscaling for clusters. We have multiple teams running the jobs on the same cluster and we had to have some autoscaling. Recently, we open sourced Shamash - autoscaling Dataproc clusters - https://github.com/doitintl/shamash. Enjoy!

[Pranjul Ahuja]

Hi Vadim,

Can this be used to autoscale  presto which is deployed on data-proc cluster ?