dataproc cluster time out NOT responding to cahnge in --max-idle VALUE

[Basanta Dhungana]

Once the cluster created, the cluster deletes itself after 10 minutes even after changing parameter to --max-idle 60m from --max-idle 10m. Unless you changed the name of cluster with new --max-idle VALUE, parameter takes the value that you used for the first time and NOT responding to new VALUE when you recreate same cluster NEXT time. I am NOT sure this is an issue or designed that way!

Here is the code for reference:

gcloud beta dataproc clusters create mycluster --enable-component-gateway --bucket mybucket --region us-west1 --subnet default --zone us-west1-a --master-machine-type n1-standard-1 --master-boot-disk-size 500 --num-workers 10 --max-idle 60m --max-age 1000 --worker-machine-type n1-standard-1 --worker-boot-disk-size 500 --image-version 1.4-debian9 --optional-components ANACONDA,JUPYTER,ZEPPELIN --scopes 'https://www.googleapis.com/auth/cloud-platform' --project myproject.

Your response would be appreciated. Thank you.

Regards.

[Jerry Ding]

Hi Basanta,

To clarify, is the problem that (a) when you delete and recreate a cluster with the same name, it fails to respect a different max-idle flag, or (b) that when you try to update the max-idle time of an existing cluster it fails to respect the change?

Best,

Jerry

[Basanta Dhungana]

Hi Jerry,  When I recreate a cluster with the same name, it fails to respect a different max-idle flag. To correct my earlier statement, it turns out when I created a new cluster name with different max-idle flag, that is still deleting cluster after 10 minutes! I hope flag is NOT sticking with account/project. It looks strange. Thanks.

[Dennis Huo]

Do you have a cluster_uuid you can share here, and/or a projectid you can send to dataproc...@google.com ? Do you have an official support case opened regarding this?

[Basanta Dhungana]

It turns out that I missed to spell out correct time unit for another parameter --max-age 1000. After replacing this  to  --max-age 1000m I repeated the test and it is working as expected. My apology for any inconvenience! 

[Dennis Huo]

Great to hear it's working, thanks for the update!