GCP Cluster Creation Fails

[Brian Moroz]

Hi,

I followed a tutorial to test out GCP for some high-throughput computing: https://cloud.google.com/solutions/running-r-at-scale

The command to create the cluster "elasticluster start myslurmcluster" produces a warning "WARNING UserWarning: Cannot access <$HOME>.elasticluster/storage/...apps.googleusercontent.com.oauth.dat: No such file or directory" which then leads to an error "ERROR Could not start node ... <class httplib.ResponseNotReady>"

Everything works as stated up to attempting to start the cluster. Looks like a credential authorization problem. Is this a known issue? I'm trying to determine if I missed something or did something wrong before I contact Google (assuming this is their tutorial). I haven't seen much on this just browsing around.

Thanks.

Google Cloud SDK on Ubuntu 18.04 VM

[Brian Moroz]

There may be a bug in elasticluster credential verification, however, you can bypass this process by setting application default credentials (https://cloud.google.com/docs/authentication/production). From what I have read, some folks don't need to do this while others do.

[Riccardo Murri]

Hello Brian,

many thanks for taking the time to work around this issue and share
your findings here on the list.

I would be interested in fixing this once and for all, so that other
people don't have to go through hoops to get ElastiCluster working.
If I read the code correctly, the ElastiCluster's Google Cloud code
should do the following:

1. If `gce_client_id` and `gce_client_secret` are given in the
configuration, try to use the credentials cached in the `oauth.dat`
file;
2. If that fails, try to use "Application Default Credentials";
3. If that fails too, try to go through the OAuth auth process again.

I understand this didn't work as expected in your case?

Kind regards,
Riccardo

[Brian Moroz]

Hi Riccardo,

Sorry for the late reply.

In my case I had the proper credentials in the config file but I did not have ADC set up yet (so no oauth.dat file). The comments in the code imply the if the oauth file does not exist then one is created. However, I don't see where the write occurs to create the file if the credentials return None. I see the storage.get() (gce.py line ~152), however if the credentials return None it just writes to the log. I don't see a storage.put() or something to create the oauth.dat file. In this case, the code then moves on to ADC. It would seem then that basically you need to have the oauth.dat file in place before starting elasticluster or you must have your environment set up correctly for ADC.

~Brian

[sgor...@woodwellclimate.org]

Hi Brian (or Riccardo),

It looks like I'm running into the same credential authorizing issue Brian was when I try to start my cluster using:
elasticluster start agbclust

I get the message:
WARNING UserWarning: Cannot access <$HOME>/.elasticluster/storage/...apps.googleusercontent.com.oauth.dat: No such file or directory

Followed by the error:
ERROR Error creating instance `<HttpError 403 when requesting https://compute.googleapis.com/compute/v1/projects/.../zones/us-east1-c/instances?alt=json returned "Request had insufficient authentication scopes.". Details: "Request had insufficient authentication scopes.">`

I've attached my config file (with sensitive info redacted). For the "gce_client_id" and "gce_client_secret" config keys, I've generated them following the instructions provided here...

https://googlegenomics.readthedocs.io/en/latest/use_cases/setup_gridengine_cluster_on_compute_engine/index.html#index-obtaining-client-id-and-client-secrets

...with the exception of selecting "Desktop" for OAuth client ID type as opposed to "Other" as suggested because that option doesn't exist.

Could you please expand on the solution Brian provided? I'm not seeing how to setup "Application Default Credentials" or create the oauth.dat file. Any help you could provide would be greatly appreciated.

Thank you,

Seth

[sgor...@woodwellclimate.org]

Never mind. This did the trick:
https://github.com/elasticluster/elasticluster/issues/626#issuecomment-479657010

Best,

Seth