Questions about gce error

29 views

Skip to first unread message

odyhpc Fdez

unread,

Aug 10, 2020, 4:07:21 PM8/10/20

to elasticluster

Hello,

I'm trying elasticluster and, after figuring out a few things, thought to be ready to deploy a tiny gce cluster. However, the deployment command (elasticluster -c /home/odyhpc/elasticluster/ody.d/config start odycluster) is returning the following chain of errors:

(elasticluster) odyhpc@deployator0002:~/elasticluster$ elasticluster -c /home/odyhpc/elasticluster/ody.d/config start odycluster

Starting cluster `odycluster` with:

* 1 frontend nodes.

* 2 compute nodes.

(This may take a while...)

2020-08-10 18:08:10 deployator0002 elasticluster[3768] WARNING DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working

2020-08-10 18:08:10 deployator0002 elasticluster[3768] WARNING UserWarning: Cannot access /home/odyhpc/.elasticluster/storage/423015455368-58j4u4st0f38a5ambe6q5bamq6k8d7ha.apps.googleusercontent.com.oauth.dat: No such file or directory

2020-08-10 18:08:10 deployator0002 elasticluster[3768] WARNING ResourceWarning: unclosed <socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.128.0.36', 52210), raddr=('169.254.169.254', 80)>

2020-08-10 18:08:11 deployator0002 elasticluster[3768] ERROR Error creating instance `<HttpError 403 when requesting https://compute.googleapis.com/compute/v1/projects/watchful-origin-244417/zones/us-central1-a/instances?alt=json returned "Request had insufficient authentication scopes.">`

2020-08-10 18:08:11 deployator0002 elasticluster[3768] ERROR Could not start node `compute002`: Error creating instance `<HttpError 403 when requesting https://compute.googleapis.com/compute/v1/projects/watchful-origin-244417/zones/us-central1-a/instances?alt=json returned "Request had insufficient authentication scopes.">` -- <class 'elasticluster.exceptions.InstanceError'>

Traceback (most recent call last):

File "/home/odyhpc/elasticluster/src/elasticluster/providers/gce.py", line 525, in start_instance

response = self._execute_request(request)

File "/home/odyhpc/elasticluster/src/elasticluster/providers/gce.py", line 208, in _execute_request

return request.execute(http=self._auth_http)

File "/home/odyhpc/elasticluster/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper

return wrapped(*args, **kwargs)

File "/home/odyhpc/elasticluster/lib/python3.8/site-packages/googleapiclient/http.py", line 907, in execute

raise HttpError(resp, content, uri=self.uri)

googleapiclient.errors.HttpError: <HttpError 403 when requesting https://compute.googleapis.com/compute/v1/projects/watchful-origin-244417/zones/us-central1-a/instances?alt=json returned "Request had insufficient authentication scopes.">

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/home/odyhpc/elasticluster/src/elasticluster/cluster.py", line 580, in _start_node

node.start()

File "/home/odyhpc/elasticluster/src/elasticluster/cluster.py", line 1312, in start

vm_data = self._cloud_provider.start_instance(

File "/home/odyhpc/elasticluster/src/elasticluster/providers/gce.py", line 531, in start_instance

raise InstanceError("Error creating instance `%s`" % e)

elasticluster.exceptions.InstanceError: Error creating instance `<HttpError 403 when requesting https://compute.googleapis.com/compute/v1/projects/watchful-origin-244417/zones/us-central1-a/instances?alt=json returned "Request had insufficient authentication scopes.">`

2020-08-10 18:08:11 deployator0002 elasticluster[3768] ERROR Could not start node `frontend001`: Error creating instance `<HttpError 403 when requesting https://compute.googleapis.com/compute/v1/projects/watchful-origin-244417/zones/us-central1-a/instances?alt=json returned "Request had insufficient authentication scopes.">` -- <class 'elasticluster.exceptions.InstanceError'>

Traceback (most recent call last):

File "/home/odyhpc/elasticluster/src/elasticluster/providers/gce.py", line 525, in start_instance

response = self._execute_request(request)

File "/home/odyhpc/elasticluster/src/elasticluster/providers/gce.py", line 208, in _execute_request

return request.execute(http=self._auth_http)

File "/home/odyhpc/elasticluster/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper

return wrapped(*args, **kwargs)

File "/home/odyhpc/elasticluster/lib/python3.8/site-packages/googleapiclient/http.py", line 907, in execute

raise HttpError(resp, content, uri=self.uri)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/home/odyhpc/elasticluster/src/elasticluster/cluster.py", line 580, in _start_node

node.start()

File "/home/odyhpc/elasticluster/src/elasticluster/cluster.py", line 1312, in start

vm_data = self._cloud_provider.start_instance(

File "/home/odyhpc/elasticluster/src/elasticluster/providers/gce.py", line 531, in start_instance

raise InstanceError("Error creating instance `%s`" % e)

2020-08-10 18:08:11 deployator0002 elasticluster[3768] ERROR Could not start node `compute001`: Error creating instance `<HttpError 403 when requesting https://compute.googleapis.com/compute/v1/projects/watchful-origin-244417/zones/us-central1-a/instances?alt=json returned "Request had insufficient authentication scopes.">` -- <class 'elasticluster.exceptions.InstanceError'>

Traceback (most recent call last):

File "/home/odyhpc/elasticluster/src/elasticluster/providers/gce.py", line 525, in start_instance

response = self._execute_request(request)

File "/home/odyhpc/elasticluster/src/elasticluster/providers/gce.py", line 208, in _execute_request

return request.execute(http=self._auth_http)

File "/home/odyhpc/elasticluster/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper

return wrapped(*args, **kwargs)

File "/home/odyhpc/elasticluster/lib/python3.8/site-packages/googleapiclient/http.py", line 907, in execute

raise HttpError(resp, content, uri=self.uri)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/home/odyhpc/elasticluster/src/elasticluster/cluster.py", line 580, in _start_node

node.start()

File "/home/odyhpc/elasticluster/src/elasticluster/cluster.py", line 1312, in start

vm_data = self._cloud_provider.start_instance(

File "/home/odyhpc/elasticluster/src/elasticluster/providers/gce.py", line 531, in start_instance

raise InstanceError("Error creating instance `%s`" % e)

2020-08-10 18:08:11 deployator0002 elasticluster[3768] WARNING ResourceWarning: unclosed running multiprocessing pool <multiprocessing.pool.ThreadPool state=RUN pool_size=3>

2020-08-10 18:08:11 deployator0002 elasticluster[3768] ERROR Could not start cluster `odycluster`: No nodes could be started!

2020-08-10 18:08:11 deployator0002 elasticluster[3768] ERROR Error: No nodes could be started!

Aborting because of errors: No nodes could be started!.

2020-08-10 18:08:11 deployator0002 elasticluster[3768] WARNING ResourceWarning: unclosed <ssl.SSLSocket fd=7, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.128.0.36', 54746), raddr=('173.194.197.95', 443)>

2020-08-10 18:08:11 deployator0002 elasticluster[3768] WARNING ResourceWarning: unclosed <socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.128.0.36', 52212), raddr=('169.254.169.254', 80)>

2020-08-10 18:08:11 deployator0002 elasticluster[3768] WARNING ResourceWarning: unclosed <ssl.SSLSocket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.128.0.36', 55886), raddr=('108.177.112.95', 443)>

The error (Request had insufficient authentication scopes) seems to be relatively general but it's unclear (to me) which is the seed problem before everything cascades down. One question that cropped up even before this was about the keys: I had not previously generated the gce_client_id or gce_client_secret keys; the thing is that, once on the OAuth 2.0 Client ID section, the system offered me 7 choices for application type but none of them fitted elasticluster's profile. I chose desktop app (as it was probably be the closest) but I am still wondering if elasticluster requires a specific type or if this could be in any way related to the said errors. I'm also attaching a redacted copy of my config file in case it's useful.

Thanks

cluster01_redacted.txt

Riccardo Murri

unread,

Aug 14, 2020, 4:54:30 PM8/14/20

to odyhpc Fdez, elasticluster

Hello Ody,

sorry for this late reply! I have tried starting a cluster with the
configuration you provided and, after swapping in my credentials, it
worked (see output log at end of email).

The only difference that I can find is that I have this additional
line in the `[cloud/google-cloud]` section:

noauth_local_webserver=True

As to your question, step by step instructions for creating the client
ID and secret keys are given here:
https://cloud.google.com/life-sciences/docs/tutorials/grid-engine-cluster#obtain_your_client_id_and_client_secret

Let me know if this helps!

Kind regards,
Riccardo

---- output from correctly starting cluster -----
(elasticluster) riccardo.murri@dev:~$ elasticluster -c ody.conf stop
-y odycluster
Destroying cluster `odycluster` ...
2020-08-14 20:46:12 dev elasticluster[12178] WARNING Node `compute001`
has no instance ID. Assuming it did not start correctly, so removing
it anyway from the cluster.
2020-08-14 20:46:12 dev elasticluster[12178] WARNING Node `compute002`
has no instance ID. Assuming it did not start correctly, so removing
it anyway from the cluster.
2020-08-14 20:46:12 dev elasticluster[12178] WARNING Node
`frontend001` has no instance ID. Assuming it did not start correctly,
so removing it anyway from the cluster.
(elasticluster) riccardo.murri@dev:~$ elasticluster -vv -c ody.conf
start odycluster
2020-08-14 20:47:53 dev elasticluster[12182] DEBUG Checking section
`cloud/google-cloud` ...
2020-08-14 20:47:53 dev elasticluster[12182] DEBUG Checking section
`login/google` ...
2020-08-14 20:47:53 dev elasticluster[12182] DEBUG Checking section
`cluster/odycluster` ...
2020-08-14 20:47:53 dev elasticluster[12182] DEBUG Checking section
`setup/gridengine` ...
2020-08-14 20:47:53 dev elasticluster[12182] DEBUG Using class <class
'elasticluster.providers.gce.GoogleCloudProvider'> from module <module
'elasticluster.providers.gce' from '/home/riccard
o.murri/elasticluster/elasticluster/providers/gce.py'> to instanciate
provider 'google'
2020-08-14 20:47:53 dev elasticluster[12182] DEBUG Using class <class
'elasticluster.providers.ansible_provider.AnsibleSetupProvider'> from
module <module 'elasticluster.providers.ansible_pr
ovider' from '/home/riccardo.murri/elasticluster/elasticluster/providers/ansible_provider.py'>
to instanciate provider 'ansible'

Starting cluster `odycluster` with:
* 1 frontend nodes.
* 2 compute nodes.
(This may take a while...)

2020-08-14 20:47:53 dev elasticluster[12182] INFO Starting cluster
nodes (timeout: 600 seconds) ...
2020-08-14 20:47:53 dev elasticluster[12182] DEBUG Note: starting 3
nodes concurrently.
2020-08-14 20:47:53 dev elasticluster[12182] DEBUG _start_node:
working on node `frontend001`
2020-08-14 20:47:53 dev elasticluster[12182] INFO Starting node
`frontend001` from image `debian-9-stretch-v20200805` with flavor
n1-standard-1 ...
2020-08-14 20:47:53 dev elasticluster[12182] WARNING UserWarning:
Cannot access /home/riccardo.murri/.elasticluster/storage/363333365221-s8atntam5ihgtdclh7odfni4jn7hmtrq.apps.googleuserconte

nt.com.oauth.dat: No such file or directory

2020-08-14 20:47:53 dev elasticluster[12182] INFO Determined that
provided credentials are not valid.
2020-08-14 20:47:53 dev elasticluster[12182] INFO Attempting to use
Google Application Default Credentials.
2020-08-14 20:47:53 dev elasticluster[12182] DEBUG _start_node:
working on node `compute001`
2020-08-14 20:47:53 dev elasticluster[12182] INFO Starting node
`compute001` from image `debian-9-stretch-v20200805` with flavor
n1-standard-1 ...
2020-08-14 20:47:53 dev elasticluster[12182] DEBUG _start_node:
working on node `compute002`
2020-08-14 20:47:53 dev elasticluster[12182] INFO Starting node
`compute002` from image `debian-9-stretch-v20200805` with flavor
n1-standard-1 ...
2020-08-14 20:47:53 dev elasticluster[12182] WARNING ResourceWarning:
unclosed <socket.socket fd=4, family=AddressFamily.AF_INET, type=2049,
proto=6, laddr=('10.166.0.2', 47110), raddr=('169
.254.169.254', 80)>
2020-08-14 20:48:06 dev elasticluster[12182] DEBUG Node `compute002`
has instance ID `odycluster-compute002`
2020-08-14 20:48:06 dev elasticluster[12182] INFO Node `compute002`
has been started.
2020-08-14 20:48:10 dev elasticluster[12182] DEBUG Node `frontend001`
has instance ID `odycluster-frontend001`
2020-08-14 20:48:10 dev elasticluster[12182] INFO Node `frontend001`
has been started.
2020-08-14 20:48:21 dev elasticluster[12182] DEBUG Node `compute001`
has instance ID `odycluster-compute001`
2020-08-14 20:48:21 dev elasticluster[12182] INFO Node `compute001`
has been started.

Reply all

Reply to author

Forward

0 new messages