Frequent [/usr/bin/ssh] exited with return code [255] error

[Jason Mellone]

I have a locally (ubuntu 20.04.1) orchestrated workflow the looks like the below.

My gcloud -v yields:

gcloud --version
Google Cloud SDK 317.0.0
alpha 2020.10.30
beta 2020.10.30
bq 2.0.62
core 2020.10.30
gsutil 4.54
kubectl 1.16.13

Code stuff:

Crontab at 19:15 launch 10 shell scripts

Each shell script looks like:

client_name=$1
instance_name_arg=imagename${client_name}
z=us-east1-b

gcloud beta compute --project=myname-123456 instances create $instance_name_arg --zone=${z} --machine-type=g1-small --subnet=default --network-tier=PREMIUM --source-machine-image projects/myname-165

gcloud compute instances start $instance_name_arg --zone=${z}


gcloud compute ssh --zone=${z} myname@${instance_name_arg} -- '/home/myname/miniconda3/envs/minicrypto/bin/python /home/myname/foo/bar/baz/bang.py'

gcloud compute instances stop $instance_name_arg --zone=${z}

Some days, all of these will run, other days, some of these will have the 255 error. If I rerun them, they'll work.

I am open to any solution that is the "correct" or "community suggested" solution.

This is reasonably homegrown, and I don't think it should be the best way to do it, but seemed reasonable.

Thank you very much for any guidance!!

[sneacsu]

Regarding the script seems fine for me and I am not sure what other ways would be better to improve it but regarding the error you are getting is more or less related to UFW when it is turned on it denies any incoming connection, so in this case you will need to disable it for port 22 that uses the SSH in order to run your scripts.

In order to fix this since it is locally is to add to your script `sudo ufw allow ssh` which will enable port 22 for SSH and the error should stop from happening.

[Chris Gerber]

You can include SSH options in your command to increase the verbosity. That may help diagnose the error. They need to appear after the double dash. It would probably look like this:

gcloud compute ssh --zone=${z} myname@${instance_name_arg} -- -vvv '/home/myname/miniconda3/envs/minicrypto/bin/python /home/myname/foo/bar/baz/bang.py'

--
You received this message because you are subscribed to the Google Groups "Google Cloud Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-dev/732e07f5-d393-4713-a933-12adfa58eb0dn%40googlegroups.com.

[Jason Mellone]

Hey Chris,

Thanks for the reply.

What I get back after adding the -vvv verbosity flag is:

Starting instance(s) imageclient...done.                                                                                                                                                                  
Updated [https://compute.googleapis.com/compute/v1/projects/myname-123456/zones/us-east1-b/instances/imageclient].
Instance internal IP is xx.xxx.x.xx
Instance external IP is zz.zz.zzz.zz
OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /home/myname/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname zz.zz.zzz.zz is address
debug2: ssh_connect_direct
debug1: Connecting to zz.zz.zzz.zz [zz.zz.zzz.zz] port 22.
debug1: connect to address zz.zz.zzz.zz port 22: Connection refused
ssh: connect to host zz.zz.zzz.zz port 22: Connection refused
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
Stopping instance(s) imageclient...done.                                                                                                                                                                  
Updated [https://compute.googleapis.com/compute/v1/projects/myname-123456/zones/us-east1-b/instances/imageclient].

This dir is empty: /etc/ssh/ssh_config.d/

/home/myname/.ssh/config contains info for another instance on the same project, but not the one I am trying to connect to above.

I think this may be key to the issue. How do I purposefully add these machines to my /home/myname/.ssh/config?

My /etc/ssh/ssh_config looks like:

Include /etc/ssh/ssh_config.d/*.conf

SendEnv LANG LC_*

HashKnownHosts yes

GSSAPIAuthentication yes

Things like:

Port 22

Are commented out.

Jason Mellone
jason....@gmail.com

You received this message because you are subscribed to a topic in the Google Groups "Google Cloud Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-cloud-dev/oQibZYN4hrc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-cloud-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-dev/CAAQOVE-pJrWYSzS6eKfJnBhEu5H0KG6bseq2tzF04FSh74WPsg%40mail.gmail.com.

[Chris Gerber]

The error indicates that the target host isn't listening for an incoming SSH request. It may be worth checking that the instance has not stopped. Or, if it was recently started, it may not have started running the SSH server yet and you'll want to wait and retry. 

[Jason Mellone]

Thanks Chris. That makes sense. I'll try what you suggested.