Cannot SSH into my instance

[Facundo Farias]

Hello there,

I've created a new instance yesterday, and so far so good. I was able to configure the instance, and it is working as expected.

But then, today I am trying to connect to my instance with the command:

gcloud compute ssh my-instance-params

But I am getting this error code instead:

Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].

Not sure why yesterday was working, but today it doesn't. If I try to connect from the "web SSH console" in the GCE console, it works. But then I copy the command, and I put it into the console, but it does not work. Bit strange.

Any ideas?

Thanks

[Navi Aujla (Google Cloud Support)]

Hello Facundo

You can get more details using the flag "--verbosity=debug" when using the command: 'gcloud compute ssh'. Verify, if there are keys present under the user's home directory.

On Linux/Unix based systems: 

cat ~/.ssh/authorized_keys 

On Windows system with Google Cloud SDK installed: 

Path: C:\Users\user-name\.ssh

Moreover, Are other users able to login to the VM instance using Google cloud shell or if problem is limited to specific users? 

[Gerrit DeWitt]

Hello Facundo,

I can help you understand what's going on here.  Here is a bit of background to understand how key-based SSH is managed using Google Cloud Platform:

BACKGROUND INFORMATION

1.  Each of your Linux VM instances needs to be running a the Google Linux Guest Environment [1] or an equivalent.  If you create an instance from one of the templates, the Guest Environment or an equivalent is included.  (Some operating systems like CoreOS include Guest Environment functionality for Google and other Cloud providers as part of the OS.)

2.  Key based SSH involves a key pair:  a private key and a public key.  Using Google Compute Engine, the public key information can be stored in project or instance metadata.  The Guest Environment on each VM instance keeps an open HTTP GET request to a metadata server (essentially, running at the hypervisor level) to receive metadata updates, including public keys.  The Guest Environment is responsible for creating user accounts and copying the public key information into ~/.ssh/authorized_keys within the home directory of the user in the VM instance itself.

3.  When you SSH in the Browser [2], you have to be logged into the Google Cloud Console as a user with the compute instance admin role [3].  A temporary private key is generated on-demand and held in your browser.  The corresponding public key is created and added to project wide or instance specific metadata.  The public key has additional information associated with it, including an expiry timestamp, which renders it invalid after a few minutes.  The public key is set on the project's metadata unless the instance to which you're connecting via SSH has the "block project wide SSH keys" attribute set; in that case, the public key is set on the instance's metadata.  This metadata appears as ssh-keys attribute (or the legacy sshKeys attribute).

4.  When you SSH using the gcloud tool (for example, gcloud compute ssh) [4], you have to be authenticated to the gcloud tool as a compute instance admin.  Assuming you're authenticated in that way, gcloud creates a SSH keypair the first time you SSH into an instance in your project.  The SSH key pair it creates stores files inside your home directory on your computer in these locations:

   ~/.ssh/google_compute_engine # the private key

   ~/.ssh/google_compute_engine.pub # the public key

Please note the above locations are on your own computer, specific to your user account.

Similar to SSH in Browser, the public key is transferred to project wide or instance specific metadata.  The public key does not have an expiry, but it does have information about your IAM user name (the user to which you authenticated to gcloud) and your computer's host name associated with it.  The public key is set on the project's metadata unless the instance to which you're connecting via SSH has the "block project wide SSH keys" attribute set; in that case, the public key is set on the instance's metadata.  This metadata appears as ssh-keys attribute (or the legacy sshKeys attribute).

The private key is not copied anywhere else - the only copy is on your computer.

With that in mind, let's get back to your issue...

YOUR ISSUE

From your description, you're able to connect to your instance via SSH in Browser, so we know that your VM instance has a working Linux Guest Environment or equivalent.  (If it did not, it wouldn't have received the temporary public key and SSH in Browser would have failed.)

You're getting a "permission denied" error with gcloud compute ssh, which means that SSH connectivity from your computer to the instance is working.  You've just failed authentication.

Here's what I would check:

1.  First make sure you're logged into gcloud as an IAM user with compute instance admin.  You can check list active logins with:

   gcloud auth list

If necessary, switch to the correct account (gcloud config set account [your-iam-user]).  You can also completely log out of all accounts and then back in using:

  gcloud auth revoke --all

  gcloud auth login [your-iam-user]

2.  Once you're sure you're logged in with the right IAM user, try connecting via SSH again (gcloud compute ssh [INSTANCE-NAME]).  If that still doesn't work, re-create your SSH key by doing the following:

   a.  Move your existing GCP key pair aside:

   mv ~/.ssh/google_compute_engine ~/.ssh/old-google_compute_engine

    mv ~/.ssh/google_compute_engine.pub ~/.ssh/old-google_compute_engine.pub

   b.  Connect to your instance via SSH.  This time, you'll be prompted to re-create your key pair:

   gcloud compute ssh [INSTANCE-NAME]

You'll see a message indicating you don't have a key pair, and you'll have the opportunity to generate a new one.  Here, gcloud is running ssh-keygen [5] for you.  You can enter a passphrase (twice) to protect your private key, or you can press return (twice) to leave the private key un-encrypted.  If you passphrase protect your private key, you'll have to enter that passphrase whenever you SSH with the key.

   WARNING: The public SSH key file for gcloud does not exist.

   WARNING: The private SSH key file for gcloud does not exist.

   WARNING: You do not have an SSH key for gcloud.

   WARNING: SSH keygen will be executed to generate a key.

   Generating public/private rsa key pair.

   Enter passphrase (empty for no passphrase): 

   c.  It will take a few moments to transfer the public key to your project's or instance's metadata.  After that, SSH should work.  Two new files were created on your computer - these are your new keys!

   ~/.ssh/google_compute_engine

    ~/.ssh/google_compute_engine.pub

3.  If you still cannot SSH into your instance, let us know.

--Gerrit

Cloud TSE, Seattle

References

1:  https://github.com/GoogleCloudPlatform/compute-image-packages#background

2:  https://cloud.google.com/compute/docs/ssh-in-browser

3:  https://cloud.google.com/compute/docs/access/iam#compute_instance_admin_role

4:  https://cloud.google.com/compute/docs/instances/connecting-to-instance#sshingcloud

5:  https://linux.die.net/man/1/ssh-keygen

[Harpreet Singh]

Hi,

I followed the steps mentioned by you however still having the same issues. 

Permission denied (publickey).ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].

I am not sure, what's I am doing wrong. 

Harpreet 

[Gerrit DeWitt]

Hello Harpreet,

You might want to make sure that the problematic instance is running its Linux Guest Environment.  Instructions for re-installing the Guest Environment are here:  https://cloud.google.com/compute/docs/instances/linux-guest-environment

These directions also include how to determine if a Guest Environment is running based on console logs, installed packages, or running services.  If you can't connect to the instance, you can restart it via the Cloud Console and inspect the console logs, looking to see if the necessary Guest Environment components load.

--Gerrit

[veerender reddy]

hi,
 
I tried to copy file from my local machine to cloud using following command

root@instance-1:~# gcloud compute scp ~/kaldi-trunk/egs/timit/s5/run1.sh \ veerenderb@instance-1:~/home/veerender/tumpri --zone us-east1-b

It resulted in following error:

ERROR: (gcloud.compute.scp) Could not SSH into the instance. It is possible that your SSH key has not propagated to the instance yet. Try running this command again. If you still cannot connect, verify that the firewall and instance are set to accept ssh traffic.

How to resolve the issue???

[Arne Claus]

FIX (for us, temporary)

SSH stopped working for newly build images yesterday at around 20:00 CEST.

We are running apt-get upgrade when provisioning our custom images.
After removing this step we're able to login again.

The following packages were being upgraded (and are now not upgraded anymore):

    googlecompute: (Reading database ... 31303 files and directories currently installed.)

    googlecompute: Preparing to unpack .../0-curl_7.52.1-5+deb9u4_amd64.deb ...

    googlecompute: Unpacking curl (7.52.1-5+deb9u4) over (7.52.1-5+deb9u3) ...

    googlecompute: Preparing to unpack .../1-libcurl3_7.52.1-5+deb9u4_amd64.deb ...

    googlecompute: Unpacking libcurl3:amd64 (7.52.1-5+deb9u4) over (7.52.1-5+deb9u3) ...

    googlecompute: Preparing to unpack .../2-google-cloud-sdk_186.0.0-0_all.deb ...

    googlecompute: Unpacking google-cloud-sdk (186.0.0-0) over (183.0.0-0) ...

    googlecompute: Preparing to unpack .../3-google-compute-engine-oslogin_1.1.4-1+deb9_amd64.deb ...

    googlecompute: Unpacking google-compute-engine-oslogin (1.1.4-1+deb9) over (1.1.2-1+deb9) ...

    googlecompute: Preparing to unpack .../4-google-compute-engine_2.7.5-1_all.deb ...

    googlecompute: Unpacking google-compute-engine (2.7.5-1) over (2.7.2-2) ...

    googlecompute: Preparing to unpack .../5-python-google-compute-engine_2.7.5-1_all.deb ...

    googlecompute: Unpacking python-google-compute-engine (2.7.5-1) over (2.7.2-2) ...

    googlecompute: Preparing to unpack .../6-python3-google-compute-engine_2.7.5-1_all.deb ...

    googlecompute: Unpacking python3-google-compute-engine (2.7.5-1) over (2.7.2-2) ...

    googlecompute: Preparing to unpack .../7-libisc-export160_1%3a9.10.3.dfsg.P4-12.3+deb9u4_amd64.deb ...

    googlecompute: Unpacking libisc-export160 (1:9.10.3.dfsg.P4-12.3+deb9u4) over (1:9.10.3.dfsg.P4-12.3+deb9u3) ...

    googlecompute: Preparing to unpack .../8-libdns-export162_1%3a9.10.3.dfsg.P4-12.3+deb9u4_amd64.deb ...

    googlecompute: Unpacking libdns-export162 (1:9.10.3.dfsg.P4-12.3+deb9u4) over (1:9.10.3.dfsg.P4-12.3+deb9u3) ...

    googlecompute: Setting up libisc-export160 (1:9.10.3.dfsg.P4-12.3+deb9u4) ...

    googlecompute: Setting up libcurl3:amd64 (7.52.1-5+deb9u4) ...

    googlecompute: Setting up python3-google-compute-engine (2.7.5-1) ...

    googlecompute: Setting up libdns-export162 (1:9.10.3.dfsg.P4-12.3+deb9u4) ...

    googlecompute: Setting up google-compute-engine-oslogin (1.1.4-1+deb9) ...

    googlecompute: Processing triggers for libc-bin (2.24-11+deb9u1) ...

    googlecompute: Processing triggers for man-db (2.7.6.1-2) ...

    googlecompute: Setting up google-cloud-sdk (186.0.0-0) ...

    googlecompute: Setting up python-google-compute-engine (2.7.5-1) ...

    googlecompute: Processing triggers for rsyslog (8.24.0-1) ...

    googlecompute: Setting up curl (7.52.1-5+deb9u4) ...

    googlecompute: Setting up google-compute-engine (2.7.5-1) ...

    googlecompute: Installing new version of config file /etc/dhcp/dhclient-exit-hooks.d/google_set_hostname ...

[Patrick Muturi]

Hello,

I was able to setup my instance and installed my software successfully but when I try to connect am getting the following error;

The VM Guest enviroment is outdated and supports the depreciated SSH Keys metadata item.

How Do I resolve that

[Justin Reiners]

have you tried logging in with gcloud api?

gcloud compute ssh <instancename>

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-discussion@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to gce-discussion@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/49b98c5f-20e4-4848-a9ad-6229ae2a5ae8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

.com

Justin Reiners / System Administrator

800.308.9712/ jus...@hotlinesinc.com

Hotlines Inc Office: 800.807.2867 / Fax: 712-388-0258 
427 E. Kanesville Blvd. Suite 403, Council Bluffs, IA 51503
http://www.partshotlines.com

[Patrick Muturi]

Hello Justin,

Let me try and get back to you

On Tue, Mar 13, 2018 at 5:21 PM, Justin Reiners <jus...@hotlinesinc.com> wrote:

have you tried logging in with gcloud api?

gcloud compute ssh <instancename>

On Tue, Mar 13, 2018 at 3:21 AM, Patrick Muturi <pmu...@talinda.net> wrote:

Hello,

I was able to setup my instance and installed my software successfully but when I try to connect am getting the following error;

The VM Guest enviroment is outdated and supports the depreciated SSH Keys metadata item.

How Do I resolve that

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-discussion@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussion+unsubscribe@googlegroups.com.

To post to this group, send email to gce-dis...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/49b98c5f-20e4-4848-a9ad-6229ae2a5ae8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

.com

Justin Reiners / System Administrator

800.308.9712/ justin@hotlinesinc.com

Hotlines Inc Office: 800.807.2867 / Fax: 712-388-0258 
427 E. Kanesville Blvd. Suite 403, Council Bluffs, IA 51503
http://www.partshotlines.com

--

Kind Regards

Patrick Muturi

Cell:+254 727 917 328

Skype: pmuturis2

Presales Engineer

[Patrick Muturi]

Hello Justin,

Yes I can access via gcloud as shown below but i need access via graphical User, would I need to change the private IP Address to my local IP adress

[Timothy Wu]

Hi Gerrit

This is happening for me as well now. Cannot `gcloud compute ssh` into instance when previously possible. Have followed your steps here to no avail, getting error

ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].

and with flag `--ssh-flag="-vvv"`

OpenSSH_7.4p1, LibreSSL 2.5.0

debug1: Reading configuration data /etc/ssh/ssh_config

debug2: resolving "xxx.xxx.xxx.xxx" port 22

debug2: ssh_connect_direct: needpriv 0

debug1: Connecting to xxx.xxx.xxx.xxx [xxx.xxx.xxx.xxx] port 22.

debug1: connect to address xxx.xxx.xxx.xxx port 22: Host is down

ssh: connect to host xxx.xxx.xxx.xxx port 22: Host is down

ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].

This seems to have happened after updating kubernetes `1.9.4-gke.1`.

Any ideas?

Thanks

Tim

[Gerrit DeWitt]

Hi Tim,

Usually re-installing the Linux Guest Environment should be a remedy of last resort (for example, after importing an image, doing an OS upgrade, making a mistake, etc.).  So that might not be your issue here.  Here are two reasons why I think your issue is different.  Thanks for providing the verbose ssh output - that helps!

Given that this is Kubernetes related, I’m assuming you’re trying to SSH into a node pool instance...

1.  In my experience, gcloud always returns error 255 when the underlying system ssh process is unable to connect.  So that error 255 code is not specific to one problem at all.

2.  The verbose ssh output shows that the client is unable to contact the server on TCP 22.

With (2) in mind, here are a few reasons why you may not be able to connect to the instance:

* Remember that gcloud compute ssh requires that the target instance have a public IP.  (If you’re connected via Cloud Interconnect or Cloud VPN, you can use gcloud beta compute ssh --internal-ip, but otherwise, the instance needs a public IP.)

* Do you have any VPC firewall rules that are prohibiting incoming access on TCP 22 to the instance in question?  By default, you probably do!  For all VPCs, there’s the default deny ingress rule, but there are a couple of exceptions.  The default VPC network has some additional firewall rules that allow SSH from any source (by default, unless you remove them).  Google-owned projects have firewall rules automatically manipulated by GCE Enforcer, though there are some rules that are added to allow connectivity from RFC 1918 IP spaces.  So check your firewall rules, even if you already did that recently.  :)

* On the instance, it's possible that ssd isn’t running or properly able to respond.  Look at your instance’s console logs to make sure it finished starting up.  Do you see any clues there?

* Are you able to create another instance and connect to it via SSH successfully?

Hopefully this will help you get started.  Let me know if you need further help!

—Gerrit

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-dis...@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---
You received this message because you are subscribed to a topic in the Google Groups "gce-discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gce-discussion/O-c10TM4ZLM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gce-discussio...@googlegroups.com.

To post to this group, send email to gce-dis...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/c954625b-85a6-42c2-bd87-98b2fac25511%40googlegroups.com.

[Timothy Wu]

Hi Gerrit

Thanks for the quick reply. Am able to access node pool instances without problems using `kubectl exec -it ... /bin/bash`. The issue is still accessing the host machines running CoreOS. As these are machines setup via the cloud console, I have not made any modifications to their defaults. Have also not made any changes to firewall rules and given was able to connect before upgrading to `1.9.4-gke.1` am assuming the issue was introduced recently, although could be an earlier version or something else entirely.

Have checked the firewall rules as you suggest and have this `ingress` record

	Name	Targets	Source filters	Protocols/ports	Action	Priority	Network

   ...

   default-allow-ssh    Apply to all     IP ranges: 0.0.0.0/0    tcp:22    Allow          65534          default

Have also checked my SSH keys in both the cloud console as well as on the machine itself under `~/.ssh` and they seem to be fine (via cloud console SSH login). Very strange, it seems something started blocking it in one of google's updates to CoreOS?

Tim

[Gerrit DeWitt]

Hi Tim,

Would it be possible for you to demonstrate the problem via Hangouts (perhaps this Friday)?  I have some time that afternoon.

—Gerrit

To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/269c2ba5-cf15-4ba6-9c26-b4324e2a6b72%40googlegroups.com.

[Timothy Wu]

Ok, let me follow up privately to arrange a time.

—Gerrit

Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-discussion@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.

--- 
You received this message because you are subscribed to a topic in the Google Groups "gce-discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gce-discussion/O-c10TM4ZLM/unsubscribe.

To unsubscribe from this group and all its topics, send an email to gce-discussion+unsubscribe@googlegroups.com.
To post to this group, send email to gce-discussion@googlegroups.com.

[Timothy Wu]

Following up here after call with Gerrit in case someone with similar case to mine stumbles upon this. In my case it was not restarting after little snitch update!

Tim

[Rob Smith]

Hi Timothy,

Sounds like I have a very similar problem to the one you had. Cannot access VM through browser SSH or gcloud SSH. Could you walk me through the steps that resolved the issue?

Thanks!

—Gerrit

[Brian McMahon]

Hi,

I am also running into error message `Permission denied (publickey).

ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].` when trying to connect to instance with `gcloud compute ssh <instance name>`

I have tried adjusting the IAM settings to add another key, as well as replace existing keys using the method outlined by Gerrit above, but neither seemed to work.  I can connect to the instance via other computers, so perhaps there is a key conflict but I can't seem to track down why this would be an issue and how to overcome it.  

My traceback from "--verbosity=debug" is as follows:

`Permission denied (publickey).

DEBUG: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].

Traceback (most recent call last):

  File "/Users/brianmcmahon/Desktop/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 841, in Execute

    resources = calliope_command.Run(cli=self, args=args)

  File "/Users/brianmcmahon/Desktop/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 770, in Run

    resources = command_instance.Run(args)

  File "/Users/brianmcmahon/Desktop/google-cloud-sdk/lib/surface/compute/ssh.py", line 203, in Run

    return_code = cmd.Run(ssh_helper.env, force_connect=True)

  File "/Users/brianmcmahon/Desktop/google-cloud-sdk/lib/googlecloudsdk/command_lib/util/ssh/ssh.py", line 1021, in Run

    raise CommandError(args[0], return_code=status)

CommandError: [/usr/bin/ssh] exited with return code [255].

ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].`

I anxiously await your reply - many thanks in advance!

Brian

[Steve Lorimer]

Have you double checked the public key you are trying to connect with is configured for either your project or the VM instance?

Do you rely on the VM having a global set of public keys installed? (ie: on a project wide basis). If so, check in "SSH Keys" under "Metadata" for your project in the cloud console and confirm your key is listed. VMs configured to have project wide keys added will have these keys added.

If the VM has "Block project-wide SSH keys" selected, then the above won't work - it will instead only have the keys individually added to the VM instance uploaded. You will need to edit the VM instance and select "Add item" under "SSH keys".

If you add a key in either of the above two methods (either project-wide under Metadata; or VM-specific in the VM instance config) you will need to restart your VM so the key can be uploaded.

--

© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-dis...@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---

You received this message because you are subscribed to the Google Groups "gce-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussio...@googlegroups.com.

To post to this group, send email to gce-dis...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/b6f5e75c-770a-447e-b495-8c046b54f7ea%40googlegroups.com.

[Brian McMahon]

Hi Steve, 

Thanks for your response and detailed guidance.  I did confirm that the public key was listed on Metadata/SSH Keys.  I was not able to find the box to un-select "Block project-wide SSH keys".     

I was, however, able to login to the instance as a new user after enabling the OS Login function.  

Many thanks,

Brian

[Veera (Google Cloud Support)]

Hello Brian,

I am glad that you were able to resolve the issue using OS login feature. 

In regards to allow project-wide public SSH keys, you will be able to mange this function using console, gcloud command and API by following this article.

[Ryan Lin]

Hi Gerrit. 

I'm having the same problem when I tried to SSH to my VM instance.

"Permission denied (publickey). ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255]."

I have followed the steps in your reply to Facundo and it didn't help. I also tried to reinstall the Linux guest environment but received:

"mv: cannot move '/tmp/sdb-root-vol/etc/rc.local' to '/tmp/sdb-root-vol/etc/moved-rc.local': No space left on device"

in "moving any existing rc.local script aside" step. Seems some problems with the disk cuz I can SSH to newly created instances. I've tried all the solutions available to me online but still can't fix it. Can you plz help me with the problem? Thank you.

Ryan

[Justin Reiners]

"

 No space left on device"

is your disk full? delete some logs and try and ssh again.

df -h should tell you

if you need to find large files, you can install ncdu, but I'd check /var/log and clean some space up. If there is no space, google can't copy the ssh keys onto the host when you log in.

--

© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-dis...@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gce-discussio...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gce-discussion/d479cbea-71ff-4499-9c29-4d85ddf3fa4c%40googlegroups.com.

[Ryan Lin]

Thanks for the rely. I just checked df -h in cloudshell and got:

Filesystem      Size  Used Avail Use% Mounted on

overlay          41G   33G  7.8G  81% /

tmpfs            64M     0   64M   0% /dev

tmpfs           848M     0  848M   0% /sys/fs/cgroup

/dev/sda1        41G   33G  7.8G  81% /root

/dev/sdb1       4.8G  2.4G  2.3G  51% /home

overlayfs       1.0M  164K  860K  17% /etc/ssh/keys

shm              64M     0   64M   0% /dev/shm

tmpfs           848M  736K  847M   1% /run/metrics

overlayfs       1.0M  164K  860K  17% /etc/ssh/ssh_host_dsa_key

tmpfs           848M     0  848M   0% /run/google/devshell

No idea if it means the disk is full. I've tried to increase the disk size of my problematic instance before but didn't help.

On Friday, August 16, 2019 at 1:49:13 PM UTC-4, Justin Reiners wrote:

" No space left on device"

is your disk full? delete some logs and try and ssh again.

df -h should tell you

if you need to find large files, you can install ncdu, but I'd check /var/log and clean some space up. If there is no space, google can't copy the ssh keys onto the host when you log in.

On Fri, Aug 16, 2019 at 12:25 PM Ryan Lin <rya...@tasteguru.ai> wrote:

Hi Gerrit. 

I'm having the same problem when I tried to SSH to my VM instance.

"Permission denied (publickey). ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255]."

I have followed the steps in your reply to Facundo and it didn't help. I also tried to reinstall the Linux guest environment but received:

"mv: cannot move '/tmp/sdb-root-vol/etc/rc.local' to '/tmp/sdb-root-vol/etc/moved-rc.local': No space left on device"

in "moving any existing rc.local script aside" step. Seems some problems with the disk cuz I can SSH to newly created instances. I've tried all the solutions available to me online but still can't fix it. Can you plz help me with the problem? Thank you.

Ryan

On Friday, January 5, 2018 at 9:24:25 PM UTC-5, Gerrit DeWitt wrote:

Hello Harpreet,

You might want to make sure that the problematic instance is running its Linux Guest Environment.  Instructions for re-installing the Guest Environment are here:  https://cloud.google.com/compute/docs/instances/linux-guest-environment

These directions also include how to determine if a Guest Environment is running based on console logs, installed packages, or running services.  If you can't connect to the instance, you can restart it via the Cloud Console and inspect the console logs, looking to see if the necessary Guest Environment components load.

--Gerrit

--
© 2018 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Compute Engine Discussion Google Group (gce-dis...@googlegroups.com) to participate in discussions with other members of the Google Compute Engine community and the Google Compute Engine Team.
---
You received this message because you are subscribed to the Google Groups "gce-discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gce-dis...@googlegroups.com.

[Ahmad P - Cloud Platform Support]

It seems that you are trying to “Clone boot disk and use startup script” as per this document[1]. It could be that the new instance disk is smaller than the disk that was assigned to the problematic instance.

It can be firewall running within the instance, to solve this situation, you need to log into the instance via Serial Console[2](since currently is not possible via SSH) and whitelist the IP from where you connect to the instance.

[1]  https://cloud.google.com/compute/docs/images/install-guest-environment#startup_script

[2] https://cloud.google.com/compute/docs/instances/interacting-with-serial-console#before-you-begin 

[ANTONIO CACHUAN]

Hi, 

I followed all your response. I would appreciate your help. I’m running a gcloud step with gcr.io/cloud-builders/gcloud using Cloud Builder (the Cloud Builder account service has all the privileges necessary-admin) and I want to copy some files from the /workspace/ to a Compute Engine Instance (created by Cloud Builder using Terraform all in the same project) but I’m getting

Step #6 — “gcloud replace files”: ERROR: (gcloud.compute.scp) Could not SSH into the instance.

My gcloud command

gcloud compute scp abc.properties instance_name:/tmp — zone=$vm_zone — quiet — project=$gcp_project_id — verbosity=”debug”

My Log (from bottom to up)

Step #6 - "gcloud replace files": DEBUG: Running command [/usr/bin/ssh -T -i /builder/home/.ssh/google_compute_engine -o CheckHostIP=no -o HostKeyAlias=compute.6367194xxxxx2630403 -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/builder/home/.ssh/google_compute_known_hosts ro...@x.x.x.x -- true].

Step #6 - "gcloud replace files": Permission denied (publickey).

Step #6 - "gcloud replace files": Warning: Permanently added 'compute.636xxx12630403' (ECDSA) to the list of known hosts.

Step #6 - "gcloud replace files": DEBUG: Executing command: [u'/usr/bin/ssh', u'-T', u'-i', u'/builder/home/.ssh/google_compute_engine', u'-o', u'CheckHostIP=no', u'-o', u'HostKeyAlias=compute.63671947xxxxxx403', u'-o', u'IdentitiesOnly=yes', u'-o', u'StrictHostKeyChecking=no', u'-o', u'UserKnownHostsFile=/builder/home/.ssh/google_compute_known_hosts', u'ro...@x.x.x.x', u'--', u'true']

Step #6 - "gcloud replace files": DEBUG: Running command [/usr/bin/ssh -T -i /builder/home/.ssh/google_compute_engine -o CheckHostIP=no -o HostKeyAlias=compute.636719xxxxx0403 -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/builder/home/.ssh/google_compute_known_hosts ro...@x.x.x.x -- true].

Step #6 - "gcloud replace files": Waiting for SSH key to propagate.

Step #6 - "gcloud replace files": .done.

Step #6 - "gcloud replace files": Updated [https://www.googleapis.com/compute/v1/projects/pilotxxxxx].

Step #6 - "gcloud replace files": warnings: []>]

Step #6 - "gcloud replace files": user: u'xxx...@cloudbuild.gserviceaccount.com'

Step #6 - "gcloud replace files": targetLink: u'https://www.googleapis.com/compute/v1/projects/pilotxxxxx'

Step #6 - "gcloud replace files": targetId: 131xxxx33173

Step #6 - "gcloud replace files": status: StatusValueValuesEnum(DONE, 0)

Step #6 - "gcloud replace files": startTime: u'2019-11-08T11:27:00.781-08:00'

Step #6 - "gcloud replace files": selfLink: u'https://www.googleapis.com/compute/v1/projects/pilotxxxx/global/operations/operation-15732xx20221-596xxxx13ca5b-5af18ee0-169xxx600'

Step #6 - "gcloud replace files": progress: 100

Step #6 - "gcloud replace files": operationType: u'compute.projects.setCommonInstanceMetadata'

Step #6 - "gcloud replace files": name: u'operation-157xxxx221-596dxxxa5b-5afxxxe0-16987600'

Step #6 - "gcloud replace files": kind: u'compute#operation'

Step #6 - "gcloud replace files": insertTime: u'2019-11-08T11:27:00.776-08:00'

Step #6 - "gcloud replace files": id: 4494473xxx605035

Step #6 - "gcloud replace files": endTime: u'2019-11-08T11:27:14.525-08:00'

Step #6 - "gcloud replace files": .DEBUG: Operations to inspect: [<Operation

Step #6 - "gcloud replace files": warnings: []>]

Step #6 - "gcloud replace files": user: u'xxx...@cloudbuild.gserviceaccount.com'

Step #6 - "gcloud replace files": targetLink: u'https://www.googleapis.com/compute/v1/projects/pilotx-xxxxx-xxx'

Step #6 - "gcloud replace files": targetId: 1317xxx133173

Step #6 - "gcloud replace files": status: StatusValueValuesEnum(RUNNING, 2)

Step #6 - "gcloud replace files": startTime: u'2019-11-08T11:27:00.781-08:00'

Step #6 - "gcloud replace files": selfLink: u'https://www.googleapis.com/compute/v1/projects/pilotx-xxxx-xxx/global/operations/operation-157xxx0221-5xxxxx3ca5b-5afxxee0-169xx7600'

Step #6 - "gcloud replace files": progress: 0

Step #6 - "gcloud replace files": operationType: u'compute.projects.setCommonInstanceMetadata'

Step #6 - "gcloud replace files": name: u'operation-157xxxx20221-596dac413ca5b-5af18ee0-16987600'

Step #6 - "gcloud replace files": kind: u'compute#operation'

Step #6 - "gcloud replace files": insertTime: u'2019-11-08T11:27:00.776-08:00'

Step #6 - "gcloud replace files": id: 449447xxxx08605035

Step #6 - "gcloud replace files": .........................DEBUG: Operations to inspect: [<Operation

Step #6 - "gcloud replace files": .DEBUG: Sleeping for 5s.

Step #6 - "gcloud replace files": warnings: []>]

Step #6 - "gcloud replace files": user: u'xx...@cloudbuild.gserviceaccount.com'

Step #6 - "gcloud replace files": targetLink: u'https://www.googleapis.com/compute/v1/projects/pilotxxxxx'

Step #6 - "gcloud replace files": targetId: 131xxx33173

Step #6 - "gcloud replace files": status: StatusValueValuesEnum(RUNNING, 2)

Step #6 - "gcloud replace files": startTime: u'2019-11-08T11:27:00.781-08:00'

Step #6 - "gcloud replace files": selfLink: u'https://www.googleapis.com/compute/v1/projects/pilotxxxx/global/operations/operation-15xx1220221-596xxx13ca5b-5axxee0-16987600'

Step #6 - "gcloud replace files": progress: 0

Step #6 - "gcloud replace files": operationType: u'compute.projects.setCommonInstanceMetadata'

Step #6 - "gcloud replace files": name: u'operation-15732xxxx0221-596xxxx3ca5b-5axxxee0-16987600'

Step #6 - "gcloud replace files": kind: u'compute#operation'

Step #6 - "gcloud replace files": insertTime: u'2019-11-08T11:27:00.776-08:00'

Step #6 - "gcloud replace files": id: 4494473816xxx05035

Step #6 - "gcloud replace files": ....................DEBUG: Operations to inspect: [<Operation

Step #6 - "gcloud replace files": .DEBUG: Sleeping for 4s.

Step #6 - "gcloud replace files": warnings: []>]

Step #6 - "gcloud replace files": user: u'xxx...@cloudbuild.gserviceaccount.com'

Step #6 - "gcloud replace files": targetLink: u'https://www.googleapis.com/compute/v1/projects/pilotxxxxx'

Step #6 - "gcloud replace files": targetId: 131735xxx173

Step #6 - "gcloud replace files": status: StatusValueValuesEnum(RUNNING, 2)

Step #6 - "gcloud replace files": startTime: u'2019-11-08T11:27:00.781-08:00'

Step #6 - "gcloud replace files": selfLink: u'https://www.googleapis.com/compute/v1/projects/pilotxxxxxx/global/operations/operation-157324xx0221-596daxxa5b-5af18ee0-169xx00'

Step #6 - "gcloud replace files": progress: 0

Step #6 - "gcloud replace files": operationType: u'compute.projects.setCommonInstanceMetadata'

Step #6 - "gcloud replace files": name: u'operation-1573xxxxx1-596daxxx5b-5axxee0-16987600'

Step #6 - "gcloud replace files": kind: u'compute#operation'

Step #6 - "gcloud replace files": insertTime: u'2019-11-08T11:27:00.776-08:00'

Step #6 - "gcloud replace files": id: 4494473xxxx605035

Step #6 - "gcloud replace files": ...............DEBUG: Operations to inspect: [<Operation

Step #6 - "gcloud replace files": .DEBUG: Sleeping for 3s.

Step #6 - "gcloud replace files": warnings: []>]

Step #6 - "gcloud replace files": user: u'xxx...@cloudbuild.gserviceaccount.com'

Step #6 - "gcloud replace files": targetLink: u'https://www.googleapis.com/compute/v1/projects/pilotxxxxxx'

Step #6 - "gcloud replace files": targetId: 13173xx173

Step #6 - "gcloud replace files": status: StatusValueValuesEnum(RUNNING, 2)

Step #6 - "gcloud replace files": startTime: u'2019-11-08T11:27:00.781-08:00'

Step #6 - "gcloud replace files": selfLink: u'https://www.googleapis.com/compute/v1/projects/pilotxxxxx/global/operations/operation-1573xxx221-59xxxxx413ca5b-5af18ee0-16987600'

Step #6 - "gcloud replace files": progress: 0

Step #6 - "gcloud replace files": operationType: u'compute.projects.setCommonInstanceMetadata'

Step #6 - "gcloud replace files": name: u'operation-157xxx20221-596dxxxxca5b-5af18ee0-16987600'

Step #6 - "gcloud replace files": kind: u'compute#operation'

Step #6 - "gcloud replace files": insertTime: u'2019-11-08T11:27:00.776-08:00'

Step #6 - "gcloud replace files": id: 449447xxxxx05035

Step #6 - "gcloud replace files": ..........DEBUG: Operations to inspect: [<Operation

Step #6 - "gcloud replace files": .DEBUG: Sleeping for 2s.

Step #6 - "gcloud replace files": warnings: []>]

Step #6 - "gcloud replace files": user: u'xxx...@cloudbuild.gserviceaccount.com'

Step #6 - "gcloud replace files": targetLink: u'https://www.googleapis.com/compute/v1/projects/pilotxxxxx'

Step #6 - "gcloud replace files": targetId: 131735xxx173

Step #6 - "gcloud replace files": status: StatusValueValuesEnum(RUNNING, 2)

Step #6 - "gcloud replace files": startTime: u'2019-11-08T11:27:00.781-08:00'

Step #6 - "gcloud replace files": selfLink: u'https://www.googleapis.com/compute/v1/projects/pilotxxxxx/global/operations/operation-1573xxx220221-596dac4xxxca5b-5af18ee0-16xx7600'

Step #6 - "gcloud replace files": progress: 0

Step #6 - "gcloud replace files": operationType: u'compute.projects.setCommonInstanceMetadata'

Step #6 - "gcloud replace files": name: u'operation-157324xxx0221-596dxxx3ca5b-5af18ee0-16987600'

Step #6 - "gcloud replace files": kind: u'compute#operation'

Step #6 - "gcloud replace files": insertTime: u'2019-11-08T11:27:00.776-08:00'

Step #6 - "gcloud replace files": id: 44944738xxxxx8605035

Step #6 - "gcloud replace files": .....DEBUG: Operations to inspect: [<Operation

Step #6 - "gcloud replace files": .DEBUG: Sleeping for 1s.

Step #6 - "gcloud replace files": warnings: []>]

Step #6 - "gcloud replace files": user: u'xxxx...@cloudbuild.gserviceaccount.com'

Step #6 - "gcloud replace files": targetLink: u'https://www.googleapis.com/compute/v1/projects/pilotxxxxx'

Step #6 - "gcloud replace files": targetId: 1317xxxxxx3

Step #6 - "gcloud replace files": status: StatusValueValuesEnum(RUNNING, 2)

Step #6 - "gcloud replace files": startTime: u'2019-11-08T11:27:00.781-08:00'

Step #6 - "gcloud replace files": selfLink: u'https://www.googleapis.com/compute/v1/projects/pilotxxxx/global/operations/operation-15xxx0221-596dac4ss13cssa5b-5af18ee0-16987600'

Step #6 - "gcloud replace files": progress: 0

Step #6 - "gcloud replace files": operationType: u'compute.projects.setCommonInstanceMetadata'

Step #6 - "gcloud replace files": name: u'operation-15732412xxx221-596dsss413ca5b-5af18ee0-16987600'

Step #6 - "gcloud replace files": kind: u'compute#operation'

Step #6 - "gcloud replace files": insertTime: u'2019-11-08T11:27:00.776-08:00'

Step #6 - "gcloud replace files": id: 4dddd6208605035

Step #6 - "gcloud replace files": .....DEBUG: Operations to inspect: [<Operation

Step #6 - "gcloud replace files": Updating project ssh metadata...

Step #6 - "gcloud replace files": DEBUG: Current SSH keys in project: [u'ac

[/builder/home/.ssh/google_compute_known_hosts]: [Errno 2] No such file or directory: '/builder/home/.ssh/google_compute_known_hosts'

Step #6 - "gcloud replace files": +----[SHA256]-----+

Step #6 - "gcloud replace files": | .=Xo |

Step #6 - "gcloud replace files": | *+.o |

Step #6 - "gcloud replace files": | .ooxxx++|

Step #6 - "gcloud replace files": | o.o =++|

Step #6 - "gcloud replace files": | .Sxxo+oo. .=|

Step #6 - "gcloud replace files": | Eo=xx=++ ..|

Step #6 - "gcloud replace files": | ++.+.+= |

Step #6 - "gcloud replace files": | .. .o.=|

Step #6 - "gcloud replace files": | ..oO|

Step #6 - "gcloud replace files": +---[RSA 2048]----+

Step #6 - "gcloud replace files": The key's randomart image is:

Step #6 - "gcloud replace files": SHA256:c5Us/ms163Tkdxxxxxy1xxxxxxxlMDnOJhTGo root@93c4a6cxxxxx

 Step #6 - "gcloud replace files": The key fingerprint is:

Step #6 - "gcloud replace files": Your public key has been saved in /builder/home/.ssh/google_compute_engine.pub.

Step #6 - "gcloud replace files": Your identification has been saved in /builder/home/.ssh/google_compute_engine.

Step #6 - "gcloud replace files": Generating public/private rsa key pair.

Step #6 - "gcloud replace files": DEBUG: Executing command: [u'/usr/bin/ssh-keygen', u'-N', u'', u'-t', u'rsa', u'-f', u'/builder/home/.ssh/google_compute_engine']

Step #6 - "gcloud replace files": DEBUG: Running command [/usr/bin/ssh-keygen -N -t rsa -f /builder/home/.ssh/google_compute_engine].

Step #6 - "gcloud replace files": WARNING: SSH keygen will be executed to generate a key.

Step #6 - "gcloud replace files": WARNING: You do not have an SSH key for gcloud.

Step #6 - "gcloud replace files": WARNING: The private SSH key file for gcloud does not exist.

Step #6 - "gcloud replace files": WARNING: The public SSH key file for gcloud does not exist.

Step #6 - "gcloud replace files": DEBUG: Running [gcloud.compute.scp] with arguments: [--project: "pilotxxxx", --quiet: "True", --verbosity: "debug", --zone: "us-centralx-x", [[USER@]INSTANCE:]DEST: "xxxx-8cexxxxx747:/tmp", [[USER@]INSTANCE:]SRC:1: "[u'xxxx_properties/common/x.properties']"]

Step #6 - "gcloud replace files": #######################################################################

Step #6 - "gcloud replace files":

Step #6 - "gcloud replace files": $ gcloud config set account `ACCOUNT`

Step #6 - "gcloud replace files": To set the active account, run:

Step #6 - "gcloud replace files":

Step #6 - "gcloud replace files": * xxxxx...@cloudbuild.gserviceaccount.com

Step #6 - "gcloud replace files": ACTIVE ACCOUNT

Step #6 - "gcloud replace files": Credentialed Accounts

Step #6 - "gcloud replace files": #######################################################################

Any help would be good. Thanks.

[Larbi (Google Cloud Support)]

Hello ANTONIO,

Just to understand your situation, is this the first time trying to copy some from your workstation to compute engine [instance]?

Are you able to ssh from your workstation to the compute engine [instance]? by running the gcloud CLI, see documentation [1] .

Are you sure that the ssh key is added to the instance or exist in the metadata ?

Do you have enough disk space available on the instance? 

Is your firewall rule allow the port 22 ?

Links:

[1]https://cloud.google.com/sdk/gcloud/reference/compute/ssh