cannot ssh into a specific instance on gcloud via any method

5,766 views
Skip to first unread message

Derek

unread,
Jun 21, 2018, 3:00:21 PM6/21/18
to gce-discussion
I have an issue with a gcloud environment I have inherited at a new job. There is an instance that I am unable to ssh into. It tells me failed [public key] and also sometimes exits with code 255. There is one developer user account that works to ssh into this instance in my group but all other gcloud project users cannot ssh into it regardless of their group permissions. As the DevOps engineer here, I need to resolve this issue for *all* accounts (with appropriate credentials).

The instance seems to fail to get the account credentials for my/other user from the project metadata on every ssh connection attempt by all project accounts newer than the instance.

Let me list the facts:
Instance has a public IP, TCP port 22 open in a firewall rule, netcat [IP] 22 returns openssh 2.0 etc (connectivity = good)
Instance has the default project service account, and all other gcloud related administration functions work with the instance (start/stop/describe/snapshot/logging/etc.).
Serial port logging works and shows when I try to connect it gives errors. (see below)
gcloud compute ssh --ssh-flag="-vvv" gives me verbose output, as does gcloud compute ssh --verbosity=debug (see below)
Instance tries to inherit the project metadata for my username and pubic key file, but fails to do so. (not checked to block ssh keys in in the instance config)
Instance cannot also be ssh'ed into via bastion host  inside the VPC network (regardless of the NAT interface on the instance, this also fails).
Instance cannot be ssh'd into via the gcloud console, or cloud shell.
I have re-created new keys for my account on gcloud and re- auth'ed multiple times in trying to resolve, and these also do not work (but they all worked for all other project instances).
Manually adding my key to the instance metadata does not work.
--force-key-file-overwrite does not fix it.

Instance was created in late 2016, running untouched since late 2017 and likely has out of date gcloud components and other out of date software packages on it.

The instance can be cloned and I can ssh into the clone no problem. But I want to 'fix' the issue of the existing instance, if possible...

I have redacted the IP's, keys, project names, user names, machine names and other private info below - maybe someone can see what the issue is and offer guidance..

Thanks!

Derek

Supporting information from logs and verbose output:

gcloud compute ssh standard response:

user@machine ~ $ gcloud compute ssh broken-instance
Permission denied (publickey).
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].

error when ssh into a bastion host with --ssh-flags="-A" and then try and ssh from there:

user@working-instance:~$ ssh broken-instance
Permission denied (publickey).


Serial log output when i try and log in (seems to have an issue pulling the cuers from the project metadata - see red text):

Getting user by name: redactedusername.
Triggering refresh due to missing user redactedusername.
Refreshing users and groups.
Fetching users and groups.
Failed refresh: googleapi: got HTTP response code 404 with body: Not Found.
Request failed: unable to find user with name "redactedusername".
Request completed.
Invalid user redactedusername from [ip]
input_userauth_request: invalid user redactedusername  [preauth]
Connection closed by [ip] [preauth]

gcloud compute ssh --ssh-flag="-vvv" output:

redactedusername@working-instance:~$ gcloud compute ssh --ssh-flag="-vvv" broken-instance
OpenSSH_7.4p1 Debian-10+deb9u3, OpenSSL 1.0.2l  25 May 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "100.100.100.100" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 100.100.100.100 [100.100.100.100] port 22.
debug1: Connection established.
debug1: identity file /home/redactedusername/.ssh/google_compute_engine type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/redactedusername/.ssh/google_compute_engine-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4p1 Debian-10+deb9u3
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.7p1 Debian-5+deb8u3
debug1: match: OpenSSH_6.7p1 Debian-5+deb8u3 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 100.100.100.100:22 as 'redactedusername'
debug1: using hostkeyalias: compute.1234567890123456789
debug3: hostkeys_foreach: reading file "/home/redactedusername/.ssh/google_compute_known_hosts"
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve255...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nis...@openssh.com,ecdsa-sha2-nis...@openssh.com,ecdsa-sha2-nis...@openssh.com,ssh-ed2551...@openssh.com,ssh-rsa-...@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes12...@openssh.com,aes25...@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: chacha20...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes12...@openssh.com,aes25...@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: umac-...@openssh.com,umac-1...@openssh.com,hmac-sha...@openssh.com,hmac-sha...@openssh.com,hmac-s...@openssh.com,uma...@openssh.com,umac...@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-...@openssh.com,umac-1...@openssh.com,hmac-sha...@openssh.com,hmac-sha...@openssh.com,hmac-s...@openssh.com,uma...@openssh.com,umac...@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zl...@openssh.com,zlib
debug2: compression stoc: none,zl...@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve255...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,aes12...@openssh.com,aes25...@openssh.com,chacha20...@openssh.com
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,aes12...@openssh.com,aes25...@openssh.com,chacha20...@openssh.com
debug2: MACs ctos: umac-...@openssh.com,umac-1...@openssh.com,hmac-sha...@openssh.com,hmac-sha...@openssh.com,hmac-s...@openssh.com,uma...@openssh.com,umac...@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-...@openssh.com,umac-1...@openssh.com,hmac-sha...@openssh.com,hmac-sha...@openssh.com,hmac-s...@openssh.com,uma...@openssh.com,umac...@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zl...@openssh.com
debug2: compression stoc: none,zl...@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve255...@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20...@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20...@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:ReDaCtEdSeRvErH0StK3YtH4Ty0uCaNn0tSeE<S0RrY
debug1: using hostkeyalias: compute.1234567890123456789
debug3: hostkeys_foreach: reading file "/home/redactedusername/.ssh/google_compute_known_hosts"
Warning: Permanently added 'compute.1234567890123456789' (ECDSA) to the list of known hosts.
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: /home/redactedusername/.ssh/google_compute_engine (0x123456789ea01), explicit
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/redactedusername/.ssh/google_compute_engine
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].

gcloud compute ssh --verbosity=debug output:
 gcloud compute ssh "--verbosity=debug" broken-instance
DEBUG: Running [gcloud.compute.ssh] with arguments: [--verbosity: "debug", [USER@]INSTANCE: "broken-instance"]
DEBUG: Current SSH keys in project: [u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redact...@domain.ext","expireOn":"2018-06-20T15:34:59+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redact...@domain.ext","expireOn":"2018-06-20T15:34:57+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redact...@domain.ext","expireOn":"2018-06-20T15:34:24+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redact...@domain.ext","expireOn":"2018-06-20T15:34:23+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redact...@domain.ext","expireOn":"2018-06-20T15:33:47+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redact...@domain.ext","expireOn":"2018-06-20T15:33:46+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redact...@domain.ext","expireOn":"2018-06-20T15:33:00+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redact...@domain.ext","expireOn":"2018-06-20T15:32:59+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redact...@domain.ext","expireOn":"2018-06-20T15:32:24+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redact...@domain.ext","expireOn":"2018-06-20T15:32:23+0000"}', u'dev:ssh-rsa REDACTEDKEY dev@templaptop', u'redactedusername:ssh-rsa REDACTEDKEY redactedusername@deverlt01', u'redacteduser2:ssh-rsa REDACTEDKEY redacteduser2@cs-6000-devshell-vm-0c8dfgc65-d401-48yh-b431-163hdbfhdbfk0', u'redactedusername:ssh-rsa REDACTEDKEY redactedusername@bastionhost2', u'root:ssh-rsa REDACTEDKEY root@bastionhost2', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redact...@domain.ext","expireOn":"2018-06-20T15:34:42+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redact...@domain.ext","expireOn":"2018-06-20T15:34:40+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redact...@domain.ext","expireOn":"2018-06-20T15:34:05+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redact...@domain.ext","expireOn":"2018-06-20T15:34:04+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redact...@domain.ext","expireOn":"2018-06-20T15:33:18+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redact...@domain.ext","expireOn":"2018-06-20T15:33:17+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redact...@domain.ext","expireOn":"2018-06-20T15:32:43+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redact...@domain.ext","expireOn":"2018-06-20T15:32:42+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redact...@domain.ext","expireOn":"2018-06-20T15:32:09+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redact...@domain.ext","expireOn":"2018-06-20T15:32:08+0000"}', u'formerdevop:ssh-rsa REDACTEDKEY forme...@schmuckss-Computer.local', u'redacteduser4:ssh-rsa REDACTEDKEY redact...@redacteduser4-this-MBP-computer.local', u'redacteduser3:ssh-rsa REDACTEDKEYREDACTEDKEY redact...@Dudes-Computer.local', u'redacteduser3:ssh-rsa REDACTEDKEY redact...@Dudes-Computer.local', u'workinguseracct:ssh-rsa REDACTEDKEYTHATWORKS workinguseracct@workinguseracctdev', u'dev:ssh-rsa REDACTEDKEY d...@schmuckss-iMac.local', u'gke-g93w8tvgn384rg8v734h:ssh-rsa REDACTEDKEY gke-789nhth34980v3488n34@gke-g93w8tvgn384rg8v734h', u'root:ssh-rsa REDACTEDKEY forme...@schmuckss-Computer.local', u'gke-hw8y4v5thow8n43htv87:ssh-rsa REDACTEDKEY, u'schmuck_2_whomayhavebrokethis_b4_me:ssh-rsa REDACTEDKEY forme...@schmuckss-Computer.local', u'mongodb:ssh-rsa REDACTEDKEY forme...@schmuckss-Computer.local', u'workinguseracct:ssh-rsa REDACTEDKEYTHATWORKS workinguseracct@workinguseracctdev', u'workinguseracct:ssh-rsa REDACTEDKEYTHATWORKS workinguseracct@workinguseracctdev', u'formerdevop:ssh-rsa REDACTEDKEY forme...@schmuckss-Computer.local', u'root:ssh-rsa REDACTEDKEY root@bastionhost', u'coworkeralsobroken:ssh-rsa REDACTEDKEY redacteduser4@usersmachine', u'dudewhosucked:ssh-rsa REDACTEDKEY schmuck@brokethisb4me', u'root:ssh-rsa REDACTEDKEY redact...@redacteduser4-this-MBP-computer.local', u'coworkeralsobroken:ssh-rsa REDACTEDKEY redacteduser4@usersmachine']
DEBUG: Running command [/usr/bin/ssh -t -i /home/redactedusername/.ssh/google_compute_engine -o CheckHostIP=no -o HostKeyAlias=compute.1234567890123456789 -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/home/redactedusername/.ssh/google_compute_known_hosts redacted...@100.100.100.100].
DEBUG: Executing command: [u'/usr/bin/ssh', u'-t', u'-i', u'/home/redactedusername/.ssh/google_compute_engine', u'-o', u'CheckHostIP=no', u'-o', u'HostKeyAlias=compute.1234567890123456789', u'-o', u'IdentitiesOnly=yes', u'-o', u'StrictHostKeyChecking=no', u'-o', u'UserKnownHostsFile=/home/redactedusername/.ssh/google_compute_known_hosts', u'redacted...@100.100.100.100']
Permission denied (publickey).
DEBUG: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
Traceback (most recent call last):
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 844, in Execute
    resources = calliope_command.Run(cli=self, args=args)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 756, in Run
    resources = command_instance.Run(args)
  File "/usr/lib/google-cloud-sdk/lib/surface/compute/ssh.py", line 190, in Run
    return_code = cmd.Run(ssh_helper.env, force_connect=True)
  File "/usr/lib/google-cloud-sdk/lib/googlecloudsdk/command_lib/util/ssh/ssh.py", line 946, in Run
    raise CommandError(args[0], return_code=status)
CommandError: [/usr/bin/ssh] exited with return code [255].
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].

Google cloud console ssh behavior (gcloud compute instance web page based ssh link as project owner level acct):
(loops/never connects)

Connecting...
Transferring SSH keys to the VM.

Connecting...
Establishing a connection to SSH server.
You can drastically improve your key transfer times by migrating to OS Login.

Connecting...
Could not connect, retrying (2/3)...
The VM serial console output may provide details to aid
in troubleshooting connection problems. See our help
document for other possible causes of this issue.
You can drastically improve your key transfer times by migrating to OS Login.

Larbi (Google Cloud Support)

unread,
Jun 22, 2018, 3:59:34 PM6/22/18
to gce-discussion
Hello Derek,

Check first if you still have available disk space on this VMs.

Try to :

(from web dashboard):

  1. Edit the instance

  2. go to the 'ssh keys' section, and add ssh key from local machine '~/.ssh/id_rsa.pub'. usually just copy and paste the contents of the file to the web interface. 

  3. The ssh key will have 'user@host' on the end, edit this to just have the username you require, leave off the @host portion.

  4. save (you may need to restart also, but try without first).

This will allow you to ssh. If you need to use the gcloud tool then copy the contents of the 'google_compute_engine.pub' above instead, then you can use the gcloud compute commands.


Derek

unread,
Jul 13, 2018, 11:57:27 AM7/13/18
to gce-discussion
Labri,

Thanks for the reply, and sorry for the delay for my response.

1. The instance has plenty of free space between 50% and 94% on all logical volumes according to df
2. I added my ssh keys with the username only as described from both id_rsa.pub AND google_compute_engine.pub. 

However the problem persists. I have not yet rebooted the instance yet but will soon and reply back if there is any change.

Since the instance has a public IP, I tried connecting directly from my linux workstation and also through another instance on the same VPC subnet via it's private address. Both failed with 255.
Here are some logs and connection data for you to look at:
Key:
instance = the instance with the ssh issue of this post
instance2 = a 2nd instance on the same VPC subnet that ssh works correctly
myusername = the linux userid I have that also is what is on the project metadata and works on all other 40+ instances.
myhost = my local linux workstation machine name
IP's and DNS addresses are modified ficticious replacements also for security

Connected from outside using [ gcloud compute ssh instance ] where I redacted/modified user and IP info from the serial log shown below:

Jul 13 14:53:33 instance gcua[374]: Accepted connection.
Jul 13 14:53:33 instance gcua[374]: Getting user by name: myusername.
Jul 13 14:53:33 instance gcua[374]: Triggering refresh due to missing user myusername.
Jul 13 14:53:33 instance gcua[374]: Refreshing users and groups.
Jul 13 14:53:33 instance gcua[374]: Fetching users and groups.
Jul 13 14:53:33 instance gcua[374]: Failed refresh: googleapi: got HTTP response code 404 with body: Not Found.
Jul 13 14:53:33 instance gcua[374]: Request failed: unable to find user with name "myusername".
Jul 13 14:53:33 instance gcua[374]: Request completed.
Jul 13 14:53:33 instance sshd[23339]: Invalid user myusername from 87.65.43.21
Jul 13 14:53:33 instance sshd[23339]: input_userauth_request: invalid user myusername [preauth]
Jul 13 14:53:33 instance sshd[23339]: Connection closed by 87.65.43.21 [preauth]

The above gave this feedback on my linux console:
myusername@myhost ~/.ssh $ gcloud compute ssh instance
Permission denied (publickey).
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
myusername@myhost ~/.ssh $ 

...

This happens constantly in the serial logs and probably is important to the issue:

Jul 13 15:00:57 instance gcua[374]: Refreshing users and groups.
Jul 13 15:00:57 instance gcua[374]: Fetching users and groups.
Jul 13 15:00:57 instance gcua[374]: Failed refresh: googleapi: got HTTP response code 404 with body: Not Found.

...

When I tried to connect internally from another host we'll call instance2 on the same VPC subnet. I connected to instance2 via [ gcloud compute ssh --ssh-flag="-A" instance2 ] and then from there using just [ ssh instance ]:

Jul 13 15:01:24 instance gcua[374]: Accepted connection.
Jul 13 15:01:24 instance gcua[374]: Getting user by name: myusername.
Jul 13 15:01:24 instance gcua[374]: Triggering refresh due to missing user myusername.
Jul 13 15:01:24 instance gcua[374]: Refreshing users and groups.
Jul 13 15:01:24 instance gcua[374]: Fetching users and groups.
Jul 13 15:01:24 instance gcua[374]: Failed refresh: googleapi: got HTTP response code 404 with body: Not Found.
Jul 13 15:01:24 instance gcua[374]: Request failed: unable to find user with name "myusername".
Jul 13 15:01:24 instance gcua[374]: Request completed.
Jul 13 15:01:24 instance sshd[23360]: Invalid user myusername from 10.20.30.40
Jul 13 15:01:24 instance sshd[23360]: input_userauth_request: invalid user myusername [preauth]
Jul 13 15:01:24 instance sshd[23360]: Connection closed by 10.20.30.40 [preauth]

above on my ssh session resulted with:

myusername@myhost:~$ gcloud compute ssh --ssh-flag="-A" instance2 
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Jul  3 18:45:22 2018 from my-dns-alias-from-isp
myusername@instance2:~$ ssh instance
Permission denied (publickey).
myusername@instance2:~$ exit
logout
Connection to 104.196.115.238 closed.
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
myusername@myhost ~/.ssh $ 

the logs for the above connection attempt from instance 2 look like this:

Jul 13 15:49:09 instance2 sshd[3116]: Accepted publickey for myusername from 87.65.43.21 port 41800 ssh2: RSA 5d:43:a7:0c:d2:34:58:2f:21:a7:2a:6b:9f:c6:c9:4c
Jul 13 15:49:16 instance2 sshd[3118]: Received disconnect from 87.65.43.21: 11: disconnected by user

The machine has a very old version of the google api [gcloud components update needed] which may contribute.

Any other ideas aside from reboot post manual ssh key add?

Derek

unread,
Jul 13, 2018, 12:22:26 PM7/13/18
to gce-discussion
Sorry for mis-typing your name - Larbi not Labri.

Larbi (Google Cloud Support)

unread,
Jul 13, 2018, 7:18:49 PM7/13/18
to gce-discussion
Hello Deric,

This is a workaround that will help you to fix your issue and you have to follow these steps:


1. Go to the VM instances page in Google Cloud Platform console.
2. Click on the instance for which you want to add a startup script.
3. Click the Edit button at the top of the page.
4. Click on ‘Enable connecting to serial ports’
5. Under Custom metadata, click Add item.
6. Set 'Key' to 'startup-script' and set 'Value' to this script:

#! /bin/bash
useradd -G sudo USERNAME
echo 'USERNAME:PASSWORD' | chpasswd

7. Click Save and then click RESET on the top of the page. You might need to wait for some time for the instance to reboot.
8. Click on 'Connect to serial port' in the page.
9. In the new window, you might need to wait a bit and press on Enter of your keyboard once; then, you should see the login prompt.
10.. Login using the USERNAME and PASSWORD you provided.

Then inside the instance you need to fetch which is not working by Validate the Guest Environment :
 
First: look in your serial console if these line below are listed :
  • Started Google Compute Engine Accounts Daemon
  • Started Google Compute Engine IP Forwarding Daemon
  • Started Google Compute Engine Clock Skew Daemon
  • Started Google Compute Engine Instance Setup
  • Started Google Compute Engine Startup Scripts
  • Started Google Compute Engine Shutdown Scripts
  • Started Google Compute Engine Network Setup
Second: Verify if the package for the guest Environment is installed run the command in your serial output
  • apt list --installed | grep google-compute
It should list the below line :
  • google-compute-engine
  • google-compute-engine-oslogin
  • python-google-compute-engine
  • python3-google-compute-engine
Third:you need to verify if all the services for the guest environment are running by running this command :
  • sudo systemctl list-unit-files | grep google | grep enabled
It should list the below line :
  • google-accounts-daemon.service      enabled
  • google-ip-forwarding-daemon.service enabled
  • google-clock-skew-daemon.service    enabled
  • google-instance-setup.service       enabled
  • google-shutdown-scripts.service     enabled
  • google-startup-scripts.service      enabled
  • google-network-setup.service        enabled
I hope will this process will fix your issue, I will be waiting your feedback.

Derek

unread,
Jul 16, 2018, 2:29:03 PM7/16/18
to gce-discussion
Larbi,

Thanks for the reply. I will get with the team and see when I can schedule a reboot of the instance, and will get back to you once I go through the procedure. It will be this week, just a matter of when.

Derek
debug2: KEX algorithms: curve25519-sha256,curve25519-sha...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384...@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v...@openssh.com,ssh-rsa-cert-v...@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes12...@openssh.com,aes256...@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: chacha20...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes12...@openssh.com,aes256...@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: umac-...@openssh.com,umac-128...@openssh.com,hmac-sha2-256...@openssh.com,hmac-sha2-512...@openssh.com,hmac-sha1-e...@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-...@openssh.com,umac-128...@openssh.com,hmac-sha2-256...@openssh.com,hmac-sha2-512...@openssh.com,hmac-sha1-e...@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1

debug2: compression ctos: none,zl...@openssh.com,zlib
debug2: compression stoc: none,zl...@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve255...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
DEBUG: Current SSH keys in project: [u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:59+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:57+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:24+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:23+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:47+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:46+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:00+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:59+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:24+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:23+0000"}', u'dev:ssh-rsa REDACTEDKEY dev@templaptop', u'redactedusername:ssh-rsa REDACTEDKEY redactedusername@deverlt01', u'redacteduser2:ssh-rsa REDACTEDKEY redacteduser2@cs-6000-devshell-vm-0c8dfgc65-d401-48yh-b431-163hdbfhdbfk0', u'redactedusername:ssh-rsa REDACTEDKEY redactedusername@bastionhost2', u'root:ssh-rsa REDACTEDKEY root@bastionhost2', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:42+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:40+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:05+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:04+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:18+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:17+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:43+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:42+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:09+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:08+0000"}', u'formerdevop:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'redacteduser4:ssh-rsa REDACTEDKEY redacteduser4@redacteduser4-this-MBP-computer.local', u'redacteduser3:ssh-rsa REDACTEDKEYREDACTEDKEY redacteduser3@Dudes-Computer.local', u'redacteduser3:ssh-rsa REDACTEDKEY redacteduser3@Dudes-Computer.local', u'workinguseracct:ssh-rsa REDACTEDKEYTHATWORKS workinguseracct@workinguseracctdev', u'dev:ssh-rsa REDACTEDKEY d...@schmuckss-iMac.local', u'gke-g93w8tvgn384rg8v734h:ssh-rsa REDACTEDKEY gke-789nhth34980v3488n34@gke-g93w8tvgn384rg8v734h', u'root:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'gke-hw8y4v5thow8n43htv87:ssh-rsa REDACTEDKEY, u'schmuck_2_whomayhavebrokethis_b4_me:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'mongodb:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'workinguseracct:ssh-rsa REDACTEDKEYTHATWORKS workinguseracct@workinguseracctdev', u'workinguseracct:ssh-rsa REDACTEDKEYTHATWORKS workinguseracct@workinguseracctdev', u'formerdevop:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'root:ssh-rsa REDACTEDKEY root@bastionhost', u'coworkeralsobroken:ssh-rsa REDACTEDKEY redacteduser4@usersmachine', u'dudewhosucked:ssh-rsa REDACTEDKEY schmuck@brokethisb4me', u'root:ssh-rsa REDACTEDKEY redacteduser4@redacteduser4-this-MBP-computer.local', u'coworkeralsobroken:ssh-rsa REDACTEDKEY redacteduser4@usersmachine']

Derek

unread,
Jul 19, 2018, 4:53:01 PM7/19/18
to gce-discussion
Larbi,

I have attempted the procedure as outlined and it will not log me in on the serial console. After editing the metadata, enabling serial, stop and start the instance, I specify the username and password set in the script, however the instance gives an error that it cannot find the user. The username is dissimilar to any project metadata based account ssh keys

Below is redacted from the gcloud serial console with USERNAME as the username and gcloud-instance as the instance name:

"gcloud-instance login: USERNAME
Jul 19 20:27:40 gcloud-instance gcua[373]: Accepted connection.
Jul 19 20:27:40 gcloud-instance gcua[373]: Getting user by name: USERNAME.
Jul 19 20:27:40 gcloud-instance gcua[373]: Triggering refresh due to missing user USERNAME.
Jul 19 20:27:40 gcloud-instance gcua[373]: Refreshing users and groups.
Jul 19 20:27:40 gcloud-instance gcua[373]: Fetching users and groups.
Jul 19 20:27:40 gcloud-instance gcua[373]: Failed refresh: googleapi: got HTTP response code 404 with body
: Not Found.
Jul 19 20:27:40 gcloud-instance gcua[373]: Request failed: unable to find user with name "USERNAME".
Jul 19 20:27:40 gcloud-instance gcua[373]: Request completed.
Jul 19 20:27:40 gcloud-instance gcua[373]: Accepted connection.
Jul 19 20:27:40 gcloud-instance gcua[373]: Getting user by name: USERNAME.
Jul 19 20:27:40 gcloud-instance gcua[373]: Triggering refresh due to missing user USERNAME.
Jul 19 20:27:40 gcloud-instance gcua[373]: Request failed: unable to find user with name "USERNAME".
Jul 19 20:27:40 gcloud-instance gcua[373]: Request completed.
Password: 
Jul 19 20:27:51 gcloud-instance gcua[373]: Accepted connection.
Jul 19 20:27:51 gcloud-instance gcua[373]: Getting user by name: USERNAME.
Jul 19 20:27:51 gcloud-instance gcua[373]: Triggering refresh due to missing user USERNAME.
Jul 19 20:27:51 gcloud-instance gcua[373]: Refreshing users and groups.
Jul 19 20:27:51 gcloud-instance gcua[373]: Fetching users and groups.
Jul 19 20:27:51 gcloud-instance gcua[373]: Failed refresh: googleapi: got HTTP response code 404 with body
: Not Found.
Jul 19 20:27:51 gcloud-instance gcua[373]: Request failed: unable to find user with name "USERNAME".
Jul 19 20:27:51 gcloud-instance gcua[373]: Request completed.
Jul 19 20:27:55 gcloud-instance gcua[373]: Accepted connection.
Jul 19 20:27:55 gcloud-instance gcua[373]: Getting user by name: USERNAME.
Jul 19 20:27:55 gcloud-instance gcua[373]: Triggering refresh due to missing user USERNAME.
Jul 19 20:27:55 gcloud-instance gcua[373]: Refreshing users and groups.
Jul 19 20:27:55 gcloud-instance gcua[373]: Fetching users and groups.
Login incorrect"

Also attempting to ssh straight to the instance from either my local machine using the public IP or from another instance on the same VPC subnet yielded an access denied [public key].

Do I need to edit/replace the sshd_conf file to enable password based logons as well for this process to work? U know:

file /etc/ssh/sshd.confsshd.conf:

add/un-comment:

PasswordAuthentication yes 

I may try that tomorrow...

Derek

On Thursday, June 21, 2018 at 3:00:21 PM UTC-4, Derek wrote:
debug2: KEX algorithms: curve25519-sha256,curve25519-sha...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384...@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v...@openssh.com,ssh-rsa-cert-v...@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes12...@openssh.com,aes256...@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: chacha20...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes12...@openssh.com,aes256...@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: umac-...@openssh.com,umac-128...@openssh.com,hmac-sha2-256...@openssh.com,hmac-sha2-512...@openssh.com,hmac-sha1-e...@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-...@openssh.com,umac-128...@openssh.com,hmac-sha2-256...@openssh.com,hmac-sha2-512...@openssh.com,hmac-sha1-e...@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zl...@openssh.com,zlib
debug2: compression stoc: none,zl...@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve255...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
DEBUG: Current SSH keys in project: [u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:59+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:57+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:24+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:23+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:47+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:46+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:00+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:59+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:24+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:23+0000"}', u'dev:ssh-rsa REDACTEDKEY dev@templaptop', u'redactedusername:ssh-rsa REDACTEDKEY redactedusername@deverlt01', u'redacteduser2:ssh-rsa REDACTEDKEY redacteduser2@cs-6000-devshell-vm-0c8dfgc65-d401-48yh-b431-163hdbfhdbfk0', u'redactedusername:ssh-rsa REDACTEDKEY redactedusername@bastionhost2', u'root:ssh-rsa REDACTEDKEY root@bastionhost2', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:42+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:40+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:05+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:04+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:18+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:17+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:43+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:42+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:09+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:08+0000"}', u'formerdevop:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'redacteduser4:ssh-rsa REDACTEDKEY redacteduser4@redacteduser4-this-MBP-computer.local', u'redacteduser3:ssh-rsa REDACTEDKEYREDACTEDKEY redacteduser3@Dudes-Computer.local', u'redacteduser3:ssh-rsa REDACTEDKEY redacteduser3@Dudes-Computer.local', u'workinguseracct:ssh-rsa REDACTEDKEYTHATWORKS workinguseracct@workinguseracctdev', u'dev:ssh-rsa REDACTEDKEY d...@schmuckss-iMac.local', u'gke-g93w8tvgn384rg8v734h:ssh-rsa REDACTEDKEY gke-789nhth34980v3488n34@gke-g93w8tvgn384rg8v734h', u'root:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'gke-hw8y4v5thow8n43htv87:ssh-rsa REDACTEDKEY, u'schmuck_2_whomayhavebrokethis_b4_me:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'mongodb:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'workinguseracct:ssh-rsa REDACTEDKEYTHATWORKS workinguseracct@workinguseracctdev', u'workinguseracct:ssh-rsa REDACTEDKEYTHATWORKS workinguseracct@workinguseracctdev', u'formerdevop:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'root:ssh-rsa REDACTEDKEY root@bastionhost', u'coworkeralsobroken:ssh-rsa REDACTEDKEY redacteduser4@usersmachine', u'dudewhosucked:ssh-rsa REDACTEDKEY schmuck@brokethisb4me', u'root:ssh-rsa REDACTEDKEY redacteduser4@redacteduser4-this-MBP-computer.local', u'coworkeralsobroken:ssh-rsa REDACTEDKEY redacteduser4@usersmachine']

Derek

unread,
Jul 19, 2018, 5:01:27 PM7/19/18
to gce-discussion
/etc/ssh/sshd_conf file... sorry typo below.


On Thursday, June 21, 2018 at 3:00:21 PM UTC-4, Derek wrote:
debug2: KEX algorithms: curve25519-sha256,curve25519-sha...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384...@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v...@openssh.com,ssh-rsa-cert-v...@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes12...@openssh.com,aes256...@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: chacha20...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes12...@openssh.com,aes256...@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: umac-...@openssh.com,umac-128...@openssh.com,hmac-sha2-256...@openssh.com,hmac-sha2-512...@openssh.com,hmac-sha1-e...@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-...@openssh.com,umac-128...@openssh.com,hmac-sha2-256...@openssh.com,hmac-sha2-512...@openssh.com,hmac-sha1-e...@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zl...@openssh.com,zlib
debug2: compression stoc: none,zl...@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve255...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
DEBUG: Current SSH keys in project: [u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:59+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:57+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:24+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:23+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:47+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:46+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:00+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:59+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:24+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:23+0000"}', u'dev:ssh-rsa REDACTEDKEY dev@templaptop', u'redactedusername:ssh-rsa REDACTEDKEY redactedusername@deverlt01', u'redacteduser2:ssh-rsa REDACTEDKEY redacteduser2@cs-6000-devshell-vm-0c8dfgc65-d401-48yh-b431-163hdbfhdbfk0', u'redactedusername:ssh-rsa REDACTEDKEY redactedusername@bastionhost2', u'root:ssh-rsa REDACTEDKEY root@bastionhost2', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:42+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:40+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:05+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:04+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:18+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:17+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:43+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:42+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:09+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:08+0000"}', u'formerdevop:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'redacteduser4:ssh-rsa REDACTEDKEY redacteduser4@redacteduser4-this-MBP-computer.local', u'redacteduser3:ssh-rsa REDACTEDKEYREDACTEDKEY redacteduser3@Dudes-Computer.local', u'redacteduser3:ssh-rsa REDACTEDKEY redacteduser3@Dudes-Computer.local', u'workinguseracct:ssh-rsa REDACTEDKEYTHATWORKS workinguseracct@workinguseracctdev', u'dev:ssh-rsa REDACTEDKEY d...@schmuckss-iMac.local', u'gke-g93w8tvgn384rg8v734h:ssh-rsa REDACTEDKEY gke-789nhth34980v3488n34@gke-g93w8tvgn384rg8v734h', u'root:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'gke-hw8y4v5thow8n43htv87:ssh-rsa REDACTEDKEY, u'schmuck_2_whomayhavebrokethis_b4_me:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'mongodb:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'workinguseracct:ssh-rsa REDACTEDKEYTHATWORKS workinguseracct@workinguseracctdev', u'workinguseracct:ssh-rsa REDACTEDKEYTHATWORKS workinguseracct@workinguseracctdev', u'formerdevop:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'root:ssh-rsa REDACTEDKEY root@bastionhost', u'coworkeralsobroken:ssh-rsa REDACTEDKEY redacteduser4@usersmachine', u'dudewhosucked:ssh-rsa REDACTEDKEY schmuck@brokethisb4me', u'root:ssh-rsa REDACTEDKEY redacteduser4@redacteduser4-this-MBP-computer.local', u'coworkeralsobroken:ssh-rsa REDACTEDKEY redacteduser4@usersmachine']

Larbi (Google Cloud Support)

unread,
Jul 20, 2018, 1:34:09 PM7/20/18
to gce-dis...@googlegroups.com
Hello Derek,

In order to interact with instance's serial console to troubleshoot a VM, you will need to have root or a sudoer privilege. You can add the following startup script to your VM and then reboot it to create a sudoer account:

useradd -G sudo USERNAME
echo 'USERNAME:PASSWORD' | chpasswd

Replace USERNAME and PASSWORD with values that you wish.

Derek

unread,
Jul 23, 2018, 1:38:37 PM7/23/18
to gce-discussion
Using this process on a new machine built using the snapshot from the broken one was successful.

When connecting to the serial at startup a pop up window told me: 

"VM guest environment outdated

The VM guest environment is outdated and only supports the depreciated 'sshKeys' metadata item. Please follow the steps here to update."

After SSH login via serial console using the defined startup script user, when I checked the installed and running components for google was not listing all the desired components so I manually installed it per the instructions here: https://cloud.google.com/compute/docs/instances/linux-guest-environment#in_place

I updated the gcloud components from 102 to 209 as well and rebooted. After that I can ssh in successfully.

So the issue is likely that this was an imported instance with an old guest environment installed. It also had 2 broken ssh keys that the google console told me were invalid so I removed them (the ony way I could edit the machine).

We actually have this same problem on the same 'instance' in our production and development environments, so I will also try on the 'production' instance.

Thanks for your help, I will follow up with more information in the next few days.

Derek


On Thursday, June 21, 2018 at 3:00:21 PM UTC-4, Derek wrote:
debug2: KEX algorithms: curve25519-sha256,curve25519-sha...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384...@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v...@openssh.com,ssh-rsa-cert-v...@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes12...@openssh.com,aes256...@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: chacha20...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes12...@openssh.com,aes256...@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: umac-...@openssh.com,umac-128...@openssh.com,hmac-sha2-256...@openssh.com,hmac-sha2-512...@openssh.com,hmac-sha1-e...@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-...@openssh.com,umac-128...@openssh.com,hmac-sha2-256...@openssh.com,hmac-sha2-512...@openssh.com,hmac-sha1-e...@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zl...@openssh.com,zlib
debug2: compression stoc: none,zl...@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve255...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
DEBUG: Current SSH keys in project: [u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:59+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:57+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:24+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:23+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:47+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:46+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:00+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:59+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:24+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:23+0000"}', u'dev:ssh-rsa REDACTEDKEY dev@templaptop', u'redactedusername:ssh-rsa REDACTEDKEY redactedusername@deverlt01', u'redacteduser2:ssh-rsa REDACTEDKEY redacteduser2@cs-6000-devshell-vm-0c8dfgc65-d401-48yh-b431-163hdbfhdbfk0', u'redactedusername:ssh-rsa REDACTEDKEY redactedusername@bastionhost2', u'root:ssh-rsa REDACTEDKEY root@bastionhost2', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:42+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:40+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:05+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:04+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:18+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:17+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:43+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:42+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:09+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:08+0000"}', u'formerdevop:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'redacteduser4:ssh-rsa REDACTEDKEY redacteduser4@redacteduser4-this-MBP-computer.local', u'redacteduser3:ssh-rsa REDACTEDKEYREDACTEDKEY redacteduser3@Dudes-Computer.local', u'redacteduser3:ssh-rsa REDACTEDKEY redacteduser3@Dudes-Computer.local', u'workinguseracct:ssh-rsa REDACTEDKEYTHATWORKS workinguseracct@workinguseracctdev', u'dev:ssh-rsa REDACTEDKEY d...@schmuckss-iMac.local', u'gke-g93w8tvgn384rg8v734h:ssh-rsa REDACTEDKEY gke-789nhth34980v3488n34@gke-g93w8tvgn384rg8v734h', u'root:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'gke-hw8y4v5thow8n43htv87:ssh-rsa REDACTEDKEY, u'schmuck_2_whomayhavebrokethis_b4_me:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'mongodb:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'workinguseracct:ssh-rsa REDACTEDKEYTHATWORKS workinguseracct@workinguseracctdev', u'workinguseracct:ssh-rsa REDACTEDKEYTHATWORKS workinguseracct@workinguseracctdev', u'formerdevop:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'root:ssh-rsa REDACTEDKEY root@bastionhost', u'coworkeralsobroken:ssh-rsa REDACTEDKEY redacteduser4@usersmachine', u'dudewhosucked:ssh-rsa REDACTEDKEY schmuck@brokethisb4me', u'root:ssh-rsa REDACTEDKEY redacteduser4@redacteduser4-this-MBP-computer.local', u'coworkeralsobroken:ssh-rsa REDACTEDKEY redacteduser4@usersmachine']
gcloudenverror.jpg

Derek

unread,
Jul 23, 2018, 3:09:51 PM7/23/18
to gce-discussion
I followed the same steps on the live production instance and was successful.

I noted after this procedure that I get a different fingerprint on the ECDSA key sent by the remote host when connecting via ssh. Is this expected behavior?:

username@userinstance01 ~ $ gcloud compute ssh fixed-instance
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:YADDA_REDACTED
Please contact your system administrator.
Add correct host key in /home/username/.ssh/google_compute_known_hosts to get rid of this message.
Offending ECDSA key in /home/username/.ssh/google_compute_known_hosts:1
  remove with:
  ssh-keygen -f "/home/username/.ssh/google_compute_known_hosts" -R compute.YADDA_REDACTED
Password authentication is disabled to avoid man-in-the-middle attacks.
Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
username@fixed-instance:~$

I was able to remove the stored key and reconnect without the warning, but does this need to be addressed further?

Derek

On Thursday, June 21, 2018 at 3:00:21 PM UTC-4, Derek wrote:
debug2: KEX algorithms: curve25519-sha256,curve25519-sha...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384...@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v...@openssh.com,ssh-rsa-cert-v...@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes12...@openssh.com,aes256...@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: chacha20...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes12...@openssh.com,aes256...@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: umac-...@openssh.com,umac-128...@openssh.com,hmac-sha2-256...@openssh.com,hmac-sha2-512...@openssh.com,hmac-sha1-e...@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-...@openssh.com,umac-128...@openssh.com,hmac-sha2-256...@openssh.com,hmac-sha2-512...@openssh.com,hmac-sha1-e...@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zl...@openssh.com,zlib
debug2: compression stoc: none,zl...@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve255...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
DEBUG: Current SSH keys in project: [u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:59+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:57+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:24+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:23+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:47+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:46+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:00+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:59+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:24+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:23+0000"}', u'dev:ssh-rsa REDACTEDKEY dev@templaptop', u'redactedusername:ssh-rsa REDACTEDKEY redactedusername@deverlt01', u'redacteduser2:ssh-rsa REDACTEDKEY redacteduser2@cs-6000-devshell-vm-0c8dfgc65-d401-48yh-b431-163hdbfhdbfk0', u'redactedusername:ssh-rsa REDACTEDKEY redactedusername@bastionhost2', u'root:ssh-rsa REDACTEDKEY root@bastionhost2', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:42+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:40+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:05+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:04+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:18+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:17+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:43+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:42+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:09+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:08+0000"}', u'formerdevop:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'redacteduser4:ssh-rsa REDACTEDKEY redacteduser4@redacteduser4-this-MBP-computer.local', u'redacteduser3:ssh-rsa REDACTEDKEYREDACTEDKEY redacteduser3@Dudes-Computer.local', u'redacteduser3:ssh-rsa REDACTEDKEY redacteduser3@Dudes-Computer.local', u'workinguseracct:ssh-rsa REDACTEDKEYTHATWORKS workinguseracct@workinguseracctdev', u'dev:ssh-rsa REDACTEDKEY d...@schmuckss-iMac.local', u'gke-g93w8tvgn384rg8v734h:ssh-rsa REDACTEDKEY gke-789nhth34980v3488n34@gke-g93w8tvgn384rg8v734h', u'root:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'gke-hw8y4v5thow8n43htv87:ssh-rsa REDACTEDKEY, u'schmuck_2_whomayhavebrokethis_b4_me:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'mongodb:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'workinguseracct:ssh-rsa REDACTEDKEYTHATWORKS workinguseracct@workinguseracctdev', u'workinguseracct:ssh-rsa REDACTEDKEYTHATWORKS workinguseracct@workinguseracctdev', u'formerdevop:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'root:ssh-rsa REDACTEDKEY root@bastionhost', u'coworkeralsobroken:ssh-rsa REDACTEDKEY redacteduser4@usersmachine', u'dudewhosucked:ssh-rsa REDACTEDKEY schmuck@brokethisb4me', u'root:ssh-rsa REDACTEDKEY redacteduser4@redacteduser4-this-MBP-computer.local', u'coworkeralsobroken:ssh-rsa REDACTEDKEY redacteduser4@usersmachine']

Derek

unread,
Jul 24, 2018, 10:52:33 AM7/24/18
to gce-discussion
Larbi,

I have now corrected both the development and the production instances. 

Thank you for your assistance.

Derek

On Thursday, June 21, 2018 at 3:00:21 PM UTC-4, Derek wrote:
debug2: KEX algorithms: curve25519-sha256,curve25519-sha...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384...@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v...@openssh.com,ssh-rsa-cert-v...@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes12...@openssh.com,aes256...@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: chacha20...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes12...@openssh.com,aes256...@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: umac-...@openssh.com,umac-128...@openssh.com,hmac-sha2-256...@openssh.com,hmac-sha2-512...@openssh.com,hmac-sha1-e...@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-...@openssh.com,umac-128...@openssh.com,hmac-sha2-256...@openssh.com,hmac-sha2-512...@openssh.com,hmac-sha1-e...@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zl...@openssh.com,zlib
debug2: compression stoc: none,zl...@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve255...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
DEBUG: Current SSH keys in project: [u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:59+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:57+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:24+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:23+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:47+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:46+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:00+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:59+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:24+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:23+0000"}', u'dev:ssh-rsa REDACTEDKEY dev@templaptop', u'redactedusername:ssh-rsa REDACTEDKEY redactedusername@deverlt01', u'redacteduser2:ssh-rsa REDACTEDKEY redacteduser2@cs-6000-devshell-vm-0c8dfgc65-d401-48yh-b431-163hdbfhdbfk0', u'redactedusername:ssh-rsa REDACTEDKEY redactedusername@bastionhost2', u'root:ssh-rsa REDACTEDKEY root@bastionhost2', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:42+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:40+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:05+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:34:04+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:18+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:33:17+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:43+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:42+0000"}', u'redacteduser2:ecdsa-sha2-nistp256 REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:09+0000"}', u'redacteduser2:ssh-rsa REDACTEDKEY= google-ssh {"userName":"redacteduser2@domain.ext","expireOn":"2018-06-20T15:32:08+0000"}', u'formerdevop:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'redacteduser4:ssh-rsa REDACTEDKEY redacteduser4@redacteduser4-this-MBP-computer.local', u'redacteduser3:ssh-rsa REDACTEDKEYREDACTEDKEY redacteduser3@Dudes-Computer.local', u'redacteduser3:ssh-rsa REDACTEDKEY redacteduser3@Dudes-Computer.local', u'workinguseracct:ssh-rsa REDACTEDKEYTHATWORKS workinguseracct@workinguseracctdev', u'dev:ssh-rsa REDACTEDKEY d...@schmuckss-iMac.local', u'gke-g93w8tvgn384rg8v734h:ssh-rsa REDACTEDKEY gke-789nhth34980v3488n34@gke-g93w8tvgn384rg8v734h', u'root:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'gke-hw8y4v5thow8n43htv87:ssh-rsa REDACTEDKEY, u'schmuck_2_whomayhavebrokethis_b4_me:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'mongodb:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'workinguseracct:ssh-rsa REDACTEDKEYTHATWORKS workinguseracct@workinguseracctdev', u'workinguseracct:ssh-rsa REDACTEDKEYTHATWORKS workinguseracct@workinguseracctdev', u'formerdevop:ssh-rsa REDACTEDKEY formerdevop@schmuckss-Computer.local', u'root:ssh-rsa REDACTEDKEY root@bastionhost', u'coworkeralsobroken:ssh-rsa REDACTEDKEY redacteduser4@usersmachine', u'dudewhosucked:ssh-rsa REDACTEDKEY schmuck@brokethisb4me', u'root:ssh-rsa REDACTEDKEY redacteduser4@redacteduser4-this-MBP-computer.local', u'coworkeralsobroken:ssh-rsa REDACTEDKEY redacteduser4@usersmachine']

Larbi (Google Cloud Support)

unread,
Jul 24, 2018, 2:43:18 PM7/24/18
to gce-discussion
Hello Derek,

I am happy to hear that you succeeded to fix your SSH connection in your both instances, keep in mind, the steps provided will give you first support to troubleshoot the ssh issue.

Have a nice day 

Danica Damljanovic

unread,
Sep 11, 2018, 6:22:11 AM9/11/18
to gce-discussion
Hi Larbi

I have the same issue as Derek, except that when I add the user following your steps:
useradd -G sudo USERNAME
echo 'USERNAME:PASSWORD' | chpasswd


and I then try to log in I get Login incorrect - I am pretty sure I supply the correct username and password - what could be wrong? Could it be something with my role - do I need any specific role? I am the owner of the project, but maybe I need to do some more role tweaking?

Thanks
Danica

Yui (Google Cloud Support)

unread,
Sep 11, 2018, 12:11:12 PM9/11/18
to gce-discussion


Hello Danica,


I am assuming you are trying to interact with serial console where a sudo user needs to be created:

Activate the “Connect to serial console” button

> Go to VM instances, click on your VM, Edit, and active “enable connecting to serial ports” in the Remote access area and click on save.

Create a username and password.

> Go to Vm instance, click on your Vm again, Edit, and fill up the custom metadata section with:

In key: startup-script

In value:

#!/bin/bash

sudo useradd -G sudo pamela

sudo echo 'pamela:pamela5' | chpasswd

(This is a script that creates a username : pamela and password: pamela5, which you are going to use later. Please use something else for security purposes)


Note that change will only take effect after a complete restart (stop/restart) of the VM.


At this point, I do not suspect a role issue as we would get permission errors instead. Let me know of your results.




Reply all
Reply to author
Forward
0 new messages