Panik Github RSH SSH host key change breaks installations

[Neil Young]

Hi,

This morning I noticed this error while attempting to update a project on my server via `git clone` (I'm using Ansible's ssh-agent-forwarding, which worked fine for months)

<quote>

"Failed to download remote objects and refs:  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s.
Please contact your system administrator.
Add correct host key in /home/ubuntu/.ssh/known_hosts to get rid of this message.
Offending RSA key in /home/ubuntu/.ssh/known_hosts:1
  remove with:
  ssh-keygen -f \"/home/ubuntu/.ssh/known_hosts\" -R \"github.com\"
  RSA host key for github.com has changed and you have requested strict checking.
  Host key verification failed.
  fatal: Could not read from remote repository.
  Please make sure you have the correct access rights
  and the repository exists.

</quote>

The fix didn't solve the problem, just changed the error message:

<quote>

TASK [SOFTWARE - Clone xxx-project (run as non-privileged user)] ********************************************************************************************************************************************************************************
fatal: [server]: FAILED! => {"changed": false, "cmd": ["/usr/bin/git", "fetch", "--tags", "--force", "origin"], "msg": "Failed to download remote objects and refs:  Host key verification failed.\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n"}

</quote>

I then discovered this blog entry which states, that Github updated their RSA SSH key this morning:

https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/

Now, everything they suggest work somehow. But it doesn't fix the problem, that I'm unable to clone my project on the target machine.

Anybody having the same issue and maybe a solution?

[Neil Young]

To be specific: I'm unable to execute this (and maybe many other lines as well) now. I can update the project with manual git command on the target host, but not with Ansible from my machine:

- name: SOFTWARE - Clone xxx-project (run as non-privileged user)

tags: software

git:

repo: "{{ xxx_procect_git_repo_clone_url }}"

dest: xxx-project

force: yes

become: yes

become_user: "{{ non_root_user }}"

[Neil Young]

OK, maybe a solution (workaround?)

I got another error message later, which clarified the failing command:

FAILED! => {"changed": false, "cmd": "/usr/bin/git ls-remote g...@github.com:xxxxxxxxxxx.git -h refs/heads/HEAD", "

So I ssh'ed to the machine and issued exactly this command:

The response lead me to an offending, but not obvious "known_host" entry (which obviously described "github.com"), maybe left by "ssh-agent" (hmm, didn't know that. Shouldn't be left there I guess).

Warning: the ECDSA host key for 'github.com' differs from the key for the IP address '140.82.121.3'

Offending key for IP in /home/ubuntu/.ssh/known_hosts:2

I removed this entry in line 2 and it worked.  So the fix is: Remove everything from your .ssh/known_hosts which looks like remains from ssh-agent. It might trap the github auth.

[Simon Kelly]

You can find out more about why this happened on this Github blog: https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/

Simon Kelly

Senior Director of Server Engineering

Dimagi | www.dimagi.com

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/1e7ac8cd-d4b5-41e0-a314-bf048257dbe7n%40googlegroups.com.

[Neil Young]

Thanks. I quoted this in my original post.

Meanwhile this is the way I found and it made the ssh-agent deployment run on 5+ machines again:

- Remove everything from the target hosts ~/.ssh/known_host

- Follow the suggestions given in the blog entry to add the. new github keys

[Neil Young]

Errata: The old keys describing github in my ~/.ssh/known_hosts have not been left by ssh-agent. Instead it seems to be a "normal behaviour" of git to add the github key again cryptically once a project is cloned locally... Not sure (just because I blamed ssh-agent) 

[Nico Kadel-Garcia]

On Fri, Mar 24, 2023 at 8:37 AM 'Neil Young' via Ansible Project
<ansible...@googlegroups.com> wrote:
>
> OK, maybe a solution (workaround?)
>
> I got another error message later, which clarified the failing command:
>
> FAILED! => {"changed": false, "cmd": "/usr/bin/git ls-remote g...@github.com:xxxxxxxxxxx.git -h refs/heads/HEAD", "
>
> So I ssh'ed to the machine and issued exactly this command:
>
> The response lead me to an offending, but not obvious "known_host" entry (which obviously described "github.com"), maybe left by "ssh-agent" (hmm, didn't know that. Shouldn't be left there I guess).
>
> Warning: the ECDSA host key for 'github.com' differs from the key for the IP address '140.82.121.3'
>
> Offending key for IP in /home/ubuntu/.ssh/known_hosts:2

And *this* is why relying on known_hosts has caused more failures of
working software than prevention of faked host access since SSH was
originally written.

The typical entry to disable it in ~/.ssh/config is:

Host *
UserKnownHostsFile /dev/null
StrictHostKeyChecking no
LogLevel error

[Neil Young]

Sounds legit and works. But isn't "StrictHostKeyChecking=no" dangerous? (To not start a religious war here :))

[Neil Young]

Tested an ansible setup from my server (ssh-agent) w/o having github key in known_hosts _and_ the "default" (empty) ssh config. Doesn't work. I had to add github's key on the host in order to make this work again. 

Not storing github *and* having the mentioned config let me do the setup, so I conclude, that now anybody could mock to be github to make me ssh connect to him. I would say this is a bit more scary as to have to replace an official github key once in a lifetime.

[Nico Kadel-Garcia]

On Sat, Mar 25, 2023 at 2:24 AM 'Neil Young' via Ansible Project
<ansible...@googlegroups.com> wrote:
>
> Sounds legit and works. But isn't "StrictHostKeyChecking=no" dangerous? (To not start a religious war here :))

There is an increased risk. The risk of needing to clean up from reset
host keys is also a significant one, and
tuning and picking which keys are and are without that filter is a
burden. Tools like ansible can, in theory,
provide just such tuning on a server-by-server and SSH-service by
SSH-service basis. But I've several times
encountered git server setups where the admin copied over the Host's
private keys, but not the exposed
git related SSH service's keys because he *did not understand the
distinction*, and it's seriously screwed up
working setups both for the Ansible server and the clients. Manually
insertinig the options into all the SSH
commands eliminates those checks on a case-by-case basis, but frankly,
I have a day job, not the
time to go implant the workaround into every developer's SSH command
line settings.

[Rowe, Walter P. (Fed)]

SSH is NOT hard to manage. You need a well defined management practice. We have a service account on our machines with populated SSH public key. We tightly control access to the private key.

Walter
--
Walter Rowe, Division Chief
Infrastructure Services, OISM
Mobile: 202.355.4123

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.

To view this discussion on the web visit https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fansible-project%2FCAOCN9ryTGixQioeW2%252Badz2vKfzHJoVCnvDgZRZKyEzGJ4j%253DMyw%2540mail.gmail.com&data=05%7C01%7Cwalter.rowe%40nist.gov%7C1264b2416c574c97927d08db2d6a076f%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C638153705643785382%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=EOumR4tipuYsITEDjq8p3KCCzOjdrhpfYSZpL7t1x2Y%3D&reserved=0.

[Nico Kadel-Garcia]

On Mon, Mar 27, 2023 at 7:34 AM 'Rowe, Walter P. (Fed)' via Ansible
Project <ansible...@googlegroups.com> wrote:
>
> SSH is NOT hard to manage. You need a well defined management practice. We have a service account on our machines with populated SSH public key. We tightly control access to the private key.
>
> Walter
> --
> Walter Rowe, Division Chief
> Infrastructure Services, OISM
> Mobile: 202.355.4123

That's true. But the return on investment of relying on SSH hostkeys
to be consistent is.... generally not worth the trouble.

Nico Kadel-Garcia