Upcoming SSH Host Key Rotation for hg.mozilla.org

[Gregory Szorc]

This message serves as a notice that the *SSH host keys* for
hg.mozilla.org will be rotated in the next ~24 hours.

When connecting to hg.mozilla.org over SSH, your SSH client should warn
that host keys have changed and refuse to connect until
accepting/trusting the new host key. After 1st host key verification
failure:

1) `ssh-keygen -R hg.mozilla.org` to remove the old host key
2) `ssh hg.mozilla.org` and verify the fingerprint of the new key
matches one of the following:

256 SHA256:7MBAdqLe8+aSYkv+5/2LUUxd+WdgYcVSV+ZQVEKA7jA hg.mozilla.org
(ED25519)
256 SHA1:Ft++OU96cvaREKNFCJ6AiuCpGac hg.mozilla.org (ED25519)
256 MD5:96:eb:3b:78:f5:ca:19:e2:0c:a0:95:ea:04:28:7d:26 hg.mozilla.org
(ED25519)

4096 SHA256:RX2OK8A1KNWdxyu6ibIPeEGLBzc5vyQW/wd7RKjBehc hg.mozilla.org (RSA)
4096 SHA1:p2MGe4wSw8ZnQ5J9ShBk/6VA+Co hg.mozilla.org (RSA)
4096 MD5:1c:f9:cf:76:de:b8:46:d6:5a:a3:00:8d:3b:0c:53:77 hg.mozilla.org
(RSA)

Q: What host key types were changed? We dropped the DSA host key and
added a ED25519 host key. The length of the RSA key has been increased
from 2048 to 4096 bits.

Q: Does this impact connections to https://hg.mozilla.org/? No. The x509
certificate to the https:// endpoint is remaining unchanged at this time.

Q: Why is this being done? We are modernizing the server infrastructure
of hg.mozilla.org. As part of this, we're bringing the hosts in
compliance with Mozilla's SSH security guidelines
(https://wiki.mozilla.org/Security/Guidelines/OpenSSH).

[Henri Sivonen]

On Fri, Apr 1, 2016 at 12:39 AM, Gregory Szorc <gsz...@mozilla.com> wrote:
> This message serves as a notice that the *SSH host keys* for
> hg.mozilla.org will be rotated in the next ~24 hours.

Are the current Mozilla SSH host keys always published on some https
page (that's not a wiki editable by anyone) on mozilla.org?

--
Henri Sivonen
hsiv...@hsivonen.fi
https://hsivonen.fi/

[Gregory Szorc]

This change was just made (we delayed because we didn't want to take
extra risks on a Friday afternoon).

A GPG signed document detailing the current keys is available at
https://hg.mozilla.org/hgcustom/version-control-tools/raw-file/tip/docs/vcs-server-info.asc

On 3/31/16 2:39 PM, Gregory Szorc wrote:
> This message serves as a notice that the *SSH host keys* for
> hg.mozilla.org will be rotated in the next ~24 hours.
>

[Gregory Szorc]

We also changed the SSH server config to only support the "modern" set of
ciphers, MACs, algorithms, etc from
https://wiki.mozilla.org/Security/Guidelines/OpenSSH#Modern. If you are
running an old SSH client, it may not be able to connect.

If you encounter problems connecting, complain in #vcs with a link to
pastebinned `ssh -v` output so we can see what your client supports so we
may consider adding legacy support on the server as a stop-gap. But
upgrading your SSH client to something that supports modern crypto is
highly preferred. More and more Mozilla systems will be adopting these
"modern" SSH server settings. So you'll have to upgrade sometime.

[Kendall Libby]

As part of this, SSH DSA keys were no longer being accepted by the server.
However, there is no easy way for most non-MoCo contributors to change
their SSH keys, whereas MoCo users and communitiy members with LDAP
accounts can (and should!) use login.mozilla.com to update their keys. So a
bunch of folks have been locked out with little recourse.

I've re-enabled the use of DSA keys on hg.mozilla.org, and we will follow
up in the next day or two with a plan for final retirement of DSA key
access. We're hoping to enable the DSA key blocking again in a week or two,
so if you can self-serve please do so.

K.

> _______________________________________________
> dev-version-control mailing list
> dev-versi...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-version-control
>

[Philip Chee]

On 04/04/2016 23:52, Gregory Szorc wrote:
> We also changed the SSH server config to only support the "modern" set of
> ciphers, MACs, algorithms, etc from
> https://wiki.mozilla.org/Security/Guidelines/OpenSSH#Modern. If you are
> running an old SSH client, it may not be able to connect.
>
> If you encounter problems connecting, complain in #vcs with a link to
> pastebinned `ssh -v` output so we can see what your client supports so we
> may consider adding legacy support on the server as a stop-gap. But
> upgrading your SSH client to something that supports modern crypto is
> highly preferred. More and more Mozilla systems will be adopting these
> "modern" SSH server settings. So you'll have to upgrade sometime.

I'm using TortoiseHg whichh uses PuTTY and PLINK internally. I've
deleted the mozilla host key and accepted the new one.

Now I can't push to comm-central via TortoiseHg. I can't push directly
via hg.exe either. Putty error message is uninformative.

Phil


--
Philip Chee <phi...@aleytys.pc.my>, <phili...@gmail.com>
http://flashblock.mozdev.org/ http://xsidebar.mozdev.org
Guard us from the she-wolf and the wolf, and guard us from the thief,
oh Night, and so be good for us to pass.

[Onno Ekker]

Op 5-4-2016 om 3:09 schreef Philip Chee:

> On 04/04/2016 23:52, Gregory Szorc wrote:
>> We also changed the SSH server config to only support the "modern" set of
>> ciphers, MACs, algorithms, etc from
>> https://wiki.mozilla.org/Security/Guidelines/OpenSSH#Modern. If you are
>> running an old SSH client, it may not be able to connect.
>>
>> If you encounter problems connecting, complain in #vcs with a link to
>> pastebinned `ssh -v` output so we can see what your client supports so we
>> may consider adding legacy support on the server as a stop-gap. But
>> upgrading your SSH client to something that supports modern crypto is
>> highly preferred. More and more Mozilla systems will be adopting these
>> "modern" SSH server settings. So you'll have to upgrade sometime.
>
> I'm using TortoiseHg whichh uses PuTTY and PLINK internally. I've
> deleted the mozilla host key and accepted the new one.
>
> Now I can't push to comm-central via TortoiseHg. I can't push directly
> via hg.exe either. Putty error message is uninformative.
>
> Phil
>
>

I had my old SSH-key not only stored as hg.mozilla.org, but also as
numeric ip-address, which prevented the new one from working correctly.
Maybe something similar also happens for you?
Check your ~/.ssh/known_hosts file.

Onno

[Philip Chee]

On 05/04/2016 14:23, Onno Ekker wrote:
> Op 5-4-2016 om 3:09 schreef Philip Chee:

>> I'm using TortoiseHg whichh uses PuTTY and PLINK internally. I've
>> deleted the mozilla host key and accepted the new one.
>>
>> Now I can't push to comm-central via TortoiseHg. I can't push directly
>> via hg.exe either. Putty error message is uninformative.
>>
>> Phil

> I had my old SSH-key not only stored as hg.mozilla.org, but also as
> numeric ip-address, which prevented the new one from working correctly.
> Maybe something similar also happens for you?
> Check your ~/.ssh/known_hosts file.
>
> Onno

Will do thanks!

[Philip Chee]

On 05/04/2016 09:09, Philip Chee wrote:
> On 04/04/2016 23:52, Gregory Szorc wrote:
>> We also changed the SSH server config to only support the "modern" set of
>> ciphers, MACs, algorithms, etc from
>> https://wiki.mozilla.org/Security/Guidelines/OpenSSH#Modern. If you are
>> running an old SSH client, it may not be able to connect.
>>
>> If you encounter problems connecting, complain in #vcs with a link to
>> pastebinned `ssh -v` output so we can see what your client supports so we
>> may consider adding legacy support on the server as a stop-gap. But
>> upgrading your SSH client to something that supports modern crypto is
>> highly preferred. More and more Mozilla systems will be adopting these
>> "modern" SSH server settings. So you'll have to upgrade sometime.
>
> I'm using TortoiseHg whichh uses PuTTY and PLINK internally. I've
> deleted the mozilla host key and accepted the new one.
>
> Now I can't push to comm-central via TortoiseHg. I can't push directly
> via hg.exe either. Putty error message is uninformative.

TortoiseHg 3.7.2 ships with a modified version of Plink from PuTTY 0.62.
I replaced this with the Plink.exe from PuTTY 0.67 and can now push to
hg.mozilla.org.

I have opened a bug in their issue tracker:
https://bitbucket.org/tortoisehg/thg/issues/4476/tortoiseplink-needs-to-be-updated-to-v067

Meanwhile is there a relevant wiki page on wiki.mo where I can add this
information?

Thanks.

[Philip Chee]

On 04/04/2016 23:52, Gregory Szorc wrote:

> We also changed the SSH server config to only support the "modern" set of
> ciphers, MACs, algorithms, etc from
> https://wiki.mozilla.org/Security/Guidelines/OpenSSH#Modern. If you are
> running an old SSH client, it may not be able to connect.
>
> If you encounter problems connecting, complain in #vcs with a link to
> pastebinned `ssh -v` output so we can see what your client supports so we
> may consider adding legacy support on the server as a stop-gap. But
> upgrading your SSH client to something that supports modern crypto is
> highly preferred. More and more Mozilla systems will be adopting these
> "modern" SSH server settings. So you'll have to upgrade sometime.

TortoiseHg:

https://bitbucket.org/tortoisehg/thg/issues/4234/bundled-tortoiseplink-cannot-connect-to

The bundled version of PLINK from PuTTY is 0.62 which doesn't work with
OpenSSH 6.9p1.

In addition TortoiseSVN 1.9.3 ships with TortoisePlink 0.66 which
apparently doesn't work with current Mozilla SSH server config.

My workaround is to copy PLINK from PuTTY 0.67 into the TortoiseHg
installation directory and rename that to TortoisePlink.exe

TortoiseSVN:

I don't need access to any Mozilla repositories that use Subversion but
anyone is using TortoiseSVN 1.9.3 or earlier might have the same problem.