How to fix SSH after October update broke it?

[Hammer, Erich F]

The October patches on a Server 2019 system appears to have broken OpenSSH (after reboot).

I see that this is known (https://stackoverflow.com/a/79087433), but none of the fixes seems to work for me.

E.G.
=========
PS C:\ProgramData\ssh> ri .\logs\ -Recurse -Force
PS C:\ProgramData\ssh> C:\Temp\OpenSSH-Win64\FixHostFilePermissions.ps1
[*] C:\ProgramData\ssh\sshd_config
looks good

[*] C:\ProgramData\ssh\ssh_host_dsa_key
looks good

[*] C:\ProgramData\ssh\ssh_host_dsa_key.pub
looks good

[*] C:\ProgramData\ssh\ssh_host_ecdsa_key
looks good

[*] C:\ProgramData\ssh\ssh_host_ecdsa_key.pub
looks good

[*] C:\ProgramData\ssh\ssh_host_ed25519_key
looks good

[*] C:\ProgramData\ssh\ssh_host_ed25519_key.pub
looks good

[*] C:\ProgramData\ssh\ssh_host_rsa_key
looks good

[*] C:\ProgramData\ssh\ssh_host_rsa_key.pub
looks good

[*] C:\Users\targetuser\.ssh\authorized_keys
looks good

Done.
==========

And

=========
C:\ProgramData\ssh> icacls .\*
.\logs NT AUTHORITY\SYSTEM:(OI)(CI)(F)
BUILTIN\Administrators:(OI)(CI)(F)

.\sshd_config NT AUTHORITY\SYSTEM:(F)
BUILTIN\Administrators:(F)

.\ssh_host_dsa_key BUILTIN\Administrators:(F)
NT AUTHORITY\SYSTEM:(F)

.\ssh_host_dsa_key.pub BUILTIN\Administrators:(F)
NT AUTHORITY\SYSTEM:(F)

.\ssh_host_ecdsa_key BUILTIN\Administrators:(F)
NT AUTHORITY\SYSTEM:(F)

.\ssh_host_ecdsa_key.pub BUILTIN\Administrators:(F)
NT AUTHORITY\SYSTEM:(F)

.\ssh_host_ed25519_key BUILTIN\Administrators:(F)
NT AUTHORITY\SYSTEM:(F)

.\ssh_host_ed25519_key.pub BUILTIN\Administrators:(F)
NT AUTHORITY\SYSTEM:(F)

.\ssh_host_rsa_key BUILTIN\Administrators:(F)
NT AUTHORITY\SYSTEM:(F)

.\ssh_host_rsa_key.pub BUILTIN\Administrators:(F)
NT AUTHORITY\SYSTEM:(F)
===========

The sshd.exe file (well, all the ssh*.exe files in C:\Windows\System32\OpenSSH) has a modification date of 10/28/2024 and version of 9.5.2.1.

Very strangely, if I run cmd as SYSTEM, I cannot even see that the c:\Windows\System32\OpenSSH directory even exists.

Anyone else dealing with this and/or have resolved it have further suggestions?

Thanks,
Erich




--
Erich Hammer Head of Library Systems
er...@albany.edu University Libraries
518-442-3891 University @ Albany

"Nearly all men can stand adversity, but if you want to test
a man's character, give him power." -- Abraham Lincoln

[James Iversen]

Can you manually assign full control to dir for SYSTEM?
Sent from my iPhone

> On Nov 7, 2024, at 1:21 PM, Hammer, Erich F <er...@albany.edu> wrote:
>
> The October patches on a Server 2019 system appears to have broken OpenSSH (after reboot).

> --
> You received this message because you are subscribed to the Google Groups "ntsysadmin" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ntsysadmin+...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/ntsysadmin/DM8PR04MB78936CC211F2048336B7FC8FCC5C2%40DM8PR04MB7893.namprd04.prod.outlook.com.

[Hammer, Erich F]

SYSTEM has full control of C:\ProgramData\ssh:

When I attempt to give SYSTEM full control of C:\Windows\System32\openssh as an administrator I get access denied. Permissions are as follows:

=====
C:\WINDOWS\system32>icacls OpenSSH
OpenSSH NT SERVICE\TrustedInstaller:(F)
NT SERVICE\TrustedInstaller:(CI)(IO)(F)
NT AUTHORITY\SYSTEM:(M)
NT AUTHORITY\SYSTEM:(OI)(CI)(IO)(F)
BUILTIN\Administrators:(M)
BUILTIN\Administrators:(OI)(CI)(IO)(F)
BUILTIN\Users:(RX)
BUILTIN\Users:(OI)(CI)(IO)(GR,GE)
CREATOR OWNER:(OI)(CI)(IO)(F)
APPLICATION PACKAGE AUTHORITY\ALL APPLICATION PACKAGES:(RX)
APPLICATION PACKAGE AUTHORITY\ALL APPLICATION PACKAGES:(OI)(CI)(IO)(GR,GE)
APPLICATION PACKAGE AUTHORITY\ALL RESTRICTED APPLICATION PACKAGES:(RX)
APPLICATION PACKAGE AUTHORITY\ALL RESTRICTED APPLICATION PACKAGES:(OI)(CI)(IO)(GR,GE)
======

Thanks,
Erich




On Thursday, November 7, 2024 at 13:43, James Iversen eloquently inscribed:

[James Iversen]

I know you said as “an” administrator. Do you have the ability to logon as administrator acct?
In the past, sometimes things just wouldn’t work unless you used ‘that’ account.
Sent from my iPhone

> On Nov 7, 2024, at 2:00 PM, Hammer, Erich F <er...@albany.edu> wrote:
>
> SYSTEM has full control of C:\ProgramData\ssh:

> To view this discussion visit https://groups.google.com/d/msgid/ntsysadmin/DM8PR04MB789393F43DF79C5B0FBFBA49CC5C2%40DM8PR04MB7893.namprd04.prod.outlook.com.

[Fut Dey]

What I did for our Win2019 servers was upgrading SSH to the same version as the clients by installing v9.5 on the servers.

https://github.com/PowerShell/Win32-OpenSSH/releases/tag/v9.5.0.0p1-Beta

If you look at the system32\OpenSSH\ssh.exe on the client machine, the file version is v9.5.2.1 and the file version on the Win2019 server is v8.1.

Make sure you backup the existing ssh configs.

HTH,

Fut

---

From: ntsys...@googlegroups.com <ntsys...@googlegroups.com> on behalf of Hammer, Erich F <er...@albany.edu>
Sent: Thursday, November 7, 2024 10:21 AM
To: ntsys...@googlegroups.com <ntsys...@googlegroups.com>
Subject: [ntsysadmin] How to fix SSH after October update broke it?

 

[Hammer, Erich F]

The problem isn't between client and server. The server service won't start with an Error 1067.

The server is the one running 9.5.2.1. The client in this case is Linux.

Thanks.

On Thursday, November 7, 2024 at 15:39, Fut Dey eloquently inscribed:

> What I did for our Win2019 servers was upgrading SSH to the same version
> as the clients by installing v9.5 on the servers.
> https://github.com/PowerShell/Win32-OpenSSH/releases/tag/v9.5.0.0p1- Bet
> a
>
> If you look at the system32\OpenSSH\ssh.exe on the client machine, the
> file version is v9.5.2.1 and the file version on the Win2019 server is
> v8.1.
>
> Make sure you backup the existing ssh configs.
>
> HTH,
> Fut
> ________________________________
>
> From: ntsys...@googlegroups.com <ntsys...@googlegroups.com> on
> behalf of Hammer, Erich F <er...@albany.edu>
> Sent: Thursday, November 7, 2024 10:21 AM
> To: ntsys...@googlegroups.com <ntsys...@googlegroups.com>
> Subject: [ntsysadmin] How to fix SSH after October update broke it?
>
> The October patches on a Server 2019 system appears to have broken
> OpenSSH (after reboot).
>
> I see that this is known (https://stackoverflow.com/a/79087433), but
> none of the fixes seems to work for me.
>
> E.G. ========= PS C:\ProgramData\ssh> ri .\logs\ -Recurse -Force PS

> C:\ProgramData\ssh> C:\Temp\OpenSSH- Win64\FixHostFilePermissions.ps1

[Hammer, Erich F]

Oof! I had to set a new password and enable it (temporarily), but in the end, I still got the same "Access is Denied".



On Thursday, November 7, 2024 at 14:53, James Iversen eloquently inscribed:

[Fut Dey]

FWIW, sshd_config's permission is slightly different than yours:

C:\ProgramData\ssh>icacls .\*

.\logs NT AUTHORITY\SYSTEM:(OI)(CI)(F)

       BUILTIN\Administrators:(OI)(CI)(F)

.\sshd.pid NT AUTHORITY\SYSTEM:(F)

           BUILTIN\Administrators:(F)

           NT AUTHORITY\Authenticated Users:(RX)

.\sshd_config NT AUTHORITY\SYSTEM:(F)

              BUILTIN\Administrators:(F)

              NT AUTHORITY\Authenticated Users:(RX)

.\sshd_config_original NT AUTHORITY\SYSTEM:(F)

                       BUILTIN\Administrators:(F)

.\ssh_host_ecdsa_key BUILTIN\Administrators:(F)

                     NT AUTHORITY\SYSTEM:(F)

.\ssh_host_ecdsa_key.pub BUILTIN\Administrators:(F)

                         NT AUTHORITY\SYSTEM:(F)

.\ssh_host_ed25519_key BUILTIN\Administrators:(F)

                       NT AUTHORITY\SYSTEM:(F)

.\ssh_host_ed25519_key.pub BUILTIN\Administrators:(F)

                           NT AUTHORITY\SYSTEM:(F)

.\ssh_host_rsa_key BUILTIN\Administrators:(F)

                   NT AUTHORITY\SYSTEM:(F)

.\ssh_host_rsa_key.pub BUILTIN\Administrators:(F)

                       NT AUTHORITY\SYSTEM:(F)

Fut

---

From: ntsys...@googlegroups.com <ntsys...@googlegroups.com> on behalf of Hammer, Erich F <er...@albany.edu>

Sent: Thursday, November 7, 2024 1:30 PM
To: ntsys...@googlegroups.com <ntsys...@googlegroups.com>
Subject: RE: [ntsysadmin] How to fix SSH after October update broke it?

 

To view this discussion visit https://groups.google.com/d/msgid/ntsysadmin/DM8PR04MB78930937A02B0E3A957C4737CC5C2%40DM8PR04MB7893.namprd04.prod.outlook.com.

[Hammer, Erich F]

I removed the OpenSSH feature (and the directory remained) and re-installed the feature, but still no dice. SYSTEM could still not see that the folder exists (and thus cannot start the service). The SYSTEM account had Full control over the contents, but Modify only rights on the folder itself.

I took ownership of the folder (from TrustedInstaller) and the contents, and gave myself Full rights to the tree. I had to reboot to delete it (something was "using" it), but I blew it completely away after that.

Now, when I add OpenSSH through Optional Features, it installs v7.7.2.1. It won't start with a 1058 error. Recall that v9.5.2.1 was what was there after the October updates. (I don't know what version was there before.) Neither of those versions is available in the repository (https://github.com/PowerShell/Win32-OpenSSH/releases/)

What kind of rabbit hole of disfunction has Microsoft created here?

Any help to get back to a functional state is appreciated.

Thanks,
Erich



On Thursday, November 7, 2024 at 16:30, Erich Hammer eloquently inscribed:

[Wright, John M]

After the reinstall of OpenSSH, how is the ssh agent service set? Auto, Manual or disabled?

--
John Wright
IT Support Specialist

1800 Old Bluegrass Avenue, Louisville, KY 40215
502.708.9953
Please submit IT requests to Hazelwoo...@bluegrass.org
24 Hour Helpline 1.800.928.8000
  
CONFIDENTIALITY NOTICE: This message contains confidential information and is intended only for the individual(s) addressed in the message. If you are not the named addressee, you should not disseminate, distribute, or copy this e-mail. If you are not the intended recipient, you are notified that disclosing, distributing, or copying this e-mail is strictly prohibited.

-----Original Message-----
From: ntsys...@googlegroups.com <ntsys...@googlegroups.com> On Behalf Of Hammer, Erich F
Sent: Friday, November 8, 2024 10:06 AM
To: ntsys...@googlegroups.com
Subject: RE: [ntsysadmin] How to fix SSH after October update broke it?

This message is from an external sender.

>>> ntsysadmin+visit
>>
>> https://groups.google.com/d/msgid/ntsysadmin/DM8PR04MB789393F43DF7
>> 9C5B0FBFBA49CC5C2%40DM8PR04MB7893.namprd04.prod.outlook.com.

>>
>
>


--
You received this message because you are subscribed to the Google Groups "ntsysadmin" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ntsysadmin+...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/ntsysadmin/DM8PR04MB7893B09E22A27B4A5F1A73DBCC5D2%40DM8PR04MB7893.namprd04.prod.outlook.com.
[CAUTION] Do not click on links or open attachments unless you recognize the sender and know the content is safe.
If you believe this is a malicious email, please forward it the local IT team and click the Report Message button in Outlook.

[Philip Elder]

Is the folder path in the %PATH%? Maybe it got removed?

Philip Elder MCTS
Senior Technical Architect
Microsoft High Availability MVP
MPECS Inc.
E-mail: Phili...@mpecsinc.ca
Phone: +1 (780) 458-2028
Web: www.mpecsinc.com
Blog: blog.mpecsinc.com
Twitter: Twitter.com/MPECSInc
Skype: MPECSInc.
 
Please note: Although we may sometimes respond to email, text and phone calls instantly at all hours of the day, our regular business hours are 8:00 AM - 5:00 PM, Monday thru Friday.

-----Original Message-----
From: ntsys...@googlegroups.com <ntsys...@googlegroups.com> On Behalf Of Hammer, Erich F
Sent: Friday, November 8, 2024 08:06
To: ntsys...@googlegroups.com
Subject: RE: [ntsysadmin] How to fix SSH after October update broke it?

>>> ntsysadmin+visit
>>
>> https://groups.google.com/d/msgid/ntsysadmin/DM8PR04MB789393F43DF7
>> 9C5B0FBFBA49CC5C2%40DM8PR04MB7893.namprd04.prod.outlook.com.

>>
>
>


--
You received this message because you are subscribed to the Google Groups "ntsysadmin" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ntsysadmin+...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/ntsysadmin/DM8PR04MB7893B09E22A27B4A5F1A73DBCC5D2%40DM8PR04MB7893.namprd04.prod.outlook.com.

[Hammer, Erich F]

The Service is Manual. It won't start manually or when set to auto.


On Friday, November 8, 2024 at 10:20, John Wright eloquently inscribed:

[Hammer, Erich F]

It is in the %path%, but all the calls are either full path or relative/local, so I don't know why that would matter.



On Friday, November 8, 2024 at 10:55, Philip Elder eloquently inscribed: