v9.8.0.0p1-Preview service continually crashes/restarts

[TokenRing]

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest version
• Search the existing issues.

Steps to reproduce

Install the 9.8.0.0p1 version via MSI
attempt to start it via Services.msc

Expected behavior

service starts

Actual behavior

Service crashes rapidly with event ID 7031, text "The OpenSSH SSH Server service terminated unexpectedly.  It has done this 1280 time(s).  The following corrective action will be taken in 0 milliseconds: Restart the service."

Error details

No response

Environment data

PS C:\WINDOWS\system32> $psversiontable

Name                           Value
----                           -----
PSVersion                      5.1.19041.4894
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.19041.4894
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

Version

9.8.0.0

Visuals

No response

[TokenRing]

I was able to reproduce this both on a Windows 11 work, domain-joined, GPO managed machine and on my personal Win10 machine at home. The included .ps1 files in the OpenSSH program files folder to FixHosts and FixUsers that are mentioned when you search for error 1067 on the web do not help.

[seansaleh]

FYI I found that v9.4.0.0p1 and later is much more strict about file AND folder permissions, and I ran into an issue like this on my fleet of machines. Any machine that had ever had sshd.exe run manually locally (and not exclusively as a service) was failing to launch the service. And there were no logs saying why.
I was able to fix it by running rm C:\ProgramData\ssh\logs, because for me that folder had too broad of permissions. And by deleting the folder it got recreated on the next service start with the correct permissions.

You may run into this if other files or folder in the ssh folder have too broad of permissions. You may be able to fix that by running the script install-ssh.ps1, which goes and checks permissions for a bunch of files and folders.

I threw in more details in #2282

[DATAPOWERELECTRICAL]

Hi all,
I just noticed with OpenSSH-Win64-v9.8.0.0.msi on Windows, i can not ssh into it after install this.
something im unsure of so i reverted back to previous versionv9.5.0.0 and works fine.

[user8446]

@tgauth

Even after fixing permissions by running install-sshd.ps1 in C:\Program Files\OpenSSH (See #2161) it errors with a missing sshd-session.exe:

PS C:\Windows\System32> sshd -ddd
debug2: load_server_config: filename __PROGRAMDATA__\\ssh/sshd_config
debug2: load_server_config: done config len = 2909
debug2: parse_server_config_depth: config __PROGRAMDATA__\\ssh/sshd_config len 2909
debug3: __PROGRAMDATA__\\ssh/sshd_config:12 setting <redacted>
debug3: __PROGRAMDATA__\\ssh/sshd_config:13 setting AddressFamily inet
debug3: __PROGRAMDATA__\\ssh/sshd_config:34 setting RekeyLimit 6G none
debug3: __PROGRAMDATA__\\ssh/sshd_config:37 setting SyslogFacility LOCAL0
debug3: __PROGRAMDATA__\\ssh/sshd_config:38 setting LogLevel INFO
debug3: __PROGRAMDATA__\\ssh/sshd_config:41 setting AuthenticationMethods publickey
debug3: __PROGRAMDATA__\\ssh/sshd_config:42 setting LoginGraceTime 7s
debug3: __PROGRAMDATA__\\ssh/sshd_config:45 setting MaxAuthTries 1
debug3: __PROGRAMDATA__\\ssh/sshd_config:69 setting PasswordAuthentication no
debug3: __PROGRAMDATA__\\ssh/sshd_config:84 setting ClientAliveInterval 15
debug3: __PROGRAMDATA__\\ssh/sshd_config:85 setting ClientAliveCountMax 3
debug3: __PROGRAMDATA__\\ssh/sshd_config:88 setting MaxStartups 1
debug1: sshd version OpenSSH_for_Windows_9.8 Win32-OpenSSH-GitHub, LibreSSL 3.9.2
debug1: private host key #0: ssh-rsa SHA256:<redacted>
debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:<redacted>
debug1: private host key #2: ssh-ed25519 SHA256:<redacted>
debug1: get_passwd: lookup_sid() failed: 1332.
debug1: rexec_argv[1]='-ddd'
c:\\program files\\openssh/sshd-session.exe does not exist or is not executable

[tgauth]

As a workaround, can you download the zip package and add the sshd-session.exe to the directory?

[user8446]

Ok adding sshd-session.exe does allow v9.8 to start but as mentioned only if no additional read permissions are granted to the logs folder.

With the previous version 9.5 after installation you could re-add the read permission to the log folder and it would still start as expected.

Obviously having log access is crucial for security.

[Superberti]

Hi,

I installed OpenSSH with install-ssh.ps1, created the firewall exception, enabled the developer mode on Windows Server 2022 but "psexec -s sshd.exe -ddd" (on an elevated power shell) still has this output:

debug2: load_server_config: filename PROGRAMDATA\ssh/sshd_config
debug2: load_server_config: done config len = 2203
debug2: parse_server_config_depth: config PROGRAMDATA\ssh/sshd_config len 2203
debug3: PROGRAMDATA\ssh/sshd_config:38 setting AuthorizedKeysFile .ssh/authorized_keys
debug3: PROGRAMDATA\ssh/sshd_config:79 setting Subsystem sftp sftp-server.exe
debug3: checking syntax for 'Match Group administrators'
debug1: sshd version OpenSSH_for_Windows_9.8 Win32-OpenSSH-GitHub, LibreSSL 3.9.2
debug1: private host key #0: ssh-rsa SHA256:---
debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:---
debug1: private host key #2: ssh-ed25519 SHA256:---
debug1: get_passwd: lookup_sid() failed: 1332.
debug1: rexec_argv[1]='-ddd'
debug3: using c:\openssh-win64/sshd-session.exe for re-exec
debug2: fd 7 setting O_NONBLOCK
debug3: sock_set_v6only: set socket 7 IPV6_V6ONLY
debug1: Bind to port 22 on ::.
debug3: socketio_bind - ERROR:10013
Bind to port 22 on :: failed: Permission denied.
debug2: fd 7 setting O_NONBLOCK
debug1: Bind to port 22 on 0.0.0.0.
debug3: socketio_bind - ERROR:10013

I read the troubleshooting page with no success and now I'm out of ideas. Any hints?

[Superberti]

OK, I've found the problem: The TeamFoundationSSHService was already running on this port, so no wonder!
Choosed a different port and OpenSSH is running now.

[tgauth]

Ok adding sshd-session.exe does allow v9.8 to start but as mentioned only if no additional read permissions are granted to the logs folder.

With the previous version 9.5 after installation you could re-add the read permission to the log folder and it would still start as expected.

Obviously having log access is crucial for security.

@user8446 read access should still be permitted with 9.8. If you are seeing otherwise, can you elaborate on the exact permissions so I can setup a repro?

[user8446]

Hi yes it's the same as mentioned here: #2282 (comment)

After accessing the log, sshd will not restart.

Re-run install-sshd.ps1 which removes the permissions and sshd will then restart:

  [*] C:\ProgramData\ssh\logs
'OFFICE\Glenn' has no more access to 'C:\ProgramData\ssh\logs'.
      Repaired permissions

EDIT: I see your solution you just wrote after posting this. I restricted to the permissions you listed and it does start:

Thank you for looking into this!