[BUG] SSLEOFError traceback in salt-master logs when using event streams via salt-api

[Orlando-Richards]

Description

Seeing a traceback in my salt-master logs with Salt 3006.9, after upgrading from 3004.2, associated with streaming events via the salt-api interface.

[WARNING ] [07/Feb/2025:15:37:43] ENGINE socket.error 8
Traceback (most recent call last):
  File "/opt/saltstack/sal**t/lib/python3.10/site-packages/cheroot/server.py", line 1287, in communicate
    req.respond()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/cheroot/server.py", line 1077, in respond
    self.server.gateway(self).respond()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/cheroot/wsgi.py", line 145, in respond
    self.write(chunk)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/cheroot/wsgi.py", line 231, in write
    self.req.write(chunk)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/cheroot/server.py", line 1131, in write
    self.conn.wfile.write(EMPTY.join(buf))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/cheroot/makefile.py", line 438, in write
    res = super().write(val, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/cheroot/makefile.py", line 36, in write
    self._flush_unlocked()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/cheroot/makefile.py", line 45, in _flush_unlocked
    n = self.raw.write(bytes(self._write_buf))
  File "/opt/saltstack/salt/lib/python3.10/socket.py", line 723, in write
    return self._sock.send(b)
  File "/opt/saltstack/salt/lib/python3.10/ssl.py", line 1239, in send
    return self._sslobj.write(data)
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2426)

Setup

salt-master running in a docker container based on Rocky 8.10
salt-api running in the container
salt-minion running on a RHEL8.10 server

All running version 3006.9

Steps to Reproduce the behavior

To reproduce:

• open up an event streamer (using curl, python requests, etc) - for example:

curl -kNsS https://localhost:8000/events -H 'Accept: application/x-yaml' -H "X-Auth-Token: $TOKEN"

• Close it (before or after receiving any events, it doesn't matter) with ctrl-c

• The next event that gets issued generates a traceback in the master logs:

Expected behavior

No traceback issued in the logs.

Screenshots
n/a

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3006.9

Python Version:
        Python: 3.10.14 (main, Jun 26 2024, 11:44:37) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.12
     gitpython: 3.1.44
        Jinja2: 3.1.4
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.17.0
         smmap: 5.0.2
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: rocky 8.10 Green Obsidian
        locale: utf-8
       machine: x86_64
       release: 4.18.0-553.16.1.el8_10.x86_64
        system: Linux
       version: Rocky Linux 8.10 Green Obsidian

salt-call --versions-report

(Provided by running salt-call --versions-report for the minion versions)

# salt-call --versions-report
Salt Version:
          Salt: 3006.9

Python Version:
        Python: 3.10.14 (main, Jun 26 2024, 11:44:37) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.12
     gitpython: 3.1.44
        Jinja2: 3.1.4
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: 0.43.0
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.17.0
         smmap: 5.0.2
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: rhel 8.10 Ootpa
        locale: utf-8
       machine: x86_64
       release: 4.18.0-553.16.1.el8_10.x86_64
        system: Linux
       version: Red Hat Enterprise Linux 8.10 Ootpa

Additional context

Problem was not present in Salt 3004.2.

Can be worked around by catching the exception in /opt/saltstack/salt/lib/python3.10/site-packages/cheroot/server.py:

--- server.py	2024-07-29 09:02:46.000000000 +0100
+++ server.py	2025-02-07 16:38:12.203278916 +0000
@@ -92,6 +92,8 @@
 from .workers import threadpool
 from .makefile import MakeFile, StreamWriter
 
+from ssl import SSLEOFError
+
 
 __all__ = (
     'HTTPRequest', 'HTTPConnection', 'HTTPServer',
@@ -1284,7 +1286,10 @@
                 return False
 
             request_seen = True
-            req.respond()
+            try:
+                req.respond()
+            except SSLEOFError:
+                return False
             if not req.close_connection:
                 return True
         except socket.error as ex: