[Bug]: Salt master cluster not able to forward events to peers.

[PCouaillierM]

What happened?

Salt master cluster is not working properly

The salt master cluster is not working properly is is not able to forward events.

Nov 16 12:25:43 salt-master-1 salt-master[2502232]: [WARNING ] Unable to forward event to cluster peer Task-7202                 13:25:43 [30/902]
Nov 16 12:25:43 salt-master-1 salt-master[2502241]: [WARNING ] Token failed to decrypt: ValueError('Decryption failed')
Nov 16 12:25:43 salt-master-1 salt-master[2502253]: [WARNING ] Token failed to decrypt: ValueError('Decryption failed')
Nov 16 12:25:43 salt-master-1 salt-master[2502239]: [WARNING ] Token failed to decrypt: ValueError('Decryption failed')
Nov 16 12:25:43 salt-master-1 salt-master[2502239]: [WARNING ] Token failed to decrypt: ValueError('Decryption failed')
Nov 16 12:25:43 salt-master-1 salt-master[2502243]: [WARNING ] Token failed to decrypt: ValueError('Decryption failed')
Nov 16 12:25:43 salt-master-1 salt-master[2502239]: [WARNING ] Token failed to decrypt: ValueError('Decryption failed')
Nov 16 12:25:43 salt-master-1 salt-master[2502232]: [CRITICAL] Unhandled error while polling master events
Nov 16 12:25:43 salt-master-1 salt-master[2502232]: Traceback (most recent call last):
Nov 16 12:25:43 salt-master-1 salt-master[2502232]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/server.py", line 1217, i
n handle_pool_publish
Nov 16 12:25:43 salt-master-1 salt-master[2502232]:     aes = data["peers"][self.opts["id"]]["aes"]
Nov 16 12:25:43 salt-master-1 salt-master[2502232]: KeyError: 'aes'

Configuration

Config file for salt-master-1:

cluster_peers:
- salt-master-2
- salt-master-3
cluster_pki_dir: /share/srv/salt/pki/master
cachedir: /share/srv/salt/cache/master

saltenv: vagrant
pillarenv: vagrant

file_roots:
  __env__:
    -  /synced/devops-scripts/saltstack/states/
    -  /synced/devops-scripts/saltstack/files/base/
    -  /synced/devops-scripts/saltstack/files/__env__/

pillar_roots:
  __env__:
    - /synced/devops-scripts/saltstack/pillars/__env__/

ext_pillar:
  - stack:
      - /synced/devops-scripts/saltstack/pillarstack/stack.cfg

salt-master-1 logs

Nov 17 16:59:10 salt-master-1 systemd[1]: Starting salt-master.service - The Salt Master Server...
Nov 17 16:59:11 salt-master-1 systemd[1]: Started salt-master.service - The Salt Master Server.
Nov 17 16:59:12 salt-master-1 salt-master[53151]: [WARNING ] Peer key missing PosixPath('/share/srv/salt/pki/master/peers/salt-master-1.pub')
Nov 17 16:59:12 salt-master-1 salt-master[53151]: [WARNING ] Peer key missing PosixPath('/share/srv/salt/pki/master/peers/salt-master-2.pub')
Nov 17 16:59:12 salt-master-1 salt-master[53151]: [WARNING ] Peer key missing PosixPath('/share/srv/salt/pki/master/peers/salt-master-3.pub')
Nov 17 16:59:12 salt-master-1 salt-master[53223]: [CRITICAL] Unhandled error while polling master events
Nov 17 16:59:12 salt-master-1 salt-master[53223]: Traceback (most recent call last):
Nov 17 16:59:12 salt-master-1 salt-master[53223]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/server.py", line 1214, in handle_pool_publish
Nov 17 16:59:12 salt-master-1 salt-master[53223]:     aes = data["peers"][self.opts["id"]]["aes"]
Nov 17 16:59:12 salt-master-1 salt-master[53223]: KeyError: 'salt-master-1_master'

salt-master-2 logs

Nov 17 16:40:01 salt-master-2 systemd[1]: Starting salt-master.service - The Salt Master Server...
Nov 17 16:40:01 salt-master-2 systemd[1]: Started salt-master.service - The Salt Master Server.
Nov 17 16:40:02 salt-master-2 salt-master[51505]: [WARNING ] Peer key missing PosixPath('/share/srv/salt/pki/master/peers/salt-master-1.pub')
Nov 17 16:40:02 salt-master-2 salt-master[51505]: [WARNING ] Peer key missing PosixPath('/share/srv/salt/pki/master/peers/salt-master-3.pub')
Nov 17 16:40:04 salt-master-2 salt-master[51575]: [CRITICAL] Unhandled error while polling master events
Nov 17 16:40:04 salt-master-2 salt-master[51575]: Traceback (most recent call last):
Nov 17 16:40:04 salt-master-2 salt-master[51575]:   File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/server.py", line 1214, in handle_pool_publish
Nov 17 16:40:04 salt-master-2 salt-master[51575]:     aes = data["peers"][self.opts["id"]]["aes"]
Nov 17 16:40:04 salt-master-2 salt-master[51575]: KeyError: 'salt-master-2_master'

Content of gluster mounted volume (minion list eluded)

root@salt-master-1:~# find /share/srv/salt -type f | grep master | grep -v minions  | grep -v sessions
/share/srv/salt/cache/master/roots/mtime_map
/share/srv/salt/cache/master/.salt_key
/share/srv/salt/cache/master/.root_key
/share/srv/salt/pki/master/.aes
/share/srv/salt/pki/master/cluster.pub
/share/srv/salt/pki/master/peers/salt-master-3_master.pub
/share/srv/salt/pki/master/peers/salt-master-2_master.pub
/share/srv/salt/pki/master/peers/salt-master-1_master.pub
/share/srv/salt/pki/master/cluster.pem

Type of salt install

Official deb

Major version

3007.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

debian-12

salt --versions-report output

Salt Version:
          Salt: 3007.8
 
Python Version:
        Python: 3.10.18 (main, Sep  5 2025, 22:48:51) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
  cryptography: 42.0.5
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.20.6
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.4.2
           ZMQ: 4.3.4
 
Salt Package Information:
  Package Type: onedir
 
System Versions:
          dist: debian 12.4 bookworm
        locale: utf-8
       machine: x86_64
       release: 6.1.0-17-amd64
        system: Linux
       version: Debian GNU/Linux 12.4 bookworm
 
Salt Version:
          Salt: 3007.8
 
Python Version:
        Python: 3.10.18 (main, Sep  5 2025, 22:48:51) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
  cryptography: 42.0.5
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.20.6
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.4.2
           ZMQ: 4.3.4
 
Salt Package Information:
  Package Type: onedir
 
System Versions:
          dist: debian 12.4 bookworm
        locale: utf-8
       machine: x86_64
       release: 6.1.0-17-amd64
        system: Linux
       version: Debian GNU/Linux 12.4 bookworm

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here's some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Salt's Contributor Guide
• Join our Community Discord
• Salt Project YouTube channel
• GitHub Discussions

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject.pdl@broadcom.com. We're glad you've joined our community and look forward to doing awesome things with you!

[dwoz]

Nov 17 16:40:02 salt-master-2 salt-master[51505]: [WARNING ] Peer key missing PosixPath('/share/srv/salt/pki/master/peers/salt-master-1.pub')
Nov 17 16:40:02 salt-master-2 salt-master[51505]: [WARNING ] Peer key missing PosixPath('/share/srv/salt/pki/master/peers/salt-master-3.pub')

These log lines indicate that the master can't find the shared keys. I'd start troubleshooting by trying to determine why.