[Bug]: salt-ssh state execution uses thin_dir path on master for caching

[hurzhurz]

What happened?

When executing a state.apply via salt-ssh, it uses the thin_dir path (which should only be used on the target-minion) as fileserver-cache on the master (instead of eg. /var/cache/salt/master/).

Example:

1. No thin_dir present after just a first test.ping:

root@test-saltssh-master:~# salt-ssh saltsshtest test.ping
saltsshtest:
    True
root@test-saltssh-master:~# ls -la /var/tmp/ | grep salt

2. But after a state.apply:

root@test-saltssh-master:~# salt-ssh saltsshtest state.apply test
saltsshtest:
----------
          ID: always-passes
    Function: test.succeed_without_changes
        Name: foo
      Result: True
     Comment: Success!
     Started: 12:21:22.094031
    Duration: 0.346 ms
     Changes:

Summary for saltsshtest
------------
Succeeded: 1
Failed:    0
------------
Total states run:     1
Total run time:   0.346 ms

root@test-saltssh-master:~# ls -la /var/tmp/ | grep salt
drwxr-xr-x  3 root root 4096 Nov 14 12:21 .root_92f580_salt

root@test-saltssh-master:~# find /var/tmp/.root_92f580_salt
/var/tmp/.root_92f580_salt
/var/tmp/.root_92f580_salt/running_data
/var/tmp/.root_92f580_salt/running_data/var
/var/tmp/.root_92f580_salt/running_data/var/cache
/var/tmp/.root_92f580_salt/running_data/var/cache/salt
/var/tmp/.root_92f580_salt/running_data/var/cache/salt/minion
/var/tmp/.root_92f580_salt/running_data/var/cache/salt/minion/roots
/var/tmp/.root_92f580_salt/running_data/var/cache/salt/minion/roots/mtime_map
/var/tmp/.root_92f580_salt/running_data/var/cache/salt/minion/files
/var/tmp/.root_92f580_salt/running_data/var/cache/salt/minion/files/base
/var/tmp/.root_92f580_salt/running_data/var/cache/salt/minion/files/base/test.sls

If you define a minion specific thin_dir path in the roster file, it also uses that minion specific path on the master.

This actually lead to permission problems for me, as I wanted to run salt-ssh as the salt user (like the salt-master service, so that there isn't a permission conflict in the shared /var/cache/salt/master/jobs directory).
With a custom thin_dir, the salt user on the master had not the permissions to create that path on the master.

I think the cause is in the general ssh function wrapper, there the fileclient is initialized with a cache dir path from the wrong opts.

salt/salt/client/ssh/__init__.py

Line 1267 in e309bf1

wrapper.fsclient.opts["cachedir"] = opts["cachedir"]

And in addition, the state class isn't initialized here with the fileclient from the general wrapper function:

salt/salt/client/ssh/state.py

Line 44 in e309bf1

super().__init__(opts, pillar_override, initial_pillar=initial_pillar)

This is my interims fix:

--- a/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/ssh/__init__.py    2025-09-17 21:43:08.000000000 +0000
+++ b/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/ssh/__init__.py     2025-11-14 12:56:30.040723283 +0000
@@ -1264,7 +1264,6 @@
             minion_opts=self.minion_opts,
             **self.target,
         )
-        wrapper.fsclient.opts["cachedir"] = opts["cachedir"]
         self.wfuncs = salt.loader.ssh_wrapper(opts, wrapper, self.context)
         wrapper.wfuncs = self.wfuncs
--- a/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/ssh/state.py       2025-09-17 21:43:08.000000000 +0000
+++ b/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/ssh/state.py        2025-11-14 12:57:14.356152667 +0000
@@ -41,7 +41,7 @@
     ):
         self.wrapper = wrapper
         self.context = context
-        super().__init__(opts, pillar_override, initial_pillar=initial_pillar)
+        super().__init__(opts, pillar_override, initial_pillar=initial_pillar, file_client=wrapper.fsclient)

     def load_modules(self, data=None, proxy=None):
         """

Type of salt install

Official deb

Major version

3006.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

ubuntu-22.04

salt --versions-report output

Salt Version:
          Salt: 3006.16

Python Version:
        Python: 3.10.18 (main, Sep  5 2025, 22:48:51) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.10.0
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.20.6
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: ubuntu 22.04.5 jammy
        locale: utf-8
       machine: x86_64
       release: 6.6.87.2-microsoft-standard-WSL2
        system: Linux
       version: Ubuntu 22.04.5 jammy

[lkubb]

The file client cache dir contains rendered states, which usually depend on minion-specific grains and pillars. Sharing the cache dir/fileclient between multiple SSH minions (and the master) thus risks cross-contamination, which is probably the reason for this behavior.

Consider using a cachedir like /var/cache/salt/master/ssh/<minion-id> instead and not sharing the fileclient between multiple targets in the same run.

Side note: In 3007+, there is a new cp wrapper, which (IIRC) relies on the shared cache path (can be refactored).

[hurzhurz]

In other situations it already uses a minion specific cache path on the master: /var/cache/salt/master/salt-ssh/<minion-id>/
And what I did see in the thin_dir path didn't look like prerendered stuff. Just collected sls files etc. that I assume are later bundled as tgz file to be transferred and used with state.pkg.

Anyway, I haven't looked that deep into the inner guts of salt-ssh and also have no idea how it was intended.

And I did there where changes about the cp wrapper. Maybe they should be backported?