[BUG] salt-ssh pkg.install fails

[smcgrat]

Description
Using salt 3006.9 on a Rocky Linux 8.10 test salt master the salt-ssh tool fails to install packages on a Rocky Linux 8.10 test minion and gives and error that "'pkg.install' is not available" or 'pkg.installed' was not found.

Setup

Create 2 Rocky Linux 8.10 virtual machines, install salt with bootstrap on both, start salt-master service on one and salt-minion service on the other. Edit the /etc/salt/master file on the minion to set master: to the fqdn of the master vm.

• on-prem machine
• [ x ] VM (OpenNebula, KVM)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• [ x ] onedir packaging
• [ x ] used bootstrap to install

Steps to Reproduce the behavior

This errors: salt-ssh -i minion.fqdn pkg.install nmap as follows:

    ----------
    retcode:
        255
    stderr:
        /var/tmp/.root_2bbfa8_salt/pyall/salt/utils/psutil_compat.py:18: DeprecationWarning: Please stop importing 'salt.utils.psutil_compat' and instead import 'psutil' directly as there's no longer a need for a compatability layer. The 'salt.utils.psutil_compat' will go away on Salt 3008.0 (Argon).
          "Please stop importing 'salt.utils.psutil_compat' and instead import "
        /var/tmp/.root_2bbfa8_salt/pyall/salt/modules/x509.py:100: DeprecationWarning: The x509 modules are deprecated. Please migrate to the replacement modules (x509_v2). They are the default from Salt 3008 (Argon) onwards.
          "The x509 modules are deprecated. Please migrate to the replacement "
        'pkg.install' is not available.
    stdout:

Or as follows if using a salt state, salt-ssh -i minion.fqdn state.sls installsalt.update

          ID: install nmap to test
    Function: pkg.installed
        Name: nmap
      Result: False
     Comment: State 'pkg.installed' was not found in SLS 'installsalt.update'
              Reason: 'pkg' __virtual__ returned False: pkg module could not be loaded
     Changes: 

The salt state contains the following in it in case relevant.

install nmap to test:
  pkg.installed:
    - name: nmap

Using salt directly, i.e. no salt-ssh, e.g. salt hprc-guest-114-230.tchpc.tcd.ie state.sls installsalt.update, works though.

So the issue seems to be limited to salt-ssh.

It also does not occur from a salt master running salt version 3005.5. I can get pkg.install to work with salt-ssh from a 3005.5 master.

As per I have set the pkg provider to yumpkg in /etc/salt/minion on the minion host as follows:

providers:
  pkg: yumpkg

I've also tried updating the state file to set the provider as follows:

install nmap to test:
  pkg.installed:
    - name: nmap
    - provider: yumpkg

This does not resolve the issue though.

Debug outputs.

salt-ssh-pkg.install.txt is the output of command salt-ssh -l debug -i 'minion.fqdn' pkg.install nmap.

salt-ssh-state.sls.txt is the output of command salt-ssh -l debug -i 'hprc-guest-114-230.tchpc.tcd.ie' state.sls installsalt.update.

Expected behaviour

pkg.install will install packages.

Versions Report

Salt Version:
          Salt: 3006.9
 
Python Version:
        Python: 3.10.14 (main, Jun 26 2024, 11:44:37) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.4
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.17.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: rocky 8.10 Green Obsidian
        locale: utf-8
       machine: x86_64
       release: 4.18.0-553.40.1.el8_10.x86_64
        system: Linux
       version: Rocky Linux 8.10 Green Obsidian

Thanks in advance.

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Salt’s Contributor Guide
• Join our Community Discord
• Salt Project YouTube channel
• Community Wiki

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at [email protected]. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[lkubb]

This is probably #67091. You can try applying the patch in #67152 and regenerating your thin package before trying your command again or downgrading to 3006.8.

The divergence between regular Salt and Salt-SSH is caused by Salt-SSH needing to use the (outdated) system Python, while regular Salt can bring its own Python environment.

[smcgrat]

Thanks for the information. Yes, downgrading to 3006.8 works around this issue. Apologies for my ignorance. What is meant by "regenerating your thin package" please?

[lkubb]

Salt-SSH packs Salt core and library dependencies into a tar archive and sends it to the target so it can be invoked (this is called the thin archive/thin tar). It's not regenerated if you patch Salt locally (it should be if the version changes though, which is why downgrading worked without).

You can regenerate it e.g. by running salt-ssh --wipe state.show_highstate targetminion.

[dmurphy18]

@smcgrat Given the solutions that @lkubb mentioned which fixed the issue, and the fix PR #67152, is merged and will be available in the next release, which is soon, please consider closing this issue.