New function similar to salt.runners.manage.status, where tgt and tgt_type can be used, but without commands being sent to minions (like alived/allowed/etc) #68081

cstrahan · 2025-06-16T19:15:11Z

cstrahan
Jun 16, 2025

Could we have a function similar to salt.runners.manage.status, but which doesn't send commands to minions (in the same way alived/alowed/etc work)?

To motivate this (or to get alternative suggestions), here's what I'm trying to do:

I have salt minions running in EC2, and when the instances are terminated, I have CloudWatch place these instance termination events in an SQS queue. I then have a script that read from the queue, and then uses the instance-id to look up the respective minion key, similar to:

curl -sS https://${salt_master}/run \
        -H 'Accept: application/json' \
        -H 'Content-type: application/json' \
        -H 'X-Requested-With: XMLHttpRequest' \
        -d '[{
                "client": "runner",
                "fun": "manage.status",
                "tgt": "G@meta-data:instance-id:'$instance_id'",
                "tgt_type": "compound",
                "timeout": 1,
                "token": "'${token}'"
        }]'

The problem I'm having is that this results in the salt master attempting to communicate with the host, which shouldn't exist any more. I tried setting the timeout to something low to prevent this from hanging, and that often seems to work. However, for some instance-ids the request results in this response:

<!DOCTYPE html PUBLIC
"-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta>
    <title>504 Gateway Timeout</title>
    <style type="text/css">
    #powered_by {
        margin-top: 20px;
        border-top: 2px solid black;
        font-style: italic;
    }

    #traceback {
        color: red;
    }
    </style>
</head>
    <body>
        <h2>504 Gateway Timeout</h2>
        <p>The gateway server did not receive a timely response</p>
        <pre id="traceback"></pre>
    <div id="powered_by">
      <span>
        Powered by <a href="http://www.cherrypy.dev">CherryPy 18.9.0</a>
      </span>
    </div>
    </body>
</html>

Ideally, I would be able to query by the salt-master's in memory cache of last known minion status and grain values without it ever attempting to send requests to the minions.

Is there some existing functionality that I can use that I've missed, or would this be a feature you would consider adding (or accepting as a contribution)?

In case it helps, I'm working with salt version 3004.1.

mattp- · 2025-06-16T19:24:24Z

mattp-
Jun 16, 2025

salt-run manage.list_state shows connected without reaching out to minion. give that a shot

8 replies

mattp- Jun 17, 2025

wrt list_not_state not returning what you expect, what the code is doing is comparing /sbin/ss output on your ret/pub port (4505/4506), and joining those ip's with grains ipv4 / ipv6 addrs to identify minions. so if your minion isnt connecting via the ip the master sees (nat, floating ips, etc), this unfortunately wont work without you augmenting the code that iterates ipv4 addrs.

cstrahan Aug 6, 2025
Author

So, after squinting at the salt source code for a while, I think this should suffice for a "no ping" variant of status:

def status_no_ping(tgt="*", tgt_type="glob"):
    ret = {}

    ckminions = salt.utils.minions.CkMinions(__opts__)
    matching = set(ckminions.check_minions(tgt, tgt_type, greedy=False).get("minions", []))

    up = set(list_state(subset=matching))
    down = matching - up

    ret["up"] = list(up)
    ret["down"] = list(down)
    return ret

Haven't tested it yet, but hopefully I can do that shortly and report back.

cstrahan Aug 7, 2025
Author

This appears to work (I still see a bunch of minions as "down", but that's probably for the same reason you mentioned earlier, regarding ports):

In /etc/salt/master, set

runner_dirs: [/srv/salt/runners/]

In /srv/salt/runners/backtrace.py:

import salt.utils.minions

def status_no_ping(
    tgt="*", tgt_type="glob"
):
    ret = {}

    ckminions = salt.utils.minions.CkMinions(__opts__)
    matching = set(ckminions.check_minions(tgt, tgt_type, greedy=False).get("minions", []))

    up = set(__salt__['manage.list_state'](subset=matching))
    down = matching - up

    ret["up"] = list(up)
    ret["down"] = list(down)
    return ret

And then I can do:

% sudo salt-run backtrace.status_no_ping tgt='G@meta-data:instance-id:i-000f9427ba7d5ec45' tgt_type='compound' --out=json
{
    "up": [],
    "down": [
        "submit-i-000f9427ba7d5ec45-aws-ap-northeast-1"
    ]
}

Now, I think what I'll do is use this runner to resolve the minion key from a reactor connected to an SQS queue containing CloudWatch event data for instance terminations (which includes the instance-id), a la https://groups.google.com/g/salt-users/c/eQOKC84XmPc?pli=1

cstrahan Aug 13, 2025
Author

Actually, it looks like list_state treats an empty subset array as if no subset were passed. So the runner needs to look something like:

import salt.runners.manage
import salt.utils.minions

def status_no_ping(
    tgt="*", tgt_type="glob"
):
    ret = {}

    ckminions = salt.utils.minions.CkMinions(__opts__)
    matching = set(ckminions.check_minions(tgt, tgt_type, greedy=False).get("minions", []))

    if matching:
        up = set(__salt__['manage.list_state'](subset=matching))
        down = matching - up
        ret["up"] = list(up)
        ret["down"] = list(down)
    else:
        ret["up"] = []
        ret["down"] = []

    return ret

cstrahan Aug 13, 2025
Author

Wow, I just realized that I have everything I need to solve my problem out-of-the-box, without any custom runners/etc.

{%- if data['message']['detail']['state'] == "terminated" %}

{% set instance_id = data['message']['detail']['instance-id'] %}
{% set tgt = 'G@meta-data:instance-id:' + instance_id %}
{% set cache = salt['saltutil.runner']('cache.grains', tgt=tgt, tgt_type='compound') %}
{% for key in cache %}
  {% set tgt_grains = cache[key] %}
  {% set service = tgt_grains['service'] %}
  # only enable this for the "submit" service, for now
  {% if service == 'submit' %}

Delete Key:
  wheel.key.delete:
    - match: "{{ key }}"

  {% endif %}

{% endfor %}

{%- endif %}

I realized that cache.grains gives me the matching minion(s)' key(s), and I need the grains for some other logic anyway. I don't really care about using salt to discover the minion status (up vs down) of the node in this case, because the message.detail.state already tells me that the instance is terminated.

Hoping that someone finds this useful, and spares them some effort.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

New function similar to salt.runners.manage.status, where tgt and tgt_type can be used, but without commands being sent to minions (like alived/allowed/etc) #68081

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 8 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

New function similar to salt.runners.manage.status, where tgt and tgt_type can be used, but without commands being sent to minions (like alived/allowed/etc) #68081

Uh oh!

cstrahan Jun 16, 2025

Replies: 1 comment · 8 replies

Uh oh!

mattp- Jun 16, 2025

Uh oh!

mattp- Jun 17, 2025

Uh oh!

cstrahan Aug 6, 2025 Author

Uh oh!

Uh oh!

cstrahan Aug 7, 2025 Author

Uh oh!

cstrahan Aug 13, 2025 Author

Uh oh!

cstrahan Aug 13, 2025 Author

cstrahan
Jun 16, 2025

Replies: 1 comment 8 replies

mattp-
Jun 16, 2025

cstrahan Aug 6, 2025
Author

cstrahan Aug 7, 2025
Author

cstrahan Aug 13, 2025
Author

cstrahan Aug 13, 2025
Author