Merge pull request #98 from mirceaulinic/develop

Release 2020.3.0rc2

Author: mirceaulinic

LICENSE

@@ -186,7 +186,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright 2019 Mircea Ulinic
+   Copyright 2019-2020 Mircea Ulinic
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

README.md

@@ -21,6 +21,42 @@ and your (Proxy) Minions at the same time.
 
 > This is NOT a SaltStack product.
 
+Why ``salt-sproxy``
+-------------------
+
+``salt-sproxy`` can be used as a standalone tool to manage your devices without
+having any further requirements, as well as an extension to your existing Salt
+environment (if you already have). In other words, if you have a Salt
+installation where you manage some network devices and servers, installing
+``salt-sproxy`` on your Master will allow you to run any Salt command as always,
+e.g., executing ``salt \* test.ping`` and ``salt-sproxy \* test.ping`` will have
+the exact same effect, and result. On top of that, using ``salt-sproxy`` allows
+you to manage other devices for which you don't run (Proxy) Minions for.
+
+Of course, if you don't already have Salt, no problem, you can start managing
+your devices straight away, check out the [quick 
+start steps](https://github.com/mirceaulinic/salt-sproxy/blob/develop/docs/quick_start.rst).
+
+In brief, here are some benefits you can get by using *salt-sproxy*:
+
+- Say goodbye to the burden of managing hundreds of system services for the
+  Proxy Minion processes.
+- You can run it locally, on your own computer.
+- Python programming made a breeze - might go well with the
+  [ISalt](https://github.com/mirceaulinic/isalt) package.
+- Integrates easily with your existing Salt environment (if you have), by
+  installing the package on your Salt Master.
+- Can continue to leverage the event-driven automation and orchestration
+  methodologies.
+- REST API, see also
+  [the Salt REST API](https://salt-sproxy.readthedocs.io/en/latest/salt_api.html)
+  documentation.
+- By sending events to a Salt Master, you are able to implement whatever
+  auditing you need (e.g., what command was executed by who and when, etc.).
+- Benefit from inheriting _all_ the native Salt features and integrations
+  contributed by thousands of users, and tested in hundreds of different
+  environments, over almost a decade of development.
+
 Prerequisites
 -------------
 
@@ -198,8 +234,8 @@ Salt has natively available an HTTP API. You can read more at
 [https://docs.saltstack.com/en/latest/ref/netapi/all/salt.netapi.rest_cherrypy.html#a-rest-api-for-salt](
 https://docs.saltstack.com/en/latest/ref/netapi/all/salt.netapi.rest_cherrypy.html#a-rest-api-for-salt)
 if you haven't used it before. The usage is very simple; for salt-sproxy 
-specifically you can follow the notes from [https://salt-sproxy.readthedocs.io/en/latest/salt_api.html](
-https://salt-sproxy.readthedocs.io/en/latest/salt_api.html) how to set it up and 
+specifically you can follow the notes from [https://salt-sproxy.readthedocs.io/en/latest/salt_sapi.html](
+https://salt-sproxy.readthedocs.io/en/latest/salt_sapi.html) how to set it up and 
 use. Usage example - apply a small configuration change on a Juniper device, by 
 executing an HTTP request via the Salt API:
 
@@ -208,12 +244,10 @@ $ curl -sS localhost:8080/run -H 'Accept: application/x-yaml' \
   -d eauth='pam' \
   -d username='mircea' \
   -d password='pass' \
-  -d client='runner' \
-  -d fun='proxy.execute' \
+  -d client='sproxy' \
   -d tgt='juniper-router' \
-  -d function='net.load_config' \
-  -d text='set system ntp server 10.10.10.1' \
-  -d sync=True
+  -d fun='net.load_config' \
+  -d text='set system ntp server 10.10.10.1'
 return:
 - juniper-router:
     already_configured: false
@@ -226,8 +260,8 @@ return:
     result: true
 ```
 
-See the [documentation](https://salt-sproxy.readthedocs.io/en/latest/salt_api.html)
-for explanation, and [this example](https://salt-sproxy.readthedocs.io/en/latest/examples/salt_api.html)
+See the [documentation](https://salt-sproxy.readthedocs.io/en/latest/salt_sapi.html)
+for explanation, and [this example](https://salt-sproxy.readthedocs.io/en/latest/examples/salt_sapi.html)
 for a quick start.
 
 What's included
@@ -240,13 +274,13 @@ backwards compatibility with older Salt versions:
 
 ```
   |-- cli.py
-  |-- __init__.py
   |-- parsers.py
   |-- _roster/
   |   |-- ansible.py
-  |   `-- netbox.py
+  |   |-- file.py
+  |   |-- netbox.py
+  |   `-- pillar.py
   |-- _runners/
-  |   |-- __init__.py
   |   `-- proxy.py
   |-- scripts.py
   `-- version.py

README.rst

@@ -42,6 +42,43 @@ and your (Proxy) Minions at the same time.
 
     This is NOT a SaltStack product.
 
+Why ``salt-sproxy``
+-------------------
+
+``salt-sproxy`` can be used as a standalone tool to manage your devices without
+having any further requirements, as well as an extension to your existing Salt
+environment (if you already have). In other words, if you have a Salt
+installation where you manage some network devices and servers, installing
+``salt-sproxy`` on your Master will allow you to run any Salt command as always,
+e.g., executing ``salt \* test.ping`` and ``salt-sproxy \* test.ping`` will have
+the exact same effect, and result. On top of that, using ``salt-sproxy`` allows
+you to manage other devices for which you don't run (Proxy) Minions for.
+
+Of course, if you don't already have Salt, no problem, you can start managing
+your devices straight away, check out the `quick 
+start steps <https://github.com/mirceaulinic/salt-sproxy/blob/develop/docs/quick_start.rst>`__.
+
+In brief, here are some benefits you can get by using *salt-sproxy*:
+
+- Say goodbye to the burden of managing hundreds of system services for the
+  Proxy Minion processes.
+- You can run it locally, on your own computer.
+- Python programming made a breeze - might go well with the
+  `ISalt <https://github.com/mirceaulinic/isalt>`__ package.
+- Integrates easily with your existing Salt environment (if you have), by
+  installing the package on your Salt Master.
+- Can continue to leverage the event-driven automation and orchestration
+  methodologies.
+- REST API, see also
+  `the Salt REST API <https://salt-sproxy.readthedocs.io/en/latest/salt_api.html>`__
+  documentation.
+- By sending events to a Salt Master, you are able to implement whatever
+  auditing you need (e.g., what command was executed by who and when, etc.).
+- Benefit from inheriting _all_ the native Salt features and integrations
+  contributed by thousands of users, and tested in hundreds of different
+  environments, over almost a decade of development.
+
+
 Prerequisites
 -------------
 
@@ -50,8 +87,8 @@ would like to install it on your computer, you might want to run it under a
 `virtual environment <https://docs.python-guide.org/dev/virtualenvs/>`__.
 
 Besides the CLI, the usage remains the same as when you're running a Salt 
-environment with Proxy or regular Minions. See the following documents on how
-to get started and fully unleash the power of Salt:
+environment with Proxy or regular Minions. For example, see the following
+documents on how to get started and fully unleash the power of Salt:
 
 - `Salt in 10 minutes 
   <https://docs.saltstack.com/en/latest/topics/tutorials/walkthrough.html>`__.
@@ -230,7 +267,7 @@ Salt has natively available an HTTP API. You can read more at
 https://docs.saltstack.com/en/latest/ref/netapi/all/salt.netapi.rest_cherrypy.html#a-rest-api-for-salt 
 if you haven't used it before. The usage is very simple; for salt-sproxy 
 specifically you can follow the notes from 
-https://salt-sproxy.readthedocs.io/en/latest/salt_api.html how to set it up and 
+https://salt-sproxy.readthedocs.io/en/latest/salt_sapi.html how to set it up and 
 use. Usage example - apply a small configuration change on a Juniper device, by 
 executing an HTTP request via the Salt API:
 
@@ -240,12 +277,10 @@ executing an HTTP request via the Salt API:
     -d eauth='pam' \
     -d username='mircea' \
     -d password='pass' \
-    -d client='runner' \
-    -d fun='proxy.execute' \
+    -d client='sproxy' \
     -d tgt='juniper-router' \
-    -d function='net.load_config' \
-    -d text='set system ntp server 10.10.10.1' \
-    -d sync=True
+    -d fun='net.load_config' \
+    -d text='set system ntp server 10.10.10.1'
   return:
   - juniper-router:
       already_configured: false
@@ -258,8 +293,8 @@ executing an HTTP request via the Salt API:
       result: true
 
 See the `documentation 
-<https://salt-sproxy.readthedocs.io/en/latest/salt_api.html>`__ for explanation,
-and `this example <https://salt-sproxy.readthedocs.io/en/latest/examples/salt_api.html>`__
+<https://salt-sproxy.readthedocs.io/en/latest/salt_sapi.html>`__ for explanation,
+and `this example <https://salt-sproxy.readthedocs.io/en/latest/examples/salt_sapi.html>`__
 for a quick start.
 
 What's included
@@ -273,13 +308,13 @@ backwards compatibility with older Salt versions:
 .. code-block:: text
 
   |-- cli.py
-  |-- __init__.py
   |-- parsers.py
   |-- _roster/
+  |   |-- file.py
   |   |-- ansible.py
-  |   `-- netbox.py
+  |   |-- netbox.py
+  |   `-- pillar.py
   |-- _runners/
-  |   |-- __init__.py
   |   `-- proxy.py
   |-- scripts.py
   `-- version.py

docs/index.rst

@@ -14,6 +14,39 @@ and your (Proxy) Minions at the same time.
 
     This is NOT a SaltStack product.
 
+Why ``salt-sproxy``
+-------------------
+
+``salt-sproxy`` can be used as a standalone tool to manage your devices without
+having any further requirements, as well as an extension to your existing Salt
+environment (if you already have). In other words, if you have a Salt
+installation where you manage some network devices and servers, installing
+``salt-sproxy`` on your Master will allow you to run any Salt command as always,
+e.g., executing ``salt \* test.ping`` and ``salt-sproxy \* test.ping`` will have
+the exact same effect, and result. On top of that, using ``salt-sproxy`` allows
+you to manage other devices for which you don't run (Proxy) Minions for.
+
+Of course, if you don't already have Salt, no problem, you can start managing
+your devices straight away, check out the :ref:`quick-start` steps.
+
+In brief, here are some benefits you can get by using *salt-sproxy*:
+
+- Say goodbye to the burden of managing hundreds of system services for the
+  Proxy Minion processes.
+- You can run it locally, on your own computer.
+- Python programming made a breeze - might go well with the
+  `ISalt <https://github.com/mirceaulinic/isalt>`__ package.
+- Integrates easily with your existing Salt environment (if you have), by
+  installing the package on your Salt Master.
+- Can continue to leverage the event-driven automation and orchestration
+  methodologies.
+- REST API, see also :ref:`salt-api` documentation.
+- By sending events to a Salt Master, you are able to implement whatever
+  auditing you need (e.g., what command was executed by who and when, etc.).
+- Benefit from inheriting _all_ the native Salt features and integrations
+  contributed by thousands of users, and tested in hundreds of different
+  environments, over almost a decade of development.
+
 Install
 -------
 

docs/opts.rst

@@ -124,13 +124,23 @@ already familiar with a vast majority of them from the `salt
 
     Cache the collected Grains. Beware that this option overwrites the existing
     Grains. This may be helpful when using the ``salt-sproxy`` only, but may 
-    lead to unexpected results when running in :ref:`mixed-environments`.
+    lead to unexpected results when running in :ref:`mixed-environments`. That 
+    said, when running together with ``--use-existing-proxy``, there shouldn't
+    be any issues, as *salt-sproxy* will attemtp to use the existing (Proxy) 
+    Minion if any, otherwise it will write the collected Grains to the cache, 
+    which is a safe operation in this case (i.e., it won't overwrite the Grains 
+    of an existing Minion).
 
 .. option:: --cache-pillar
 
     Cache the collected Pillar. Beware that this option overwrites the existing
     Pillar. This may be helpful when using the ``salt-sproxy`` only, but may 
-    lead to unexpected results when running in :ref:`mixed-environments`.
+    lead to unexpected results when running in :ref:`mixed-environments`. That 
+    said, when running together with ``--use-existing-proxy``, there shouldn't
+    be any issues, as *salt-sproxy* will attemtp to use the existing (Proxy) 
+    Minion if any, otherwise it will write the compiled Pillar to the cache, 
+    which is a safe operation in this case (i.e., it won't overwrite the cached
+    Pillar of an existing Minion).
 
 .. option:: --no-cached-grains
 
@@ -356,6 +366,10 @@ already familiar with a vast majority of them from the `salt
 
     Avoid loading the list of targets from the cache.
 
+    .. versionchanged:: 2020.3.0
+
+        This option now defaults to ``True``.
+
 .. option:: --pillar-root
 
     .. versionadded:: 2020.2.0
@@ -532,3 +546,29 @@ Output Options
 
     Override the configured state_verbose value for minion
     output. Set to True or False. Default: none.
+
+Configuration file options
+--------------------------
+
+All the previous options can be provided via the CLI, as in-line arguments, as 
+well as configured in the configuration file. There are however options that 
+are available only through the configuration file:
+
+.. option:: ``target_use_cache_grains``
+
+    .. versionadded: 2020.3.0
+
+    Whether targeting should look up into the existing cache to compute the 
+    list of matching devices. This option may be particularly useful when using 
+    one of the following targeting mechanisms: ``-G`` (grain), ``-P`` (grain 
+    PCRE), or ``-C`` (compound). Default: ``True`` (it will check the cache).
+
+
+.. option:: ``target_use_cache_pillar``
+
+    .. versionadded: 2020.3.0
+
+    Whether targeting should look up into the existing cache to compute the 
+    list of matching devices. This option may be particularly useful when using 
+    one of the following targeting mechanisms: ``-I`` (pillar), ``-J`` (pillar 
+    PCRE), or ``-C`` (compound). Default:: ``True`` (it will check the cache).

salt_sproxy/_roster/__init__.py

@@ -7,6 +7,7 @@
 import fnmatch
 import logging
 
+import salt.cache
 from salt.ext import six
 import salt.utils.minions
 
@@ -26,6 +27,15 @@ def load_cache(pool, __runner__, opts, tgt, tgt_type=None):
     Load the Pillar and Grain cache, as required, and merge the Roster Grains
     and Pillar into.
     '''
+    if opts.get('grains'):
+        for device, device_opts in six.iteritems(pool):
+            if 'minion_opts' not in device_opts:
+                device_opts['minion_opts'] = {}
+            if 'grains' not in device_opts['minion_opts']:
+                device_opts['minion_opts']['grains'] = {}
+            device_opts['minion_opts']['grains'] = salt.utils.dictupdate.merge(
+                opts['grains'], device_opts['minion_opts']['grains'], merge_lists=True,
+            )
     if tgt_type in ('glob', 'pcre', 'list'):
         # When the target type is glob, pcre, or list, we don't require grains
         # or pillar loaded from the cache, because the targeting won't depend on
@@ -39,38 +49,40 @@ def load_cache(pool, __runner__, opts, tgt, tgt_type=None):
     # Runners as they rely on fetching data from the Master, for Minions that
     # are accepted. What we're doing here is reading straight from the cache.
     log.debug('Loading cached and merging into the Roster data')
-    cache_pool = __runner__['cache.list']('minions')
+    cache = salt.cache.factory(opts)
+    cache_pool = cache.list('minions')
     for device in cache_pool:
         if device not in pool:
+            log.trace('%s has cache, but is not in the Roster pool', device)
             continue
         if 'minion_opts' not in pool[device]:
             pool[device]['minion_opts'] = {'grains': {}, 'pillar': {}}
         cache_key = 'minions/{}/data'.format(device)
-        if opts.get('use_cached_grains', True) and tgt_type in (
+        if opts.get('target_use_cached_grains', True) and tgt_type in (
             'compound',
             'grain',
             'grain_pcre',
             'nodegroup',
         ):
             log.debug('Fetching cached Grains for %s', device)
-            cache_grains = __runner__['cache.fetch'](cache_key, 'grains')
-            if cache_grains:
+            cached_grains = cache.fetch(cache_key, 'grains')
+            if cached_grains:
                 pool[device]['minion_opts']['grains'] = salt.utils.dictupdate.merge(
-                    cache_grains,
+                    cached_grains,
                     pool[device]['minion_opts'].get('grains', {}),
                     merge_lists=True,
                 )
-        if opts.get('use_cached_pillar', True) and tgt_type in (
+        if opts.get('target_use_cached_pillar', True) and tgt_type in (
             'compound',
             'pillar',
             'pillar_pcre',
             'nodegroup',
         ):
             log.debug('Fetching cached Pillar for %s', device)
-            cache_pillar = __runner__['cache.fetch'](cache_key, 'pillar')
-            if cache_pillar:
+            cached_pillar = cache.fetch(cache_key, 'pillar')
+            if cached_pillar:
                 pool[device]['minion_opts']['pillar'] = salt.utils.dictupdate.merge(
-                    cache_pillar,
+                    cached_pillar,
                     pool[device]['minion_opts'].get('pillar', {}),
                     merge_lists=True,
                 )