Merge pull request #236 from nautobot/develop

Version 1.0.0 to main

.github/workflows/ci.yml

@@ -1,6 +1,6 @@
 ---
 name: "CI"
-on:  # yamllint disable
+on: # yamllint disable
   - "push"
   - "pull_request"
 
@@ -100,7 +100,7 @@ jobs:
       fail-fast: true
       matrix:
         python-version: ["3.6", "3.7", "3.8", "3.9"]
-        nautobot-version: ["1.0.1"]
+        nautobot-version: ["1.2.1"]
     runs-on: "ubuntu-20.04"
     env:
       INVOKE_NAUTOBOT_GOLDEN_CONFIG_PYTHON_VER: "${{ matrix.python-version }}"
@@ -138,7 +138,7 @@ jobs:
       - name: "Upload binaries to release"
         uses: "svenstaro/upload-release-action@v2"
         with:
-          repo_token: "${{ secrets.NTC_GITHUB_TOKEN }}"
+          repo_token: "${{ secrets.GH_NAUTOBOT_BOT_TOKEN }}"
           file: "dist/*"
           tag: "${{ github.ref }}"
           overwrite: true

.pydocstyle.ini

@@ -2,7 +2,7 @@
 convention = google
 inherit = false
 match = (?!__init__).*\.py
-match-dir = (?!tests)[^\.].*
+match-dir = (?!tests|migrations)[^\.].*
 # D212 is enabled by default in google convention, and complains if we have a docstring like:
 # """
 # My docstring is on the line after the opening quotes instead of on the same line as them.

CHANGELOG.md

@@ -1,5 +1,43 @@
 # Changelog
 
+## v1.0.0 - 2022-04
+
+### Announcements
+
+- Nautobot Golden Config 1.0.X will officially only support versions 1.2.0 - 1.3.99
+
+### Added
+
+- #180 Added Renovate for proactive package management
+- #158 Allow for Jinja2 Filters to be used by GoldenConfig templates 
+- #167 Added support for multiple repos
+- #205 Added support for multiple repos via multiple golden config settings
+- #206 Add Git datasource to load GC properties
+- #218 Added ability to storre SoTAgg field leveraging Nautobot saved GraphQl query
+- #225 Added support for nautobot secrets group on git repos
+- #234 Minor update to FAQ
+
+### Changed
+
+- #171 Changed the release policy
+- #158 Changed variable job_result to nautobot_job
+- #186 Update mariadb Docker tag to v10.7
+- #187 Update postgres Docker tag to v14
+- #188 Update Markdown dependency
+- #190 Update to Nautobot 1.2.0
+- #190 Remove Nautobot 1.0 specific code
+- #211 Update dependency mariadb to v10.8 
+- #229 Updated navigation to a dedicated top level menu
+
+### Fixed
+
+- #176 Fixed Pylint issue
+- #182 Add reference to Nornir plugin for installation
+- #183 Fixed documentation for sot_agg_transposer default
+- #184 Fix markdown links in quick-start
+- #194 Detailed Error Handling in get_job_filter helper
+- #229 Fixed #165, Configuration Compliance List View "Device" filter doesn't work
+
 ## v0.9.10 - 2021-11
 
 ### Announcements

README.md

@@ -32,7 +32,10 @@ but not want to use other features, which is perfectly fine to do so.
 - [Navigating SoTAgg](./docs/navigating-sot-agg.md)
 - [Navigating Compliance](./docs/navigating-compliance.md)
 - [Navigating JSON Compliance](./docs/navigating-compliance-json.md)
+- [Navigating Custom Compliance](./docs/navigating-compliance-custom.md)
 - [FAQ](./docs/FAQ.md)
+- [Upgrade Notes](./docs/upgrade.md)
+
 
 ## Screenshots
 
@@ -67,10 +70,12 @@ The project is following Network to Code software development guidelines and are
 
 The branching policy includes the following tenets:
 
-- The develop branch is the branch of the next major or minor version planned.
-- The `stable-<major>.<minor>` branch is the branch of the latest version within that major/minor version
-- PRs intended to add new features should be sourced from the develop branch
-- PRs intended to address bug fixes and security patches should be sourced from `stable-<major>.<minor>`
+- The develop branch is the branch of the next major and minor paired version planned.
+- The `stable-<major>.<minor>` branch is the branch of the latest version within that major/minor version.
+- The `stable-<major>.<minor>` branch will have all of the latest bug fixes and security patches, and may or may not represent the released version.
+- PRs intended to add new features should be sourced from the develop branch.
+- PRs intended to add new features that break backward compatability should be discussed before a PR is created.
+- PRs intended to address bug fixes and security patches should be sourced from `stable-<major>.<minor>`.
 
 Nautobot Golden Config will observe semantic versioning, as of 1.0. This may result in an quick turn around in minor versions to keep
 pace with an ever growing feature set.
@@ -79,16 +84,35 @@ pace with an ever growing feature set.
 
 Nautobot Golden Config has currently no intended scheduled release schedule, and will release new feature in minor versions.
 
+When a new release of any kind (e.g. from develop to main, or a release of a `stable-<major>.<minor>`) is created the following should happen.
+- A release PR is created with:
+  - Update to the CHANGELOG.md file to reflect the changes.
+  - Change the version from `<major>.<minor>.<patch>-beta` to `<major>.<minor>.<patch>` in both pyproject.toml and `nautobot.__init__.__version__`.
+  - Set the PR to the proper branch, e.g. either `main` or `stable-<major>.<minor>`.
+- Ensure the tests for the PR pass.
+- Merge the PR.
+- Create a new tag:
+  - The tag should be in the form of `v<major>.<minor>.<patch>`.
+  - The title should be in the form of `v<major>.<minor>.<patch>`.
+  - The description should be the changes that were added to the CHANGELOG.md document.
+- If merged into main, then push from main to develop, in order to retain the merge commit created when the PR was merged
+- If the is a new `<major>.<minor>`, create a `stable-<major>.<minor>` branch and push that to the repo.
+- A post release PR is created with.
+  - Change the version from `<major>.<minor>.<patch>` to `<major>.<minor>.<patch + 1>-beta` in both pyproject.toml and `nautobot.__init__.__version__`.
+  - Set the PR to the proper branch, e.g. either `develop` or `stable-<major>.<minor>`.
+  - Once tests pass, merge. 
+
 ## Deprecation Policy
 
 Support of upstream Nautobot will be announced 1 minor or major version ahead. Deprecation policy will be announced within the
-CHANGELOG.md file, and updated in the table below. There will be a `stable-<major>.<minor>` branch that will be minimally maintained.
-Any security enhancements or major bugs will be supported for a limited time. 
+CHANGELOG.md file, and updated in the table below. There will be a `stable-<major>.<minor>` branch that will be minimally maintained, 
+for any security enhancements or major bugs will be supported for a limited time. 
 
 | Golden Config Version | Nautobot First Support Version | Nautobot Last Support Version |
 | --------------------- | ------------------------------ | ----------------------------- |
-| 0.9.X                 | 1.0                            | 1.2 [Official]                |
-| 1.0.X                 | 1.2                            | 1.2 [Tentative]               |
+| 0.9.X                 | 1.0.0                          | 1.2.99 [Official]             |
+| 0.10.X                | 1.0.0                          | 1.2.99 [Official]             |
+| 1.0.X                 | 1.2.0                          | 1.3.99 [Official]             |
 
 ## CLI Helper Commands
 

development/Dockerfile

@@ -18,4 +18,4 @@ RUN apt update
 RUN apt install -y libmariadb-dev-compat gcc
 RUN pip install mysqlclient
 
-COPY development/nautobot_config.py /opt/nautobot/nautobot_config.py
+COPY development/nautobot_config.py /opt/nautobot/nautobot_config.py

development/docker-compose.mysql.yml

@@ -1,7 +1,7 @@
 ---
 services:
   db:
-    image: "mariadb:10.6"
+    image: "mariadb:10.8"
     env_file:
       - "dev.env"
     volumes:

development/docker-compose.yml

@@ -37,7 +37,7 @@ services:
     entrypoint: "nautobot-server celery worker -B -l INFO"
     <<: *nautobot-base
   db:
-    image: "postgres:13"
+    image: "postgres:14"
     env_file:
       - "dev.env"
     volumes:

development/nautobot_config.py

@@ -141,6 +141,17 @@
                 },
             },
         },
+        # dispatcher_mapping may be necessary if you get an error `Cannot import "<foo>". Is the library installed?`
+        # when you run a backup job, and <foo> is the name of the platform applied to the device.
+        # to the Nornir driver names ("arista_eos", "cisco_ios", etc.).
+        # "dispatcher_mapping": {
+        #     "eos": "nornir_nautobot.plugins.tasks.dispatcher.arista_eos.NautobotNornirDriver",
+        #     "arbitrary_platform_name": "nornir_nautobot.plugins.tasks.dispatcher.arista_eos.NautobotNornirDriver",
+        #     "ios": "nornir_nautobot.plugins.tasks.dispatcher.cisco_ios.NautobotNornirDriver",
+        #     "iosxe": "nornir_nautobot.plugins.tasks.dispatcher.cisco_ios.NautobotNornirDriver",
+        #     "junos": "nornir_nautobot.plugins.tasks.dispatcher.juniper_junos.NautobotNornirDriver",
+        #     "nxos": "nornir_nautobot.plugins.tasks.dispatcher.cisco_nxos.NautobotNornirDriver",
+        # },
     },
     "nautobot_golden_config": {
         "per_feature_bar_width": float(os.environ.get("PER_FEATURE_BAR_WIDTH", 0.15)),
@@ -151,5 +162,34 @@
         "enable_intended": is_truthy(os.environ.get("ENABLE_INTENDED", True)),
         "enable_sotagg": is_truthy(os.environ.get("ENABLE_SOTAGG", True)),
         "sot_agg_transposer": os.environ.get("SOT_AGG_TRANSPOSER"),
+        # The platform_slug_map maps an arbitrary platform slug to its corresponding parser.
+        # Use this if the platform slug names in your Nautobot instance don't correspond exactly
+        # to the Nornir driver names ("arista_eos", "cisco_ios", etc.).
+        # Each key should == the slug of the Nautobot platform object.
+        # "platform_slug_map": {
+        #     "eos": "arista_eos",
+        #     "ios": "cisco_ios",
+        #     "iosxe": "cisco_ios",
+        #     "junos": "juniper_junos",
+        #     "nxos": "cisco_nxos",
+        # },
+        # "get_custom_compliance": "my.custom_compliance.func",
     },
 }
+
+# Modify django_jinja Environment for test cases
+django_jinja_config = None
+for template in TEMPLATES:
+    if template["BACKEND"].startswith("django_jinja"):
+        django_jinja_config = template
+
+if django_jinja_config is not None:
+    jinja_options = django_jinja_config.get("OPTIONS")
+    if not jinja_options:
+        jinja_options = {}
+        django_jinja_config["OPTIONS"] = jinja_options
+    # Default behavior ignores UndefinedErrors
+    jinja_options["undefined"] = "jinja2.StrictUndefined"
+
+# Import filter function to have it register filter with django_jinja
+from nautobot_golden_config.tests import jinja_filters  # noqa: E402

docs/FAQ.md

@@ -7,7 +7,10 @@ rules a subject to a given vendor's OS implementation. The ability to track thes
 this practice would be error prone and not follow the principal of least astonishment. Notwithstanding a major change in the network industry, adjusting 
 this strategy is outside the scope of the plugin. 
 
-Instead, it is up to the operator to ensure their configurations match exactly as the configurations show in the running configuration.
+Instead, the operator is required to ensure their configurations match exactly as the configurations show in the running configuration. This includes all
+spacing, special characters, or literally anything that result in a string comparison not returning true.
+
+Any other interpretations of how compliance should be viewed are encouraged to use custom compliance option.
 
 _Why doesn't the config overview page reflect the inclusion changes immediately?_
 

docs/installation.md

@@ -7,8 +7,8 @@ Plugins can be installed manually or use Python's `pip`. See the [nautobot docum
 **Prerequisite:** The plugin relies on [`nautobot_plugin_nornir`](https://pypi.org/project/nautobot-plugin-nornir/) to be installed and both plugins to be enabled in your configuration settings.
 
 **Required:** The following block of code below shows the additional configuration required to be added to your `nautobot_config.py` file:
-- append `"nautobot_golden_config"` to the `PLUGINS` list
-- append the `"nautobot_golden_config"` dictionary to the `PLUGINS_CONFIG` dictionary
+- append `"nautobot_golden_config"` to the `PLUGINS` list, and `"nautobot_plugin_nornir"` if it was not already there (More info [here](https://github.com/nautobot/nautobot-plugin-nornir))
+- append the `"nautobot_golden_config"` dictionary to the `PLUGINS_CONFIG` dictionary, and `"nautobot_plugin_nornir"` if it was not already there.
 
 ```python
 PLUGINS = ["nautobot_plugin_nornir", "nautobot_golden_config"]
@@ -35,12 +35,13 @@ PLUGINS_CONFIG = {
         "enable_sotagg": True,
         "sot_agg_transposer": None,
         "platform_slug_map": None,
+        # "get_custom_compliance": "my.custom_compliance.func"
     },
 }
 
 ```
 
-## Plugin Settings
+## Plugin Configuration
 
 The plugin behavior can be controlled with the following list of settings. 
 
@@ -54,11 +55,11 @@ The plugin behavior can be controlled with the following list of settings.
 | enable_intended | True | True | A boolean to represent whether or not to generate intended configurations within the plugin. |
 | enable_sotagg | True | True | A boolean to represent whether or not to provide a GraphQL query per device to allow the intended configuration to provide data variables to the plugin. |
 | platform_slug_map | {"cisco_wlc": "cisco_aireos"} | None | A dictionary in which the key is the platform slug and the value is what netutils uses in any "network_os" parameter. |
-| sot_agg_transposer | mypkg.transposer | - | A string representation of a function that can post-process the graphQL data. |
+| sot_agg_transposer | "mypkg.transposer" | None | A string representation of a function that can post-process the graphQL data. |
 | per_feature_bar_width | 0.15 | 0.15 | The width of the table bar within the overview report |
 | per_feature_width | 13 | 13 | The width in inches that the overview table can be. |
 | per_feature_height | 4 | 4 | The height in inches that the overview table can be. |
 
-> Note: Over time the intention is to make the compliance report more dynamic, but for now allow users to configure the `per_*` configs in a way that fits best for them.
+> Note: Over time the compliance report will become more dynamic, but for now allow users to configure the `per_*` configs in a way that fits best for them.
 
 > Note: Review [`nautobot_plugin_nornir`](https://pypi.org/project/nautobot-plugin-nornir/) for Nornir and dispatcher configuration options. 

docs/navigating-backup.md

@@ -1,13 +1,14 @@
 # Configuration Backup
 
-The backup configuration process relies on the ability for the nautobot worker to connect via Nornir to the device, run the `show run` or equivalent command 
+The backup configuration process requires the Nautobot worker to connect via Nornir to the device, and run the `show run` or equivalent command, 
 and save the configuration. The high-level process to run backups is:
 
-* Download the latest Git repository, based on `backup config` type Git repo within Nautobot.
+* Download the latest version of each of the Git repositories configured with the `backup configs` capability within Nautobot.
 * Run a Nornir play to obtain the cli configurations.
 * Optionally perform some lightweight processing of the backup.
-* Store the backup configurations locally.
-* Push configurations to the remote Git repository.
+* Store each device's backup configuration file on the local filesystem.
+* Commit all files added or changed in each repository.
+* Push configuration files to the remote Git repositories.
 
 ## Configuration Backup Settings
 
@@ -18,11 +19,16 @@ uses cases, the following settings are available and further documented below.
 * Config Removals - provides the ability to remove a line based on a regex match.
 * Config Replacements - provides the ability to swap out parts of a line based on a regex match.
 
-Backup configurations rely on a Git Repo, and the plugin registers an additional repository for Git source this access. Within the Nautobot Git 
-repositories, there will be a `backup config` option, which there must be one and only one configured for the process to work. For further details, refer 
-[to](./navigating-golden.md#git-settings).
+### Backup Repositories
 
-The `backup_path_template` provides the ability to dynamically state per device where the configurations should end up in the file structure. Every device is a Django ORM object, tied to the model instance of a `Device` model, and that is represented as `obj`. That means that any valid Device model method is available. This is then compiled via Jinja. This may seem complicated, but the equivalent of `obj` by example would be:
+In the `Backup Repository` field of the Settings, configure the repository which you intend to use for backed-up device configurations as part of Golden Config.
+
+Backup repositories must first be configured under **Extensibility -> Git Repositories**. When you configure a repository, look for the `Provides` field in the UI. To serve as a configuration backup store, the repository must be configured with the `backup configs` capability under the `Provides` field. For further details, refer to [Navigating Nautobot Git Settings](./navigating-golden.md#git-settings).
+
+
+### Backup Path Template
+
+The `backup_path_template` setting gives you a way to dynamically place each device's configuration file in the repository file structure. This setting uses the GraphQL query configured for the plugin. It works in a similar way to the Backup Repository Matching Rule above. Since the setting uses a GraphQL query, any valid Device model method is available. The plugin renders the values from the query, using Jinja2, to the relative path and file name in which to store a given device's configuration inside its backup repository. This may seem complicated, but the equivalent of `obj` by example would be:
 
 ```python
 obj = Device.objects.get(name="nyc-rt01")
@@ -33,19 +39,23 @@ An example would be:
 backup_path_template = "{{obj.site.slug}}/{{obj.name}}.cfg"
 ```
 
+With a Sydney, AU device `SYD001AURTR32`, in the site named `Sydney001` and the GraphQL query and `backup_path_template` configured above, our backed-up config would be placed in the repo in `/sydney001/SYD001AURTR32.cfg`.  The site value `sydney001` here is lower case because our template refers to the `slug` value, which by default will be lower case.
+
 The backup process will automatically create folders as required based on the path definition. 
 
 The `backup_path_template` can be set in the UI.  For navigation details [see](./navigating-golden.md#application-settings).
 
+### Device Login Credentials
+
 The credentials/secrets management is further described within the [nautbot-plugin-nornir](https://github.com/nautobot/nautobot-plugin-nornir)
-repo. For the simplist use case you can set environment variables for `NAPALM_USERNAME`, `NAPALM_PASSWORD`, and `DEVICE_SECRET`. For more
+repository. For the simplist use case you can set environment variables for `NAPALM_USERNAME`, `NAPALM_PASSWORD`, and `DEVICE_SECRET`. For more
 complicated use cases, please refer to the plugin documentation linked above.
 
 ## Starting a Backup Job
 
 To start a backup job manually:
 
-1. Navigate to the Plugin Home (Plugins->Home), with Home being in the `Golden Configuration` section
+1. Navigate to the Plugin Home (Golden Config->Home), with Home being in the `Golden Configuration` section
 2. Select _Execute_ on the upper right buttons, then _Backup_
 3. Fill in the data that you wish to have backed up
 4. Select _Run Job_
@@ -55,7 +65,7 @@ To start a backup job manually:
 The line removals settings is a series of regex patterns to identify lines that should be removed. This is helpful as there are usually parts of the
 configurations that will change each time. A match simply means to remove.
 
-In order to specify line removals. Navigate to **Plugins -> Config Removals**.  Click the **Add** button and fill out the details.
+In order to specify line removals. Navigate to **Golden Config -> Config Removals**.  Click the **Add** button and fill out the details.
 
 The remove setting is based on `Platform`.  An example is shown below.
 ![Config Removals View](./img/00-navigating-backup.png)