Environment Variable Resolver (#254)

* Functionality for env resolver and crypto functionality. also added all currently installed packages to the info dump (in comments) such that a minimal python env should be able to be re-constructed

* added unit tests. cleaned up bugs found when creating unit tests

* added docs for resolvers

* updated README

Author: ncilfone

.github/ISSUE_TEMPLATE/bug_report.md

@@ -23,16 +23,10 @@ A clear and concise description of what you expected to happen.
 **Screenshots**
 If applicable, add screenshots to help explain your problem.
 
-**Desktop (please complete the following information):**
- - OS: [e.g. iOS]
- - Browser [e.g. chrome, safari]
- - Version [e.g. 22]
-
-**Smartphone (please complete the following information):**
- - Device: [e.g. iPhone6]
- - OS: [e.g. iOS8.1]
- - Browser [e.g. stock browser, safari]
- - Version [e.g. 22]
+**Environment (please complete the following information):**
+- OS: [e.g. Ubuntu 16.04, Windows, etc.]
+- Python version [e.g. 3.6.3, 3.7.5]:
+- Other installed packages [e.g. torch, scipy, etc.]
 
 **Additional context**
 Add any other context about the problem here.

.github/pull_request_template.md

@@ -12,7 +12,8 @@ E.g. Describe the added feature or what issue it fixes #(issue)...
   - [ ] Did you run black and isort prior to submitting your PR? 
   - [ ] Does your PR pass all existing unit tests?
   - [ ] Did you add associated unit tests for any additional functionality?
-  - [ ] Did you provide documentation ([Google Docstring format](https://google.github.io/styleguide/pyguide.html)) whenever possible, even for simple functions or classes?
+  - [ ] Did you provide code documentation ([Google Docstring format](https://google.github.io/styleguide/pyguide.html)) whenever possible, even for simple functions or classes?
+  - [ ] Did you add necessary documentation to the website?
 
 ## Review
 Request will go to reviewers to approve for merge.

NOTICE.txt

@@ -11,9 +11,11 @@ FMR LLC (https://www.fidelity.com/).
 
 This product relies on the following works (and the dependencies thereof), installed separately:
 - attrs | https://github.com/python-attrs/attrs | MIT License
+- cryptography | https://github.com/pyca/cryptography | Apache License 2.0 + BSD License
 - GitPython | https://github.com/gitpython-developers/GitPython | BSD 3-Clause License
 - pytomlpp | https://github.com/bobfang1992/pytomlpp | MIT License
 - PyYAML | https://github.com/yaml/pyyaml | MIT License
+- setuptools | https://github.com/pypa/setuptools | MIT License
 
 
 Optional extensions rely on the following works (and the dependencies thereof), installed separately:

README.md

@@ -52,6 +52,8 @@ generating CLI arguments, and hierarchical configuration by composition.
 * Automatic type checked CLI generation w/o argparser boilerplate (i.e click and/or typer for free!)
 * Easily maintain parity between CLIs and Python APIs (i.e. single line changes between CLI and Python API definitions)
 * Unified hyper-parameter definitions and interface (i.e. don't write different definitions for Ax or Optuna)
+* Resolver that supports value definitions from environmental variables, dynamic template re-injection, and 
+encryption of sensitive values
 
 ## Key Features
 
@@ -101,6 +103,13 @@ See [Releases](https://github.com/fidelity/spock/releases) for more information.
 
 <details>
 
+#### May 17th, 2022
+* Added support for resolving value definitions from environmental variables with the following syntax, 
+`${spock.env:name, default}`
+* Added `.inject` annotation that will write back the original env notation to the saved output
+* Added the `.crypto` annotation which provides a simple way to hide sensitive environmental
+variables while still maintaining the written/loadable state of the spock config
+
 #### March 17th, 2022
 * Added support for `typing.Callable` types (includes advanced types such as `List[List[Callable]]`)
 * Added support for `typing.Dict` types with type checking for types of both keys and values (includes advanced types

REQUIREMENTS.txt

@@ -1,4 +1,6 @@
 attrs~=21.4
+cryptography~=37.0
 GitPython~=3.1
 pytomlpp~=1.0
 pyYAML~=5.4
+setuptools~=59.6

spock/addons/tune/payload.py

@@ -5,8 +5,10 @@
 
 """Handles the tuner payload backend"""
 
+from typing import Optional
+
 from spock.backend.payload import BasePayload
-from spock.backend.utils import get_attr_fields
+from spock.backend.utils import _T, get_attr_fields
 
 
 class TunerPayload(BasePayload):
@@ -20,7 +22,7 @@ class TunerPayload(BasePayload):
 
     """
 
-    def __init__(self, s3_config=None):
+    def __init__(self, s3_config: Optional[_T] = None):
         """Init for TunerPayload
 
         Args:

spock/backend/builder.py

@@ -7,7 +7,7 @@
 import argparse
 from abc import ABC, abstractmethod
 from enum import EnumMeta
-from typing import Dict, List
+from typing import ByteString, Dict, List
 
 import attr
 
@@ -34,22 +34,36 @@ class BaseBuilder(ABC):  # pylint: disable=too-few-public-methods
         _module_name: module name to register in the spock module space
         save_path: list of path(s) to save the configs to
         _lazy: attempts to lazily find @spock decorated classes registered within sys.modules["spock"].backend.config
+        _salt: salt use for crypto purposes
+        _key: key used for crypto purposes
 
     """
 
     def __init__(
-        self, *args, max_indent: int = 4, module_name: str, lazy: bool, **kwargs
+        self,
+        *args,
+        max_indent: int = 4,
+        module_name: str,
+        lazy: bool,
+        salt: str,
+        key: ByteString,
+        **kwargs,
     ):
         """Init call for BaseBuilder
 
         Args:
             *args: iterable of @spock decorated classes
             max_indent: max indent for pretty print of help
             module_name: module name to register in the spock module space
+            lazy: lazily find @spock decorated classes
+            salt: cryptographic salt
+            key: cryptographic key
             **kwargs: keyword args
         """
         self._input_classes = args
         self._lazy = lazy
+        self._salt = salt
+        self._key = key
         self._graph = Graph(input_classes=self.input_classes, lazy=self._lazy)
         # Make sure the input classes are updated -- lazy evaluation
         self._input_classes = self._graph.nodes
@@ -144,7 +158,7 @@ def resolve_spock_space_kwargs(self, graph: Graph, dict_args: Dict) -> Dict:
         for spock_cls in graph.roots:
             # Initial call to the RegisterSpockCls generate function (which will handle recursing if needed)
             spock_instance, special_keys = RegisterSpockCls.recurse_generate(
-                spock_cls, builder_space
+                spock_cls, builder_space, self._salt, self._key
             )
             builder_space.spock_space[spock_cls.__name__] = spock_instance