allow setting configuration using env variables (#600)

## Overview



Previously configuration options must be set in the local environment
file (`birdhouse/env.local` by default).
This change allows configuration options to be set as environment
variables which would take precedence over those
  set in the local environment file.

For example, you can now set the `BIRDHOUSE_FQDN` variable when starting
the stack like so:

  ```sh
  BIRDHOUSE_FQDN=myhost.example.com bin/birdhouse compose up -d

  # OR 

  export BIRDHOUSE_FQDN=myhost.example.com
  bin/birdhouse compose up -d
  ```

  This change has the following advantages:

- flexibility: the user has more options for how they can customize
their deployment
- good dev-ops: this change further aligns birdhouse with the [12 factor
app principles](https://12factor.net/),
specifically the [Config](https://12factor.net/config) principle which
recommends that configuration
                  options be settable as environment variables.
- security: sensitive settings (credentials, secrets) can be set as
environment variables, ensuring that they are
not easily visible in the plain text local environment file.
- consistency: users can store non-sensitive settings in the local
environment file and share that file freely without
                 worrying that sensitive setting will be leaked.

Note that this includes some necessary changes to the tests that are
unrelated to this change directly. The tests explicitly set
`BIRDHOUSE_BACKWARD_COMPATIBLE_ALLOWED=False` as an environment variable
in order to get around the fact that it is set to True by default when
running scripts not through a supported interface. For tests that want
to explicitly set `BIRDHOUSE_BACKWARD_COMPATIBLE_ALLOWED=True` we now
have to set that as an environment variable (not in env.local) since
environment variables now take precedence over settings in env.local.

## Changes

**Non-breaking changes**
- Adds new configuration strategy
- test refactoring

**Breaking changes**
- None

## Related Issue / Discussion

## Additional Information

## CI Operations

<!--
The test suite can be run using a different DACCS config with
``birdhouse_daccs_configs_branch: branch_name`` in the PR description.
To globally skip the test suite regardless of the commit message use
``birdhouse_skip_ci`` set to ``true`` in the PR description.

Using ``[<cmd>]`` (with the brackets) where ``<cmd> = skip ci`` in the
commit message will override ``birdhouse_skip_ci`` from the PR
description.
Such commit command can be used to override the PR description behavior
for a specific commit update.
However, a commit message cannot 'force run' a PR which the description
turns off the CI.
To run the CI, the PR should instead be updated with a ``true`` value,
and a running message can be posted in following PR comments to trigger
tests once again.
-->

birdhouse_daccs_configs_branch: master
birdhouse_skip_ci: false

Author: mishaschwartz

.bumpversion.toml

@@ -1,5 +1,5 @@
 [tool.bumpversion]
-current_version = "2.18.7"
+current_version = "2.18.8"
 commit = true
 tag = false
 tag_name = "{new_version}"

CHANGES.md

@@ -17,6 +17,40 @@
 
 [//]: # (list changes here, using '-' for each new entry, remove this when items are added)
 
+[2.18.8](https://github.com/bird-house/birdhouse-deploy/tree/2.18.8) (2025-10-30)
+------------------------------------------------------------------------------------------------------------------
+
+## Changes
+
+- Allow configuration options to be set as environment variables
+
+  Previously configuration options must be set in the local environment file (`birdhouse/env.local` by default).
+  This change allows configuration options to be set as environment variables which would take precedence over those
+  set in the local environment file.
+
+  For example, you can now set the `BIRDHOUSE_FQDN` variable when starting the stack like so:
+
+  ```sh
+  BIRDHOUSE_FQDN=myhost.example.com bin/birdhouse compose up -d
+
+  # OR 
+
+  export BIRDHOUSE_FQDN=myhost.example.com
+  bin/birdhouse compose up -d
+  ```
+
+  This change has the following advantages:
+
+  - flexibility: the user has more options for how they can customize their deployment
+  - good dev-ops: this change further aligns birdhouse with the [12 factor app principles](https://12factor.net/), 
+                  specifically the [Config](https://12factor.net/config) principle which recommends that configuration 
+                  options be settable as environment variables.
+  - security: sensitive settings (credentials, secrets) can be set as environment variables, ensuring that they are 
+              not easily visible in the plain text local environment file.
+  - consistency: users can store non-sensitive settings in the local environment file and share that file freely without
+                 worrying that sensitive setting will be leaked.
+
+
 [2.18.7](https://github.com/bird-house/birdhouse-deploy/tree/2.18.7) (2025-10-17)
 ------------------------------------------------------------------------------------------------------------------
 

Makefile

@@ -8,7 +8,7 @@ override BIRDHOUSE_MAKE_DIR := $(shell realpath -P $$(dirname $(BIRDHOUSE_MAKE_C
 # Generic variables
 override SHELL       := bash
 override APP_NAME    := birdhouse-deploy
-override APP_VERSION := 2.18.7
+override APP_VERSION := 2.18.8
 
 # utility to remove comments after value of an option variable
 override clean_opt = $(shell echo "$(1)" | $(_SED) -r -e "s/[ '$'\t'']+$$//g")

README.rst

@@ -18,13 +18,13 @@ for a full-fledged production platform.
     * - citation
       - | |citation|
 
-.. |commits-since| image:: https://img.shields.io/github/commits-since/bird-house/birdhouse-deploy/2.18.7.svg
+.. |commits-since| image:: https://img.shields.io/github/commits-since/bird-house/birdhouse-deploy/2.18.8.svg
     :alt: Commits since latest release
-    :target: https://github.com/bird-house/birdhouse-deploy/compare/2.18.7...master
+    :target: https://github.com/bird-house/birdhouse-deploy/compare/2.18.8...master
 
-.. |latest-version| image:: https://img.shields.io/badge/tag-2.18.7-blue.svg?style=flat
+.. |latest-version| image:: https://img.shields.io/badge/tag-2.18.8-blue.svg?style=flat
     :alt: Latest Tag
-    :target: https://github.com/bird-house/birdhouse-deploy/tree/2.18.7
+    :target: https://github.com/bird-house/birdhouse-deploy/tree/2.18.8
 
 .. |readthedocs| image:: https://readthedocs.org/projects/birdhouse-deploy/badge/?version=latest
     :alt: ReadTheDocs Build Status (latest version)

RELEASE.txt

@@ -1 +1 @@
-2.18.7 2025-10-17T16:11:17Z
+2.18.8 2025-10-30T19:03:01Z

birdhouse/README.rst

@@ -106,6 +106,9 @@ This will source the ``env.local`` file, apply the appropriate variable substitu
 ".template", and run ``docker-compose`` with all the command line arguments after the ``compose`` argument.
 See `env.local.example <env.local.example>`_ (:download:`download </birdhouse/env.local.example>`) for more details on what can go into the ``env.local`` file.
 
+Most variables that can be set in the local environment file (``env.local`` by default) can also be specified as environment variables when running ``bin/birdhouse`` 
+commands. Environment variables will take precedence over those specified in the ``env.local`` file.
+
 If the file `env.local` is somewhere else, symlink it here, next to `docker-compose.yml <docker-compose.yml>`_ (:download:`download </birdhouse/docker-compose.yml>`) because many scripts assume this location.
 If autodeploy scheduler job is enabled, the folder containing the `env.local` file needs to be added to `BIRDHOUSE_AUTODEPLOY_EXTRA_REPOS`.
 

birdhouse/components/canarie-api/docker_configuration.py.template

@@ -108,8 +108,8 @@ SERVICES = {
             # NOTE:
             #   Below version and release time auto-managed by 'make VERSION=x.y.z bump'.
             #   Do NOT modify it manually. See 'Tagging policy' in 'birdhouse/README.rst'.
-            'version': '2.18.7',
-            'releaseTime': '2025-10-17T16:11:17Z',
+            'version': '2.18.8',
+            'releaseTime': '2025-10-30T19:03:01Z',
             'institution': '${BIRDHOUSE_INSTITUTION}',
             'researchSubject': '${BIRDHOUSE_SUBJECT}',
             'supportEmail': '${BIRDHOUSE_SUPPORT_EMAIL}',
@@ -141,8 +141,8 @@ PLATFORMS = {
             # NOTE:
             #   Below version and release time auto-managed by 'make VERSION=x.y.z bump'.
             #   Do NOT modify it manually. See 'Tagging policy' in 'birdhouse/README.rst'.
-            'version': '2.18.7',
-            'releaseTime': '2025-10-17T16:11:17Z',
+            'version': '2.18.8',
+            'releaseTime': '2025-10-30T19:03:01Z',
             'institution': '${BIRDHOUSE_INSTITUTION}',
             'researchSubject': '${BIRDHOUSE_SUBJECT}',
             'supportEmail': '${BIRDHOUSE_SUPPORT_EMAIL}',

birdhouse/read-configs.include.sh

@@ -459,9 +459,36 @@ set_backwards_compatible_as_default() {
   fi
 }
 
+# Record all currently exported environment variables and store them in the __BIRDHOUSE_PROCESS_ENV
+# variable. Calling reset_process_env later will re-export all these variables, setting them back to their
+# original value. See reset_process_env for an example.
+stage_process_env() {
+  __BIRDHOUSE_PROCESS_ENV="$(export -p)"
+}
+
+# Re-export and set all environment variables that were exported when stage_process_env was last called. This 
+# effectively resets these variables to the values that they had when stage_process_env was last called.
+#
+# Note that this will not unset variables that were not exported when stage_process_env was last called.
+#
+# For example:
+# 
+# export X=10
+# stage_process_env
+# echo $X  # prints 10
+# X=11
+# export Y=12
+# echo $X  # prints 11
+# reset_process_env
+# echo $X  # prints 10
+# echo $Y  # prints 12
+reset_process_env() {
+  eval "${__BIRDHOUSE_PROCESS_ENV}"
+}
 
 ### Similar code between read_configs() and read_basic_configs_only().
 _read_basic_configs_pre() {
+    stage_process_env
     set_backwards_compatible_as_default
     discover_compose_dir
     discover_env_local
@@ -470,8 +497,9 @@ _read_basic_configs_pre() {
 
 
 _read_basic_configs_post() {
-    read_env_local  # again to override components default.env, need discover_env_local
-    set_old_backwards_compatible_variables  # after read_env_local to use updated value and not default value
+    read_env_local  # override default env (for a second time if called from read_configs), needs discover_env_local to run first
+    reset_process_env  # override local env and default env with variables declared in the calling process' environment, needs stage_process_env to run first
+    set_old_backwards_compatible_variables  # after read_env_local to use updated value and not default value for backwards compatible variables
     process_backwards_compatible_variables
     check_default_vars
     process_delayed_eval
@@ -485,7 +513,8 @@ read_configs() {
     _read_basic_configs_pre
 
     ### This section is different than read_basic_configs_only() below, the rest should be IDENTICAL.
-    read_env_local  # for BIRDHOUSE_EXTRA_CONF_DIRS and BIRDHOUSE_DEFAULT_CONF_DIRS, need discover_env_local
+    read_env_local  # for BIRDHOUSE_EXTRA_CONF_DIRS and BIRDHOUSE_DEFAULT_CONF_DIRS, needs discover_env_local to run first
+    reset_process_env  # use BIRDHOUSE_EXTRA_CONF_DIRS and BIRDHOUSE_DEFAULT_CONF_DIRS if they're declared in the calling process' environment, needs stage_process_env to run first
     process_backwards_compatible_variables pre-components
     read_components_default_env  # uses BIRDHOUSE_EXTRA_CONF_DIRS and BIRDHOUSE_DEFAULT_CONF_DIRS, sets ALL_CONF_DIRS
     ### END This section is different than read_basic_configs_only() below, the rest should be IDENTICAL.

docs/source/conf.py

@@ -69,9 +69,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '2.18.7'
+version = '2.18.8'
 # The full version, including alpha/beta/rc tags.
-release = '2.18.7'
+release = '2.18.8'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.