Fix a few recent regressions to Autodeploy (#605)

## Overview

## Changes

- README: remind the user to source control `env.local` securely as it
may contains passwords.

## Fixes

- Autodeploy broken due to new config variable precedence order.

Broken since `2.18.8`
(#600).

  Example breakage scenario:

  Autodeploy starts and calls `read_configs`, then git pull a newer
  version of `env.local` (which might modify `EXTRA_CONF_DIRS` to enable
  new components for example).  But since the new var precedence locked
  all the values to the first call of `read_configs`, the new change to
  `EXTRA_CONF_DIRS` (or any vars in `env.local`) never went into
  effect at the end of the autodeploy process when we start the full
  stack.

  This is because `read_configs` returns all variables as environment
  variables and the new variable precedence order gives environment
  variables precedence higher than the values in `env.local`.  So after
the first call to `read_configs` (or `read_basic_configs_only`), all the
  values are "locked" for all child processes even if they try to call
  `read_configs` again to get the newest values from `env.local`.

  The fix is to avoid getting these config variables as environment
  variables which will avoid "locking" their values for child processes.

- Autodeploy broken due to attempt to be compat with changing the
location of `env.local` via `BIRDHOUSE_LOCAL_ENV`.

Broken since `2.18.9`
(#601).

It was working for the first run of autodeploy but on the next run, the
value of
`BIRDHOUSE_LOCAL_ENV` in
`optional-components/scheduler-job-autodeploy/config.yml`
  is wrong !

  Good value in first run:

  `--volume /real/path/to/env.local:/tmp/birdhouse-local-env`

  Bad value in subsequent run:

  `--volume /tmp/birdhouse-local-env:/tmp/birdhouse-local-env`

This is because we override `--env
BIRDHOUSE_LOCAL_ENV=/tmp/birdhouse-local-env` in
`optional-components/scheduler-job-autodeploy/config.yml.template` as
env var so on
  the subsequent runs the bad value persists because it is an env var.

This `BIRDHOUSE_LOCAL_ENV` is meant to be set "outside" the stack as env
var
so there is nothing in the stack that will fix this value if it is bad.

The fix is to revert that attempt. So autodeploy only supports changing
the
  location of `env.local` via symlink, or by manually adding the folder
containing the file `env.local` to `BIRDHOUSE_AUTODEPLOY_EXTRA_REPOS` if
`env.local` is set by `BIRDHOUSE_LOCAL_ENV` environment variable and not
via
  symlink.

- One feature of autodeploy is broken due to a missing mapping in
back-compat config var name change.

Broken since `2.4.0` but only if back-compat var name mode is required.

The `AUTODEPLOY_CODE_OWNERSHIP` old name mapping is missing in
`BIRDHOUSE_BACKWARDS_COMPATIBLE_VARIABLES`.

Autodeploy is missing the `chown` step to the good code owner because it
does
not "see" the value of the old variable name since the mapping to the
new
  variable name is missing.

- Missing default value for `BIRDHOUSE_HTTP_ONLY` causing inconsistent
behavior
when the var is used in `env.local` then removed from `env.local`. The
  behavior never revert back as if it has never been initially used in
`env.local`. See
[discussion](#600 (comment)).

## CI Operations

<!--
The test suite can be run using a different DACCS config with
``birdhouse_daccs_configs_branch: branch_name`` in the PR description.
To globally skip the test suite regardless of the commit message use
``birdhouse_skip_ci`` set to ``true`` in the PR description.

Using ``[<cmd>]`` (with the brackets) where ``<cmd> = skip ci`` in the
commit message will override ``birdhouse_skip_ci`` from the PR
description.
Such commit command can be used to override the PR description behavior
for a specific commit update.
However, a commit message cannot 'force run' a PR which the description
turns off the CI.
To run the CI, the PR should instead be updated with a ``true`` value,
and a running message can be posted in following PR comments to trigger
tests once again.
-->

birdhouse_daccs_configs_branch: master
birdhouse_skip_ci: false

Author: tlvu

.bumpversion.toml

@@ -1,5 +1,5 @@
 [tool.bumpversion]
-current_version = "2.18.14"
+current_version = "2.18.15"
 commit = true
 tag = false
 tag_name = "{new_version}"

CHANGES.md

@@ -17,6 +17,84 @@
 
 [//]: # (list changes here, using '-' for each new entry, remove this when items are added)
 
+[2.18.15](https://github.com/bird-house/birdhouse-deploy/tree/2.18.15) (2025-12-01)
+------------------------------------------------------------------------------------------------------------------
+
+## Changes
+
+- README: remind the user to source control `env.local` securely as it may contains passwords.
+
+## Fixes
+
+- Autodeploy broken due to new config variable precedence order.
+
+  Broken since `2.18.8` (https://github.com/bird-house/birdhouse-deploy/pull/600).
+
+  Example breakage scenario:
+
+  Autodeploy starts and calls `read_configs`, then git pull a newer
+  version of `env.local` (which might modify `EXTRA_CONF_DIRS` to enable
+  new components for example).  But since the new var precedence locked
+  all the values to the first call of `read_configs`, the new change to
+  `EXTRA_CONF_DIRS` (or any vars in `env.local`) never went into
+  effect at the end of the autodeploy process when we start the full
+  stack.
+
+  This is because `read_configs` returns all variables as environment
+  variables and the new variable precedence order gives environment
+  variables precedence higher than the values in `env.local`.  So after
+  the first call to `read_configs` (or `read_basic_configs_only`), all the
+  values are "locked" for all child processes even if they try to call
+  `read_configs` again to get the newest values from `env.local`.
+
+  The fix is to avoid getting these config variables as environment
+  variables which will avoid "locking" their values for child processes.
+
+- Autodeploy broken due to attempt to be compat with changing the location of `env.local` via `BIRDHOUSE_LOCAL_ENV`.
+
+  Broken since `2.18.9` (https://github.com/bird-house/birdhouse-deploy/pull/601).
+
+  It was working for the first run of autodeploy but on the next run, the value of
+  `BIRDHOUSE_LOCAL_ENV` in `optional-components/scheduler-job-autodeploy/config.yml`
+  is wrong !
+
+  Good value in first run:
+
+  `--volume /real/path/to/env.local:/tmp/birdhouse-local-env`
+
+  Bad value in subsequent run:
+
+  `--volume /tmp/birdhouse-local-env:/tmp/birdhouse-local-env`
+
+  This is because we override `--env BIRDHOUSE_LOCAL_ENV=/tmp/birdhouse-local-env` in
+  `optional-components/scheduler-job-autodeploy/config.yml.template` as env var so on
+  the subsequent runs the bad value persists because it is an env var.
+
+  This `BIRDHOUSE_LOCAL_ENV` is meant to be set "outside" the stack as env var
+  so there is nothing in the stack that will fix this value if it is bad.
+
+  The fix is to revert that attempt.  So autodeploy only supports changing the
+  location of `env.local` via symlink, or by manually adding the folder
+  containing the file `env.local` to `BIRDHOUSE_AUTODEPLOY_EXTRA_REPOS` if
+  `env.local` is set by `BIRDHOUSE_LOCAL_ENV` environment variable and not via
+  symlink.
+
+- One feature of autodeploy is broken due to a missing mapping in back-compat config var name change.
+
+  Broken since `2.4.0` but only if back-compat var name mode is required.
+
+  The `AUTODEPLOY_CODE_OWNERSHIP` old name mapping is missing in `BIRDHOUSE_BACKWARDS_COMPATIBLE_VARIABLES`.
+
+  Autodeploy is missing the `chown` step to the good code owner because it does
+  not "see" the value of the old variable name since the mapping to the new
+  variable name is missing.
+
+- Missing default value for `BIRDHOUSE_HTTP_ONLY` causing inconsistent behavior
+  when the var is used in `env.local` then removed from `env.local`.  The
+  behavior never revert back as if it has never been initially used in
+  `env.local`.  See [discussion](https://github.com/bird-house/birdhouse-deploy/pull/600#issuecomment-3528058708).
+
+
 [2.18.14](https://github.com/bird-house/birdhouse-deploy/tree/2.18.14) (2025-12-01)
 ------------------------------------------------------------------------------------------------------------------
 

Makefile

@@ -8,7 +8,7 @@ override BIRDHOUSE_MAKE_DIR := $(shell realpath -P $$(dirname $(BIRDHOUSE_MAKE_C
 # Generic variables
 override SHELL       := bash
 override APP_NAME    := birdhouse-deploy
-override APP_VERSION := 2.18.14
+override APP_VERSION := 2.18.15
 
 # utility to remove comments after value of an option variable
 override clean_opt = $(shell echo "$(1)" | $(_SED) -r -e "s/[ '$'\t'']+$$//g")

README.rst

@@ -18,13 +18,13 @@ for a full-fledged production platform.
     * - citation
       - | |citation|
 
-.. |commits-since| image:: https://img.shields.io/github/commits-since/bird-house/birdhouse-deploy/2.18.14.svg
+.. |commits-since| image:: https://img.shields.io/github/commits-since/bird-house/birdhouse-deploy/2.18.15.svg
     :alt: Commits since latest release
-    :target: https://github.com/bird-house/birdhouse-deploy/compare/2.18.14...master
+    :target: https://github.com/bird-house/birdhouse-deploy/compare/2.18.15...master
 
-.. |latest-version| image:: https://img.shields.io/badge/tag-2.18.14-blue.svg?style=flat
+.. |latest-version| image:: https://img.shields.io/badge/tag-2.18.15-blue.svg?style=flat
     :alt: Latest Tag
-    :target: https://github.com/bird-house/birdhouse-deploy/tree/2.18.14
+    :target: https://github.com/bird-house/birdhouse-deploy/tree/2.18.15
 
 .. |readthedocs| image:: https://readthedocs.org/projects/birdhouse-deploy/badge/?version=latest
     :alt: ReadTheDocs Build Status (latest version)

RELEASE.txt

@@ -1 +1 @@
-2.18.14 2025-12-01T15:42:49Z
+2.18.15 2025-12-01T18:03:36Z

birdhouse/README.rst

@@ -114,7 +114,11 @@ If autodeploy scheduler job is enabled, the folder containing the `env.local` fi
 
 To follow infrastructure-as-code, it is encouraged to source control the above
 `env.local` file and any override needed to customized this Birdhouse deployment
-for your organization.  For an example of possible override, see how the `emu service <optional-components/emu/docker-compose-extra.yml>`_ (:download:`download </birdhouse/optional-components/emu/docker-compose-extra.yml>`)
+for your organization.  Note this `env.local` file might contains **sensitive**
+infos like passwords so it should be in a limitted access private source control
+repo, idealy not on the internet.
+
+For an example of possible override, see how the `emu service <optional-components/emu/docker-compose-extra.yml>`_ (:download:`download </birdhouse/optional-components/emu/docker-compose-extra.yml>`)
 (`README <optional-components/README.rst#emu-wps-service-for-testing>`_) can be optionally added to the deployment via the `override mechanism <https://docs.docker.com/compose/extends/>`_.
 Ouranos specific override can be found in this `birdhouse-deploy-ouranos <https://github.com/bird-house/birdhouse-deploy-ouranos>`_ repo.
 

birdhouse/components/canarie-api/docker_configuration.py.template

@@ -108,8 +108,8 @@ SERVICES = {
             # NOTE:
             #   Below version and release time auto-managed by 'make VERSION=x.y.z bump'.
             #   Do NOT modify it manually. See 'Tagging policy' in 'birdhouse/README.rst'.
-            'version': '2.18.14',
-            'releaseTime': '2025-12-01T15:42:49Z',
+            'version': '2.18.15',
+            'releaseTime': '2025-12-01T18:03:36Z',
             'institution': '${BIRDHOUSE_INSTITUTION}',
             'researchSubject': '${BIRDHOUSE_SUBJECT}',
             'supportEmail': '${BIRDHOUSE_SUPPORT_EMAIL}',
@@ -141,8 +141,8 @@ PLATFORMS = {
             # NOTE:
             #   Below version and release time auto-managed by 'make VERSION=x.y.z bump'.
             #   Do NOT modify it manually. See 'Tagging policy' in 'birdhouse/README.rst'.
-            'version': '2.18.14',
-            'releaseTime': '2025-12-01T15:42:49Z',
+            'version': '2.18.15',
+            'releaseTime': '2025-12-01T18:03:36Z',
             'institution': '${BIRDHOUSE_INSTITUTION}',
             'researchSubject': '${BIRDHOUSE_SUBJECT}',
             'supportEmail': '${BIRDHOUSE_SUPPORT_EMAIL}',

birdhouse/components/proxy/default.env

@@ -6,6 +6,7 @@ export PROXY_IMAGE="nginx:1.23.4"
 # Any WPS processes taking longer than this should use async mode.
 export PROXY_READ_TIMEOUT_VALUE="240s"
 
+export BIRDHOUSE_HTTP_ONLY="False"
 export BIRDHOUSE_PROXY_SCHEME='$([ x"$BIRDHOUSE_HTTP_ONLY" = x"True" ] && echo http || echo https )'
 export BIRDHOUSE_ALLOW_UNSECURE_HTTP='$([ x"$BIRDHOUSE_HTTP_ONLY" = x"True" ] && echo True || echo )'
 export PROXY_INCLUDE_HTTPS='$([ x"$BIRDHOUSE_HTTP_ONLY" = x"True" ] || echo "include /etc/nginx/conf.d/https.include;" )'

birdhouse/default.env

@@ -215,6 +215,7 @@ BIRDHOUSE_BACKWARDS_COMPATIBLE_VARIABLES="
     AUTODEPLOY_PLATFORM_FREQUENCY=BIRDHOUSE_AUTODEPLOY_PLATFORM_FREQUENCY
     AUTODEPLOY_NOTEBOOK_FREQUENCY=BIRDHOUSE_AUTODEPLOY_NOTEBOOK_FREQUENCY
     AUTODEPLOY_EXTRA_SCHEDULER_JOBS=BIRDHOUSE_AUTODEPLOY_EXTRA_SCHEDULER_JOBS
+    AUTODEPLOY_CODE_OWNERSHIP=BIRDHOUSE_AUTODEPLOY_CODE_OWNERSHIP
     LOGROTATE_DATA_DIR=BIRDHOUSE_LOGROTATE_DATA_DIR
     ALLOW_UNSECURE_HTTP=BIRDHOUSE_ALLOW_UNSECURE_HTTP
     DOCKER_NOTEBOOK_IMAGES=JUPYTERHUB_DOCKER_NOTEBOOK_IMAGES

birdhouse/deployment/deploy.sh

@@ -74,10 +74,24 @@ BIRDHOUSE_LOCAL_ENV="${1:-${BIRDHOUSE_LOCAL_ENV:-"${COMPOSE_DIR}/env.local"}}"
 # container and manually from the host.
 cd "${COMPOSE_DIR}" || exit
 
-. "${COMPOSE_DIR}/read-configs.include.sh"
-
-# Read BIRDHOUSE_AUTODEPLOY_EXTRA_REPOS
-read_basic_configs_only
+set -x
+BIRDHOUSE_EXE="${COMPOSE_DIR}/../bin/birdhouse"
+BIRDHOUSE_EXE_CONFIGS_OPTS="--backwards-compatible --quiet configs --basic --command"
+
+# Similar to "read_basic_configs_only" but only get the vars we will need, not
+# everything like the full "read_basic_configs_only".
+# Note: NEVER turn these vars into env var because this will "lock" these
+# vars for all subsequent "read_configs" even if the env.local changes.  This
+# is because of the new variable precedence that gives higher precedence to env
+# var than env.local.  This will break the last step when we bring up the stack
+# that will have to read and use all the latest values of any changed vars.
+CONFIG_VARS_TO_EVAL="$("${BIRDHOUSE_EXE}" ${BIRDHOUSE_EXE_CONFIGS_OPTS} '
+    echo "BIRDHOUSE_LOG_DIR=\"${BIRDHOUSE_LOG_DIR}\"";
+    echo "BIRDHOUSE_AUTODEPLOY_EXTRA_REPOS=\"${BIRDHOUSE_AUTODEPLOY_EXTRA_REPOS}\"";
+    echo "BIRDHOUSE_AUTODEPLOY_DEPLOY_KEY_ROOT_DIR=\"${BIRDHOUSE_AUTODEPLOY_DEPLOY_KEY_ROOT_DIR}\"";
+')"
+eval "${CONFIG_VARS_TO_EVAL}"
+set +x
 
 if [ ! -z "${AUTODEPLOY_SILENT}" ]; then
     LOG_FILE="${BIRDHOUSE_LOG_DIR}/autodeploy.log"
@@ -132,10 +146,6 @@ done
 
 cd "${COMPOSE_DIR}" || exit
 
-set +x  # avoid excessive forced logging
-read_basic_configs_only
-set -x
-
 # stop all to force reload any changed config that are volume-mount into the containers
 "${BIRDHOUSE_COMPOSE}" stop
 
@@ -176,14 +186,10 @@ done
 
 cd "${COMPOSE_DIR}" || exit
 
-# reload again after git pull because this file could be changed by the pull
-. "${COMPOSE_DIR}/read-configs.include.sh"
-
-set +x  # avoid excessive forced logging
-# reload again after default.env since env.local can override default.env
-# (ex: JUPYTERHUB_USER_DATA_DIR)
-read_basic_configs_only
-set -x
+# NOTE: if other operations are needed at this point, re-read again the config
+# vars because both the code and the env.local could have changed after git pull.
+# And NEVER turn these vars into env var, see explanation at the first config
+# var read earlier in this file.
 
 # restart everything, only changed containers will be destroyed and recreated
 "${BIRDHOUSE_COMPOSE}" up -d