Ensure Jupyterlab container healthchecks don't fail because of leftover files (#607)

## Overview

Leftover files from Jupyterlab containers that didn't shut down properly
could cause the healthcheck to always fail. Old files are now removed
when the container starts up so this is avoided.

The full explanation:

If a Jupyterlab container is not shut down properly then it does not
have a chance to [clean
up](https://github.com/jupyter-server/jupyter_server/blob/dd435c/jupyter_server/serverapp.py#L3152)
old runtime files. These runtime files are stored in the user's home
directory which is persisted between containers as a bind-mount.

When the Jupyterlab container is created again, it will create new
versions of some of these files. One of which is the jpserver JSON files
which contains [a URL used by the
healthchecks](https://github.com/jupyter/docker-stacks/blob/3b8a79/images/base-notebook/docker_healthcheck.py#L27).

This URL contains the hostname of the Jupyterlab container which is
different each time a new container is created. Different files will
have different hostnames, only one of which will be the hostname for the
current container.

Since the
[healthchecks](https://github.com/jupyter/docker-stacks/blob/3b8a79/images/base-notebook/docker_healthcheck.py#L27)
use `glob` which returns files in an arbitrary order, the healthcheck
can easily choose an old file with an old hostname. If that is the case,
the healthcheck will always fail.

## Changes

**Non-breaking changes**
- bugfix

**Breaking changes**
- None

## Related Issue / Discussion

## Additional Information

## CI Operations

<!--
The test suite can be run using a different DACCS config with
``birdhouse_daccs_configs_branch: branch_name`` in the PR description.
To globally skip the test suite regardless of the commit message use
``birdhouse_skip_ci`` set to ``true`` in the PR description.

Using ``[<cmd>]`` (with the brackets) where ``<cmd> = skip ci`` in the
commit message will override ``birdhouse_skip_ci`` from the PR
description.
Such commit command can be used to override the PR description behavior
for a specific commit update.
However, a commit message cannot 'force run' a PR which the description
turns off the CI.
To run the CI, the PR should instead be updated with a ``true`` value,
and a running message can be posted in following PR comments to trigger
tests once again.
-->

birdhouse_daccs_configs_branch: master
birdhouse_skip_ci: false

Author: mishaschwartz

.bumpversion.toml

@@ -1,5 +1,5 @@
 [tool.bumpversion]
-current_version = "2.18.13"
+current_version = "2.18.14"
 commit = true
 tag = false
 tag_name = "{new_version}"

CHANGES.md

@@ -17,6 +17,35 @@
 
 [//]: # (list changes here, using '-' for each new entry, remove this when items are added)
 
+[2.18.14](https://github.com/bird-house/birdhouse-deploy/tree/2.18.14) (2025-12-01)
+------------------------------------------------------------------------------------------------------------------
+
+## Fixes
+
+- Ensure Jupyterlab container healthchecks don't fail because of leftover files
+
+  Leftover files from Jupyterlab containers that didn't shut down properly could cause the
+  healthcheck to always fail. Old files are now removed when the container starts up so this is avoided.
+
+  The full explanation:
+
+  If a Jupyterlab container is not shut down properly then it does not have a chance to 
+  [clean up](https://github.com/jupyter-server/jupyter_server/blob/dd435c/jupyter_server/serverapp.py#L3152) 
+  old runtime files. These runtime files are stored in the user's home directory which is
+  persisted between containers as a bind-mount.
+
+  When the Jupyterlab container is created again, it will create new versions of some of these files. One of which
+  is the jpserver JSON files which contains [a URL used by the 
+  healthchecks](https://github.com/jupyter/docker-stacks/blob/3b8a79/images/base-notebook/docker_healthcheck.py#L27).
+  
+  This URL contains the hostname of the Jupyterlab container which is different each time a new container is created. 
+  Different files will have different hostnames, only one of which will be the hostname for the current container.
+
+  Since the [healthchecks](https://github.com/jupyter/docker-stacks/blob/3b8a79/images/base-notebook/docker_healthcheck.py#L27)
+  use `glob` which returns files in an arbitrary order, the healthcheck can easily choose an old file with an old hostname.
+  If that is the case, the healthcheck will always fail.
+
+
 [2.18.13](https://github.com/bird-house/birdhouse-deploy/tree/2.18.13) (2025-12-01)
 ------------------------------------------------------------------------------------------------------------------
 

Makefile

@@ -8,7 +8,7 @@ override BIRDHOUSE_MAKE_DIR := $(shell realpath -P $$(dirname $(BIRDHOUSE_MAKE_C
 # Generic variables
 override SHELL       := bash
 override APP_NAME    := birdhouse-deploy
-override APP_VERSION := 2.18.13
+override APP_VERSION := 2.18.14
 
 # utility to remove comments after value of an option variable
 override clean_opt = $(shell echo "$(1)" | $(_SED) -r -e "s/[ '$'\t'']+$$//g")

README.rst

@@ -18,13 +18,13 @@ for a full-fledged production platform.
     * - citation
       - | |citation|
 
-.. |commits-since| image:: https://img.shields.io/github/commits-since/bird-house/birdhouse-deploy/2.18.13.svg
+.. |commits-since| image:: https://img.shields.io/github/commits-since/bird-house/birdhouse-deploy/2.18.14.svg
     :alt: Commits since latest release
-    :target: https://github.com/bird-house/birdhouse-deploy/compare/2.18.13...master
+    :target: https://github.com/bird-house/birdhouse-deploy/compare/2.18.14...master
 
-.. |latest-version| image:: https://img.shields.io/badge/tag-2.18.13-blue.svg?style=flat
+.. |latest-version| image:: https://img.shields.io/badge/tag-2.18.14-blue.svg?style=flat
     :alt: Latest Tag
-    :target: https://github.com/bird-house/birdhouse-deploy/tree/2.18.13
+    :target: https://github.com/bird-house/birdhouse-deploy/tree/2.18.14
 
 .. |readthedocs| image:: https://readthedocs.org/projects/birdhouse-deploy/badge/?version=latest
     :alt: ReadTheDocs Build Status (latest version)

RELEASE.txt

@@ -1 +1 @@
-2.18.13 2025-12-01T15:35:01Z
+2.18.14 2025-12-01T15:42:49Z

birdhouse/components/canarie-api/docker_configuration.py.template

@@ -108,8 +108,8 @@ SERVICES = {
             # NOTE:
             #   Below version and release time auto-managed by 'make VERSION=x.y.z bump'.
             #   Do NOT modify it manually. See 'Tagging policy' in 'birdhouse/README.rst'.
-            'version': '2.18.13',
-            'releaseTime': '2025-12-01T15:35:01Z',
+            'version': '2.18.14',
+            'releaseTime': '2025-12-01T15:42:49Z',
             'institution': '${BIRDHOUSE_INSTITUTION}',
             'researchSubject': '${BIRDHOUSE_SUBJECT}',
             'supportEmail': '${BIRDHOUSE_SUPPORT_EMAIL}',
@@ -141,8 +141,8 @@ PLATFORMS = {
             # NOTE:
             #   Below version and release time auto-managed by 'make VERSION=x.y.z bump'.
             #   Do NOT modify it manually. See 'Tagging policy' in 'birdhouse/README.rst'.
-            'version': '2.18.13',
-            'releaseTime': '2025-12-01T15:35:01Z',
+            'version': '2.18.14',
+            'releaseTime': '2025-12-01T15:42:49Z',
             'institution': '${BIRDHOUSE_INSTITUTION}',
             'researchSubject': '${BIRDHOUSE_SUBJECT}',
             'supportEmail': '${BIRDHOUSE_SUPPORT_EMAIL}',

birdhouse/components/jupyterhub/jupyterhub_config.py.template

@@ -295,10 +295,14 @@ if jupyter_idle_server_cull_timeout or jupyter_idle_kernel_cull_timeout:
 # It also makes a symlink between the kernels directory in the user space to another
 # location outside of the home directory. This is because jupyterlab is currently unable
 # to detect kernels inside the user's home directory for some reason.
+# This also removes any old server metadata files which may not be cleaned up if the container
+# exits unexpectedly. This is necessary to ensure that the healthcheck can detect the correct
+# hostname of the jupyterlab server (this changes every time a new container is created).
 _post_start_command = (
     "ln -s $HOME/.local /home/$NB_USER/.local; "
     f"mkdir -p {user_kernels_dir}; "
-    f"ln -s $HOME/.local/share/jupyter/kernels {user_kernels_dir}/kernels"
+    f"ln -s $HOME/.local/share/jupyter/kernels {user_kernels_dir}/kernels; "
+    "rm $HOME/.local/share/jupyter/runtime/jpserver-*.json"
 )
 c.DockerSpawner.post_start_cmd = f"sh -c '{_post_start_command}'"
 

docs/source/conf.py

@@ -69,9 +69,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '2.18.13'
+version = '2.18.14'
 # The full version, including alpha/beta/rc tags.
-release = '2.18.13'
+release = '2.18.14'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.