Add mechanism to set CPU and memory limits on Jupyterlab containers based on Magpie group (#608)

## Overview
  
Creates a new variable `JUPYTERHUB_RESOURCE_LIMITS` which sets resource
limits for JupyterLab containers per Magpie user or group.
 
The value for this variable is a whitespace delimited string. Each
section is delimited by colons (:) where the first element is either
`group` or `user` and the second element is the name of the user or
group to apply the limits to. The rest are resource limits of the form
`limit=amount`. For example:

  ```sh
  export JUPYTERHUB_RESOURCE_LIMITS="
     user:user1:mem_limit=30G
     group:group1:mem_limit=10G:cpu_limit=1
     group:group2:cpu_limit=3
  "
  ```

Supported limits are:
[mem_limit](https://jupyterhub-dockerspawner.readthedocs.io/en/latest/api/index.html#dockerspawner.DockerSpawner.mem_limit)
and
[cpu_limit](https://jupyterhub-dockerspawner.readthedocs.io/en/latest/api/index.html#dockerspawner.DockerSpawner.cpu_limit).

Note that this will not create the groups in Magpie, that must be done
by some other means (through configuration files or the Magpie API or
UI).

Note that if a user belongs to multiple groups, later values in
`JUPYTERHUB_RESOURCE_LIMITS` will take precedence. For example, if a
user named user1 belongs to group1 and group2 then the following limits
will apply:

  - mem_limit=10G (because group1 is later in the list)
  - cpu_limit=3 (because group2 is later in the list)

## Changes

**Non-breaking changes**
- can specify per-group jupyterlab resource limits

**Breaking changes**
- None

## Related Issue / Discussion

These limits are the simplest to implement. Future PRs could use this
mechanism to set other limits (if possible) such as GPU/TPU allocations,
disk allocations, IO limits, etc.

## Additional Information

This only works with the version of the `MagpieAuthenticator` in this
PR: Ouranosinc/jupyterhub#35

That PR needs to be merged first and the new `pavics/jupyterhub` docker
image needs to be updated before all features introduced here will take
effect.

## CI Operations

<!--
The test suite can be run using a different DACCS config with
``birdhouse_daccs_configs_branch: branch_name`` in the PR description.
To globally skip the test suite regardless of the commit message use
``birdhouse_skip_ci`` set to ``true`` in the PR description.

Using ``[<cmd>]`` (with the brackets) where ``<cmd> = skip ci`` in the
commit message will override ``birdhouse_skip_ci`` from the PR
description.
Such commit command can be used to override the PR description behavior
for a specific commit update.
However, a commit message cannot 'force run' a PR which the description
turns off the CI.
To run the CI, the PR should instead be updated with a ``true`` value,
and a running message can be posted in following PR comments to trigger
tests once again.
-->

birdhouse_daccs_configs_branch: master
birdhouse_skip_ci: false

Author: mishaschwartz

.bumpversion.toml

@@ -1,5 +1,5 @@
 [tool.bumpversion]
-current_version = "2.18.11"
+current_version = "2.18.12"
 commit = true
 tag = false
 tag_name = "{new_version}"

CHANGES.md

@@ -17,6 +17,41 @@
 
 [//]: # (list changes here, using '-' for each new entry, remove this when items are added)
 
+[2.18.12](https://github.com/bird-house/birdhouse-deploy/tree/2.18.12) (2025-11-25)
+------------------------------------------------------------------------------------------------------------------
+
+## Changes
+
+- Add mechanism to set CPU and memory limits on Jupyterlab containers based on Magpie user or group name
+
+  Creates a new variable `JUPYTERHUB_RESOURCE_LIMITS` which sets resource limits for JupyterLab containers per 
+  Magpie user or group.
+ 
+  The value for this variable is a whitespace delimited string. Each section is delimited by colons (:) 
+  where the first element is either `group` or `user` and the second element is the name of the user or group
+  to apply the limits to. The rest are resource limits of the form `limit=amount`. For example:
+
+  ```sh
+  export JUPYTERHUB_RESOURCE_LIMITS="
+     user:user1:mem_limit=30G
+     group:group1:mem_limit=10G:cpu_limit=1
+     group:group2:cpu_limit=3
+  "
+  ```
+
+  Supported limits are: 
+  [mem_limit](https://jupyterhub-dockerspawner.readthedocs.io/en/latest/api/index.html#dockerspawner.DockerSpawner.mem_limit) 
+  and [cpu_limit](https://jupyterhub-dockerspawner.readthedocs.io/en/latest/api/index.html#dockerspawner.DockerSpawner.cpu_limit). 
+
+  Note that this will not create the groups in Magpie, that must be done by some other means (through configuration files or the
+  Magpie API or UI).
+
+  Note that if a user belongs to multiple groups, later values in `JUPYTERHUB_RESOURCE_LIMITS` will take
+  precedence. For example, if a user named user1 belongs to group1 and group2 then the following limits will apply:
+
+  - mem_limit=10G (because group1 is later in the list)
+  - cpu_limit=3 (because group2 is later in the list)
+
 [2.18.11](https://github.com/bird-house/birdhouse-deploy/tree/2.18.11) (2025-11-13)
 ------------------------------------------------------------------------------------------------------------------
 

Makefile

@@ -8,7 +8,7 @@ override BIRDHOUSE_MAKE_DIR := $(shell realpath -P $$(dirname $(BIRDHOUSE_MAKE_C
 # Generic variables
 override SHELL       := bash
 override APP_NAME    := birdhouse-deploy
-override APP_VERSION := 2.18.11
+override APP_VERSION := 2.18.12
 
 # utility to remove comments after value of an option variable
 override clean_opt = $(shell echo "$(1)" | $(_SED) -r -e "s/[ '$'\t'']+$$//g")

README.rst

@@ -18,13 +18,13 @@ for a full-fledged production platform.
     * - citation
       - | |citation|
 
-.. |commits-since| image:: https://img.shields.io/github/commits-since/bird-house/birdhouse-deploy/2.18.11.svg
+.. |commits-since| image:: https://img.shields.io/github/commits-since/bird-house/birdhouse-deploy/2.18.12.svg
     :alt: Commits since latest release
-    :target: https://github.com/bird-house/birdhouse-deploy/compare/2.18.11...master
+    :target: https://github.com/bird-house/birdhouse-deploy/compare/2.18.12...master
 
-.. |latest-version| image:: https://img.shields.io/badge/tag-2.18.11-blue.svg?style=flat
+.. |latest-version| image:: https://img.shields.io/badge/tag-2.18.12-blue.svg?style=flat
     :alt: Latest Tag
-    :target: https://github.com/bird-house/birdhouse-deploy/tree/2.18.11
+    :target: https://github.com/bird-house/birdhouse-deploy/tree/2.18.12
 
 .. |readthedocs| image:: https://readthedocs.org/projects/birdhouse-deploy/badge/?version=latest
     :alt: ReadTheDocs Build Status (latest version)

RELEASE.txt

@@ -1 +1 @@
-2.18.11 2025-11-13T20:15:43Z
+2.18.12 2025-11-25T02:46:23Z

birdhouse/components/canarie-api/docker_configuration.py.template

@@ -108,8 +108,8 @@ SERVICES = {
             # NOTE:
             #   Below version and release time auto-managed by 'make VERSION=x.y.z bump'.
             #   Do NOT modify it manually. See 'Tagging policy' in 'birdhouse/README.rst'.
-            'version': '2.18.11',
-            'releaseTime': '2025-11-13T20:15:43Z',
+            'version': '2.18.12',
+            'releaseTime': '2025-11-25T02:46:23Z',
             'institution': '${BIRDHOUSE_INSTITUTION}',
             'researchSubject': '${BIRDHOUSE_SUBJECT}',
             'supportEmail': '${BIRDHOUSE_SUPPORT_EMAIL}',
@@ -141,8 +141,8 @@ PLATFORMS = {
             # NOTE:
             #   Below version and release time auto-managed by 'make VERSION=x.y.z bump'.
             #   Do NOT modify it manually. See 'Tagging policy' in 'birdhouse/README.rst'.
-            'version': '2.18.11',
-            'releaseTime': '2025-11-13T20:15:43Z',
+            'version': '2.18.12',
+            'releaseTime': '2025-11-25T02:46:23Z',
             'institution': '${BIRDHOUSE_INSTITUTION}',
             'researchSubject': '${BIRDHOUSE_SUBJECT}',
             'supportEmail': '${BIRDHOUSE_SUPPORT_EMAIL}',

birdhouse/components/jupyterhub/default.env

@@ -5,7 +5,7 @@
 # are applied and must be added to the list of DELAYED_EVAL.
 
 export JUPYTERHUB_DOCKER=pavics/jupyterhub
-export JUPYTERHUB_VERSION=5.2.1-20241114
+export JUPYTERHUB_VERSION=5.4.2-20251123
 export JUPYTERHUB_IMAGE='${JUPYTERHUB_DOCKER}:${JUPYTERHUB_VERSION}'
 export JUPYTERHUB_IMAGE_URI='registry.hub.docker.com/${JUPYTERHUB_IMAGE}'
 
@@ -33,7 +33,6 @@ export JUPYTER_GOOGLE_DRIVE_SETTINGS=""
 export JUPYTER_DEMO_USER="demo"
 # Changing any limits requires restarting the jupyter user server
 export JUPYTER_DEMO_USER_MEM_LIMIT="2G"  # ex: 2G, 500M
-# CPU limit seems not honored by DockerSpawner
 export JUPYTER_DEMO_USER_CPU_LIMIT="0.5"  # 50% of 1 CPU
 
 # See config/jupyterhub/custom_templates/login.html.template
@@ -73,6 +72,26 @@ export JUPYTERHUB_AUTHENTICATOR_REFRESH_AGE=60
 # Usernames that should be given admin access in jupyterhub
 export JUPYTERHUB_ADMIN_USERS='{\"${MAGPIE_ADMIN_USERNAME}\"}'  # python set syntax
 
+# Resource limits for JupyterLab containers. Resource limits can be set per Magpie user or group.
+# The value for this variable is a whitespace delimited string. Each section is delimited by colons (:) 
+# where the first element is either `group` or `user` and the second element is the name of the user or group
+# to apply the limits to. The rest are resource limits of the form `limit=amount`. For example:
+#
+#  export JUPYTERHUB_RESOURCE_LIMITS="
+#     user:user1:mem_limit=30G
+#     group:group1:mem_limit=10G:cpu_limit=1
+#     group:group2:cpu_limit=3
+#  "
+#
+#  Supported limits are: `mem_limit` and `cpu_limit`. See the Jupyterhub Dockerspawner documentation
+#  for details and supported values.
+#  Note that this will not create the groups in Magpie, that must be done manually.
+#  Note that if a user belongs to multiple groups, later values in `JUPYTERHUB_RESOURCE_LIMITS` will take
+#  precedence. For example, if a user named user1 belongs to group1 and group2 then the following limits will apply:
+#  - mem_limit=10G (because group1 is later in the list)
+#  - cpu_limit=3 (because group2 is later in the list)
+export JUPYTERHUB_RESOURCE_LIMITS=
+
 export DELAYED_EVAL="
   $DELAYED_EVAL
   JUPYTERHUB_USER_DATA_DIR
@@ -105,6 +124,7 @@ OPTIONAL_VARS="
   \$JUPYTER_IDLE_KERNEL_CULL_TIMEOUT
   \$JUPYTER_IDLE_KERNEL_CULL_INTERVAL
   \$JUPYTERHUB_USER_DATA_DIR
+  \$JUPYTERHUB_RESOURCE_LIMITS
 "
 
 # add any component that this component requires to run

birdhouse/components/jupyterhub/jupyterhub_config.py.template

@@ -137,6 +137,11 @@ if os.environ['WORKSPACE_DIR'] != jupyterhub_data_dir:
 container_gdrive_settings_path = join(container_home_dir, ".jupyter/lab/user-settings/@jupyterlab/google-drive/drive.jupyterlab-settings")
 host_gdrive_settings_path = os.environ['JUPYTER_GOOGLE_DRIVE_SETTINGS']
 
+# resource_limits: dict[tuple[Literal["user", "group"], str], dict[Literal["cpu_limit", "mem_limit"], str]]
+resource_limits = {tuple(lim[:2]): dict(li.split("=") for li in lim[2:] if "=" in li)
+                    for limit in """${JUPYTERHUB_RESOURCE_LIMITS}""".strip().split() 
+                    if (lim := limit.split(":"))}
+
 if len(host_gdrive_settings_path) > 0:
     c.DockerSpawner.volumes[host_gdrive_settings_path] = {
         "bind": container_gdrive_settings_path,
@@ -169,13 +174,29 @@ def create_dir_hook(spawner):
         subprocess.call(["chown", f"{os.environ['USER_WORKSPACE_UID']}:{os.environ['USER_WORKSPACE_GID']}",
                          workspace_user_dir])
 
-    if username == os.environ['JUPYTER_DEMO_USER']:
+
+def limit_resource_hook(spawner):
+    if spawner.user.name == os.environ['JUPYTER_DEMO_USER']:
         # Restrict resources for the public demo user
         # CPU limit, seems not honored by DockerSpawner
         spawner.cpu_limit = float(os.environ['JUPYTER_DEMO_USER_CPU_LIMIT'])
         spawner.mem_limit = os.environ['JUPYTER_DEMO_USER_MEM_LIMIT']
-
-c.Spawner.pre_spawn_hook = create_dir_hook
+    
+    user_groups = {g.name for g in spawner.user.groups}
+    for (name_type, name), limits in resource_limits.items():
+        if (name_type == "user" and name == spawner.user.name) or (name_type == "group" and name in user_groups):
+            for limit, value in limits.items():
+                if limit == "cpu_limit":
+                    spawner.cpu_limit = float(value)
+                elif limit == "mem_limit":
+                    spawner.mem_limit = value
+
+
+def pre_spawn_hook(spawner):
+    create_dir_hook(spawner)
+    limit_resource_hook(spawner)
+
+c.Spawner.pre_spawn_hook = pre_spawn_hook
 
 ## Disable per-user configuration of single-user servers.
 c.Spawner.disable_user_config = True

birdhouse/env.local.example

@@ -363,6 +363,28 @@ export GEOSERVER_ADMIN_PASSWORD="${__DEFAULT__GEOSERVER_ADMIN_PASSWORD}"
 # JUPYTERHUB_CRYPT_KEY is set.
 #export JUPYTERHUB_AUTHENTICATOR_REFRESH_AGE=60
 
+# Resource limits for JupyterLab containers. Resource limits can be set per Magpie user or group.
+# The value for this variable is a whitespace delimited string. Each section is delimited by colons (:) 
+# where the first element is either `group` or `user` and the second element is the name of the user or group
+# to apply the limits to. The rest are resource limits of the form `limit=amount`. For example:
+#
+#  export JUPYTERHUB_RESOURCE_LIMITS="
+#     user:user1:mem_limit=30G
+#     group:group1:mem_limit=10G:cpu_limit=1
+#     group:group2:cpu_limit=3
+#  "
+#
+#  Supported limits are: `mem_limit` and `cpu_limit`. See the Jupyterhub Dockerspawner documentation
+#  for details and supported values.
+#   - https://jupyterhub-dockerspawner.readthedocs.io/en/latest/api/index.html#dockerspawner.DockerSpawner.cpu_limit
+#   - https://jupyterhub-dockerspawner.readthedocs.io/en/latest/api/index.html#dockerspawner.DockerSpawner.mem_limit
+#  Note that this will not create the groups in Magpie, that must be done manually.
+#  Note that if a user belongs to multiple groups, later values in `JUPYTERHUB_RESOURCE_LIMITS` will take
+#  precedence. For example, if a user named user1 belongs to group1 and group2 then the following limits will apply:
+#  - mem_limit=10G (because group1 is later in the list)
+#  - cpu_limit=3 (because group2 is later in the list)
+#export JUPYTERHUB_RESOURCE_LIMITS=
+
 # Allow for adding new config or override existing config in
 # config/jupyterhub/jupyterhub_config.py.template.
 #
@@ -574,6 +596,7 @@ export THREDDS_ADDITIONAL_CATALOG=''
 #export JUPYTER_DEMO_USER="demo"
 # Changing any limits requires restarting the jupyter user server
 #export JUPYTER_DEMO_USER_MEM_LIMIT="2G"  # ex: 2G, 500M
+#export JUPYTER_DEMO_USER_CPU_LIMIT="0.5"  # 50% of 1 CPU
 
 # See config/jupyterhub/custom_templates/login.html.template
 #export JUPYTER_LOGIN_BANNER_TOP_SECTION=""

docs/source/conf.py

@@ -69,9 +69,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '2.18.11'
+version = '2.18.12'
 # The full version, including alpha/beta/rc tags.
-release = '2.18.11'
+release = '2.18.12'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.