fix(testing): Add Variable expansion testing

[zacharyblasczyk]

Add dynamic test generator for environment variable validation

Implement snapshots.sh generate-env-tests to automatically generate
test-env-expansion.yaml from Chart.yaml and values.yaml. This creates
individual test containers for each service that validate environment
variables don't contain unresolved shell-style $(VAR) references.

Key features:

• Generates 20+ service-specific test containers
• Each container replicates its service's exact envFrom, envTpls, and env
• Uses YAML anchor to DRY the test command across all containers
• Runs automatically in CI via helm test during chart-testing
• Catches bugs where ConfigMap values use $(VAR) instead of Helm templating

The test prevents accidentally shipping configurations with unresolved
variable references like: GORILLA_GLUE_CLICKHOUSE_ADDRESS="http://$(WF_CLICKHOUSE_HOST)"
when WF_CLICKHOUSE_HOST isn't defined in that service's environment.

Here is an example of one such case:

| ------------------------------------------------------------------------------------------------------------------------
| ==> Logs of container operator-wandb-6aney5tys4-***-env-expansion
| ------------------------------------------------------------------------------------------------------------------------
| Checking for unresolved shell variable references in environment...
| 
| ERROR: Found unresolved shell variable references:
| GORILLA_GLUE_CLICKHOUSE_ADDRESS=http://$(WF_CLICKHOUSE_HOST):$(WF_CLICKHOUSE_PORT)
| 
| These values contain unresolved $(VARIABLE) references that should be
| expanded using Helm templating instead of shell-style variable interpolation.
| ------------------------------------------------------------------------------------------------------------------------
| <== Logs of container operator-wandb-6aney5tys4-***-env-expansion
| ------------------------------------------------------------------------------------------------------------------------

Summary by CodeRabbit

• New Features

  • Added automated environment-expansion tests to detect unresolved template variables across services.
  • Added a test-generation utility and accompanying usage documentation to produce those tests.

• Tests

  • Introduced a comprehensive Helm/Kubernetes test manifest that validates env template expansion per service.

• Chores

  • Bumped chart version to 0.40.1.
  • Simplified ClickHouse environment template references.
  • Improved deployment tooling with prerequisite checks and expanded helper commands.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds a Helm test generator and test manifest to validate environment-variable expansion for operator-wandb, renames/ inlines ClickHouse env template, updates ClickHouse address templating, and bumps the chart version.

Changes

Cohort / File(s)	Summary
Chart Version charts/operator-wandb/Chart.yaml	Bumped chart version from 0.40.0 → 0.40.1.
ClickHouse env templates & usages charts/operator-wandb/templates/_env.tpl, charts/operator-wandb/templates/glue.yaml, charts/operator-wandb/values.yaml	Renamed/ inlined wandb.clickhouseConfigEnvs → wandb.clickhouseEnvs; updated glue to use include "wandb.clickhouse.host"/port templating instead of env vars; updated values references.
Helm test manifest charts/operator-wandb/templates/tests/test-env-expansion.yaml	New Helm/Kubernetes test Pod manifest with an anchor and per-service containers to detect unresolved $(VARIABLE) env templates.
Test generation tooling scripts/generate-env-tests.go, scripts/go.mod, scripts/README.md	Added Go generator that reads Chart.yaml and values.yaml to emit the test manifest; added module file and README documenting usage.
Snapshot tooling snapshots.sh	Added Helm & plugin prereq checks, new generate-env-tests command, default chart handling, and helper to run the new Go generator.

Sequence Diagram(s)

sequenceDiagram
    participant Dev as Developer/CI
    participant Gen as generate-env-tests.go
    participant Chart as Chart.yaml (charts/)
    participant Values as values.yaml (charts/)
    participant TestFile as templates/tests/test-env-expansion.yaml
    participant K8s as Kubernetes (helm test)

    Dev->>Gen: run generator (chart name)
    Gen->>Chart: load Chart.yaml dependencies
    Chart-->>Gen: wandb-base service aliases
    Gen->>Values: load values.yaml
    Values-->>Gen: per-service env/envFrom/envTpls
    Gen->>TestFile: emit anchor + per-service containers
    TestFile-->>Gen: test manifest written
    Dev->>K8s: deploy/run helm test
    K8s->>K8s: run anchor container (env validation)
    K8s->>K8s: run per-service env-check containers (grep for '$(VARIABLE)')
    K8s-->>Dev: test pass/fail

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• add support for ch secret refs #536: Related ClickHouse env/template work — overlaps with renaming/inlining and ClickHouse wiring changes.
• feat: Add clickhouse local install via bitnami charts #390: Previously modified ClickHouse templating functions referenced here (host/port), directly related.
• chore: Add snapshot testing #434: Related snapshot/test automation changes and snapshots.sh adjustments.

Suggested reviewers

• vanpelt
• jsbroks
• danielpanzella

Poem

🐰 I hopped through charts and YAML glens,

I chased the sneaky $(VARIABLE) pens,
With Go I stitched a proving pod,
Now envs expand and tests applaud,
A rabbit cheers — no stray vars then!

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The PR title 'fix(testing): Add Variable expansion testing' directly and clearly summarizes the main change: adding variable expansion testing functionality to catch unresolved environment variable references in Helm charts.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch testing-env-expansion

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}