Create reusable deployment scripts

[troyraen]

This is "Future work" that could follow #256.

Directory structure

Could look something like this (updated 13 Mar 2025):

broker/
├── deploy/
│   ├── configs-template.yaml       # env.yaml becomes this.
│   ├                               #   Modules will use it to define all resources and their
│   ├                               #   they rely on, including all required configurations.
│   ├── deploy.sh                   # Current classify_snn/deploy.sh generalizes to this.
│   ├                               #   Should be able to accept any cloud run module's configs.yaml as input.
│   ├── cloudbuild.cloud-run.yaml   # Every cloud run module will use this.
│   ├── cloudbuild.consumer.yaml    # For example.
│   ├── _create-ancillaries.sh       # Called by deploy.sh.
│   ├                               #   Should be able to iterate over entries found in
│   ├                               #   configs.yaml of a given type (eg, pubsub topic)
│   ├                               #   and create or delete them, agnostic to the details.
│   ├── _construct_name.sh          # Called by deploy.sh and others.
├── cloud_run/classify_snn/
│   └── main.py ... etc.            # All surveys use these classify_snn/ files.
│   ├── configs.lsst.yaml           # All surveys define their own configs file.
│   ├── configs.ztf.yaml
├── cloud_run/ps_to_storage/
│   └── main.py ... etc.            # All surveys use these ps_to_storage/ files.
│   ├── configs.lsst.yaml           # All surveys define their own configs file.
│   ├── configs.lvk.yaml
│   ├── configs.ztf.yaml
# ... etc...

Steps

All of these are 'classify_snn/' --> 'deploy/':

• Move 'construct_name.sh' --> '_construct_name.sh'.
• Convert 'env.yaml' --> 'configs-template.yaml'.
• Convert 'deploy.sh' --> 'deploy.sh'.
• Convert 'cloudbuild.yaml' --> 'cloudbuild.cloud-run.yaml'.
• Generalize 'create-ancillaries.sh' --> '/_create-ancillaries.sh'.

How this could be used with Terraform

Terraform can take the 'cloudbuild*.yaml' files and execute them. To do that, we'll need to write a Terraform script that will replace 'deploy.sh'. I gave 'deploy.sh' to Google Gemini and asked it to convert it to a Terraform script and it gave me what looked like a reasonable start. See also, #152. (If/when anyone's ready to actually implement Terraform, you can either use that PR directly or else close it and open your own.)

Similar to our current setup, an individual module could then be deployed by "applying" its Terraform script. We could also write a "main.tf" Terraform script that could be used to deploy all modules at once. Terraform could then be used in combination with, e.g., GitHub Actions to create automatic deployments for both test and production purposes.

[troyraen]

Maybe it would be better to get rid of survey directories, more like this:

├── cloud_run/classify_snn/
│   └── main.py ... etc.            # All surveys use these classify_snn/ files.
│   ├── configs.lsst.yaml           # All surveys define their own configs file.
│   ├── configs.ztf.yaml

Updated the main comment.