Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Cloud Build to create all resources #256

Open
wants to merge 2 commits into
base: u/ch/lsst/SuperNNova
Choose a base branch
from

Conversation

troyraen
Copy link
Collaborator

@troyraen troyraen commented Mar 9, 2025

@hernandezc1 This is an example of how I envision using Cloud Build to create all resources needed for a given module (supernnova, in this case). Some notes:

  • This is just a draft -- I have not tested it at all. It contains several FIXMEs and CHECKMEs, and it likely contains some bugs.
  • Example usage is at the top of 'deploy.sh'. Start looking there.
  • It's up to you whether you want to actually merge this into your branch and then iterate on it, or if you want to leave it unmerged and just reference it for ideas when writing your own code. Makes no difference to me. I just wanted to show you what I had in mind. I'm assigning it to you so that it's clear that you can do what you want with it from here.

Included changes

  • Add 'create-ancillaries.sh' -- A dedicated script to create all ancillary resources (e.g., Pub/Sub topics and subscriptions).
    • Add a step in 'cloudbuild.yaml' that executes this script.
    • Remove these calls from 'deploy.sh'.
  • Add 'env.yaml' -- A file containing all configurable variables for the module.
    • Update 'deploy.sh' to get variables from this yaml rather than from command line arguments.
  • Add 'construct-name.sh' -- A script that takes a resource name stem as input and constructs the resource name by prepending/appending the survey name and testid as appropriate.
    • Update other scripts to use this whenever a resource name needs to be constructed.

Known omissions (will need to be added sooner rather than later)

  • No functionality to teardown/delete resources. Need to think about whether to create dedicated script(s) for this or add it to existing scripts.

Future work

Turn 'deploy.sh', etc. into reusable scripts. Individual modules should define all resources that they rely on, including all configuration, in an 'env.yaml'. The reusable scripts should take that as input and be able to handle (eg., iterate over) whatever they find in there.

First steps will be:

  • There are two FIXMEs in 'deploy.sh' indicating that those blocks of code need to be taken out of that script. Goal is for that script to load env vars and launch cloud build, that's it.
  • Define resource name stems in 'env.yaml' rather than in 'create-ancillaries.sh'.

The rest has moved to #264.

@troyraen troyraen requested a review from hernandezc1 March 9, 2025 05:07
@troyraen troyraen added the Enhancement New feature or request label Mar 9, 2025
@troyraen
Copy link
Collaborator Author

Opened #264 and updated the main comment.

@troyraen troyraen force-pushed the u/raen/draft/cloud-build branch from 92f3df7 to ce60feb Compare March 11, 2025 09:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants