Merge pull request #2888 from cal-itp/infra-docs

kubernetes and gitops docs updates

.github/README.md → .github/workflows/README.md

@@ -1,6 +1,6 @@
 # GitHub Actions
 
-All CI/CD automation in this project is executed via GitHub Actions, whose workflow files live in the [./workflows/](./workflows) directory.
+All CI/CD automation in this project is executed via GitHub Actions, whose workflow files live in this directory.
 
 ## deploy-airflow.yml
 

.github/workflows/build-dbt.yml → .github/workflows/build-warehouse-image.yml

@@ -5,11 +5,11 @@ on:
     branches:
       - 'main'
     paths:
-      - '.github/workflows/build-dbt.yml'
+      - '.github/workflows/build-warehouse-image.yml'
       - 'warehouse/**'
   pull_request:
     paths:
-      - '.github/workflows/build-dbt.yml'
+      - '.github/workflows/build-warehouse-image.yml'
       - 'warehouse/**'
 
 concurrency:

airflow/README.md

@@ -59,6 +59,22 @@ docker-compose run airflow tasks test download_gtfs_schedule_v2 download_schedul
 
 Additional reading about this setup can be found on the [Airflow Docs](https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html)
 
+### PodOperators
+Airflow PodOperator tasks execute a specific Docker image; as of 2023-08-24 these images are pushed to [GitHub Container Registry](https://ghcr.io/) and production uses `:latest` tags while local uses `:development`. If you want to test these tasks locally, you must build and push development versions of the images used by the tasks. The Dockerfiles and code that make up the images live in the [../jobs](../jobs) directory. For example:
+
+```bash
+# running from jobs/gtfs-schedule-validator/
+docker build -t ghcr.io/cal-itp/data-infra/gtfs-schedule-validator:development .
+docker push ghcr.io/cal-itp/data-infra/gtfs-schedule-validator:development
+```
+
+Then, you could execute a task using this updated image.
+
+```bash
+# running from airflow/
+docker-compose run airflow tasks test unzip_and_validate_gtfs_schedule_hourly validate_gtfs_schedule 2023-06-07T16:00:00
+```
+
 ### Common Issues
 
 * `docker-compose up` exits with code 137 - Check that your docker has enough RAM (e.g. 8Gbs). See [this post](https://stackoverflow.com/questions/44533319/how-to-assign-more-memory-to-docker-container) on how to increase its resources.

airflow/dags/download_gtfs_schedule_v2/README.md

@@ -3,3 +3,6 @@
 Type: [Now / Scheduled](https://docs.calitp.org/data-infra/airflow/dags-maintenance.html)
 
 This DAG orchestrates raw data capture for GTFS schedule data. It reads GTFS data configuration files that are generated by the [`airtable_loader_2` DAG](../airtable_loader_v2/README.md) to determine the list of GTFS schedule URLs to scrape (this DAG will just find the latest such configuration file, so there is no formal dependency between the two DAGs on a daily run basis.)
+
+## Secrets
+You may need to change authentication information in [Secret Manager](https://console.cloud.google.com/security/secret-manager); auth keys are loaded from Secret Manager at the start of DAG executions. You may create new versions of existing secrets, or add entirely new secrets. Secrets must be tagged with `gtfs_schedule: true` to be loaded and are referenced by `url_secret_key_name` or `header_secret_key_name` in Airtable's GTFS dataset records.

ci/README.md

@@ -15,3 +15,29 @@ Individual release channels/environments are config files that are passed to inv
 ```bash
 poetry run invoke release -f channels/test.yaml
 ```
+
+## GitOps
+
+In this diagram, arrows represent human actions such as opening and merging PRs and nodes (except for the very first) represent automated actions such as `invoke` deploying to the cluster. Green nodes indicate a deployment while white nodes indicate an automated git action such as branch creation or commenting on a pull request.
+
+```mermaid
+flowchart TD
+classDef default fill:white, color:black, stroke:black
+classDef initial fill:lightblue, color:black
+classDef deploy fill:lightgreen, color:black
+
+pr[Push commits to a branch.\nDoes a test environment exist?]
+candidates_branch[GitHub Action renders candidates/branch-name]
+branch_diff[invoke diff renders on test PR]
+branch_invoke[invoke releases to test]
+
+candidates_main[GitHub Action builds images and renders candidates/main\nNote: if you stop here, no Kubernetes changes will actually be deployed.]
+prod_diff[invoke diff renders on prod PR]
+prod_invoke[invoke releases to prod]
+
+pr -- Yes --> candidates_branch -- "Open PR from candidates/branch-name to releases/test" --> branch_diff -- "Merge candidate PR to releases/test" --> branch_invoke -- Merge original PR to main after review and testing --> candidates_main -- "Open PR from candidates/main to releases/prod" --> prod_diff -- "Merge candidate PR to releases/prod" --> prod_invoke
+pr -- "No; merge to main after review" --> candidates_main
+
+class pr initial
+class branch_invoke,prod_invoke deploy
+```

docs/_toc.yml

@@ -47,12 +47,6 @@ parts:
           - file: architecture/data
       - file: airflow/dags-maintenance
       - file: transit_database/transitdatabase
-      - file: kubernetes/README
-        sections:
-        - file: kubernetes/JupyterHub
-        - file: kubernetes/architecture
-        - file: kubernetes/deployment
-      - file: backups/metabase
   - caption: Contribute to the Docs!
     chapters:
       - file: contribute/overview