Skip to content

Commit 218fbeb

Browse files
cortadocodescortadocodes
authored
MRG: Merge pull request #1 from octue/create-initial-module
Create initial module
2 parents dbc40f0 + 9283abc commit 218fbeb

16 files changed

+570
-2
lines changed

‎.github/workflows/release.yml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# This workflow releases a new version of the Terraform module.
2+
3+
name: Release
4+
5+
# Only trigger when a pull request into main branch is merged.
6+
on:
7+
pull_request:
8+
types: [closed]
9+
branches:
10+
- main
11+
12+
jobs:
13+
release:
14+
runs-on: ubuntu-latest
15+
steps:
16+
- name: Checkout Repository
17+
uses: actions/checkout@v4
18+
19+
- name: Get module version
20+
id: get-version
21+
run: echo "version=$(cat VERSION.txt)" >> $GITHUB_OUTPUT
22+
23+
- name: Create Release
24+
uses: actions/create-release@v1
25+
env:
26+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # This token is provided by Actions, no need to create your own.
27+
with:
28+
tag_name: ${{ steps.get-version.outputs.version }}
29+
release_name: ${{ github.event.pull_request.title }}
30+
body: ${{ github.event.pull_request.body }}
31+
draft: false
32+
prerelease: false

‎.github/workflows/update-pull-request.yml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# This workflow updates the pull request description with an auto-generated section containing the categorised commit
2+
# message headers of the pull request's commits. The auto generated section is enveloped between two comments:
3+
# "<!--- START AUTOGENERATED NOTES --->" and "<!--- END AUTOGENERATED NOTES --->". Anything outside these in the
4+
# description is left untouched. Auto-generated updates can be skipped for a commit if
5+
# "<!--- SKIP AUTOGENERATED NOTES --->" is added to the pull request description.
6+
7+
name: update-pull-request
8+
9+
on: [pull_request]
10+
11+
jobs:
12+
description:
13+
uses: octue/workflows/.github/workflows/generate-pull-request-description.yml@main
14+
secrets:
15+
token: ${{ secrets.GITHUB_TOKEN }}
16+
permissions:
17+
contents: read
18+
pull-requests: write

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,3 +35,5 @@ override.tf.json
3535
# Ignore CLI configuration files
3636
.terraformrc
3737
terraform.rc
38+
39+
.idea

‎README.md‎

Lines changed: 149 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,149 @@
1-
# terraform-octue-twined-static
2-
A partner terraform module to terraform-octue-twined providing static resources.
1+
# terraform-octue-twined-core
2+
A Terraform module for deploying the core storage and IAM resources for an Octue Twined services network to google cloud.
3+
4+
5+
# Resources
6+
These resources are automatically deployed:
7+
- An artifact registry repository for storing Octue Twined service revision docker images
8+
- A BigQuery table acting as an event store for Twined service events
9+
- IAM service accounts and roles for:
10+
- Any number of maintainers to use with Twined services
11+
- GitHub Actions to a) build and push Twined service images to the artifact registry; b) test services
12+
- A workload identity pool and provider allowing GitHub actions to authenticate with google cloud
13+
- A cloud storage bucket to store input, output, and diagnostics data for Twined services
14+
15+
16+
# Installation and usage
17+
18+
> [!IMPORTANT]
19+
> Deploying this Terraform module is a prerequisite to deploying the [terraform-octue-twined-cluster](https://github.com/octue/terraform-octue-twined-cluster).
20+
> module. You must deploy both to have a cloud-based Octue Twined services network. See [a live example here](https://github.com/octue/twined-infrastructure).
21+
22+
> [!TIP]
23+
> Deploy this module in a separate Terraform configuration (directory/workspace) to the [terraform-octue-twined-cluster](https://github.com/octue/terraform-octue-twined-cluster)
24+
> module. This allows the option to spin down the Kubernetes cluster provided by the other module while keeping the core
25+
> resources that contain all data produced by your Twined services. Spinning the cluster down entirely can save on
26+
> running costs in periods of extended non-use while keeping all data available.
27+
28+
Add the below blocks to your Terraform configuration and run:
29+
```shell
30+
terraform plan
31+
```
32+
33+
If you're happy with the plan, run:
34+
```shell
35+
terraform apply
36+
```
37+
and approve the run.
38+
39+
## Minimal configuration
40+
41+
```terraform
42+
# main.tf
43+
44+
terraform {
45+
required_version = ">= 1.8.0"
46+
47+
required_providers {
48+
google = {
49+
source = "hashicorp/google"
50+
version = "~>6.12"
51+
}
52+
}
53+
}
54+
55+
56+
provider "google" {
57+
project = var.google_cloud_project_id
58+
region = var.google_cloud_region
59+
}
60+
61+
62+
module "octue_twined_core" {
63+
source = "git::github.com/octue/terraform-octue-twined-core.git?ref=0.1.0"
64+
google_cloud_project_id = var.google_cloud_project_id
65+
google_cloud_region = var.google_cloud_region
66+
github_account = var.github_account
67+
maintainer_service_account_names = var.maintainer_service_account_names
68+
}
69+
```
70+
71+
```terraform
72+
# variables.tf
73+
74+
variable "google_cloud_project_id" {
75+
type = string
76+
default = "<google-cloud-project-id>"
77+
}
78+
79+
variable "google_cloud_region" {
80+
type = string
81+
default = "<google-cloud-region>"
82+
}
83+
84+
variable "github_account" {
85+
type = string
86+
default = "<your-github-account>"
87+
}
88+
89+
variable "maintainer_service_account_names" {
90+
type = set(string)
91+
default = ["person1", "person2"]
92+
}
93+
```
94+
95+
## Dependencies
96+
- Terraform: `>= 1.8.0, <2`
97+
- Providers:
98+
- `hashicorp/google`: `~>6.12`
99+
- Google cloud APIs:
100+
- The Cloud Resource Manager API must be [enabled manually](https://console.developers.google.com/apis/api/cloudresourcemanager.googleapis.com)
101+
before using the module
102+
- All other required google cloud APIs are enabled automatically by the module
103+
104+
## Authentication
105+
The module needs to authenticate with google cloud before it can be used:
106+
107+
1. Create a service account for Terraform and assign it the `editor` and `owner` basic IAM permissions
108+
2. Download a JSON key file for the service account
109+
3. If using Terraform Cloud, follow [these instructions](https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/provider_reference#using-terraform-cloud).
110+
before deleting the key file from your computer
111+
4. If not using Terraform Cloud, follow [these instructions](https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/provider_reference#authentication-configuration)
112+
or use another [authentication method](https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/provider_reference#authentication).
113+
114+
115+
## Destruction
116+
> [!WARNING]
117+
> If the `deletion_protection` input is set to `true`, it must first be set to `false` and `terraform apply` run before
118+
> running `terraform destroy` or any other operation that would result in the destruction or replacement of the cloud
119+
> storage bucket or event store BigQuery table. Not doing this can lead to a state needing targeted Terraform commands
120+
> and/or manual configuration changes to recover from.
121+
122+
Disable `deletion_protection` and run:
123+
```shell
124+
terraform destroy
125+
```
126+
127+
128+
# Input reference
129+
130+
| Name | Type | Required | Default |
131+
|-------------------------------------|---------------|----------|-----------|
132+
| `google_cloud_project_id` | `string` | Yes | N/A |
133+
| `google_cloud_region` | `string` | Yes | N/A |
134+
| `github_account` | `string` | Yes | N/A |
135+
| `maintainer_service_account_names` | `set(string)` | Yes | N/A |
136+
| `deletion_protection` | `bool` | No | `true` |
137+
138+
See [`variables.tf`](/variables.tf) for descriptions.
139+
140+
141+
# Output reference
142+
143+
| Name | Type |
144+
|-------------------------------------|----------|
145+
| `artifact_registry_repository_name` | `string` |
146+
| `storage_bucket_url` | `string` |
147+
| `event_store_id` | `string` |
148+
149+
See [`outputs.tf`](/outputs.tf) for descriptions.

‎VERSION.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
0.1.0

‎artifact_registry.tf‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
resource "google_artifact_registry_repository" "service_docker_images" {
2+
repository_id = "octue-twined-services"
3+
description = "Docker image repository for Octue Twined services"
4+
format = "DOCKER"
5+
location = var.google_cloud_region
6+
depends_on = [time_sleep.wait_for_google_apis_to_enable]
7+
}

‎bigquery.tf‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
resource "google_bigquery_dataset" "service_event_dataset" {
2+
dataset_id = "octue_twined"
3+
description = "A dataset for storing Octue Twined service events."
4+
location = var.google_cloud_region
5+
depends_on = [time_sleep.wait_for_google_apis_to_enable]
6+
}
7+
8+
resource "google_bigquery_table" "service_event_table" {
9+
table_id = "service-events"
10+
dataset_id = google_bigquery_dataset.service_event_dataset.dataset_id
11+
clustering = ["sender", "question_uuid"]
12+
depends_on = [time_sleep.wait_for_google_apis_to_enable]
13+
deletion_protection = var.deletion_protection
14+
15+
schema = <<EOF
16+
[
17+
{
18+
"name": "datetime",
19+
"type": "TIMESTAMP",
20+
"mode": "REQUIRED"
21+
},
22+
{
23+
"name": "uuid",
24+
"type": "STRING",
25+
"mode": "REQUIRED"
26+
},
27+
{
28+
"name": "kind",
29+
"type": "STRING",
30+
"mode": "REQUIRED"
31+
},
32+
{
33+
"name": "event",
34+
"type": "JSON",
35+
"mode": "REQUIRED"
36+
},
37+
{
38+
"name": "other_attributes",
39+
"type": "JSON",
40+
"mode": "REQUIRED"
41+
},
42+
{
43+
"name": "originator",
44+
"type": "STRING",
45+
"mode": "REQUIRED"
46+
},
47+
{
48+
"name": "parent",
49+
"type": "STRING",
50+
"mode": "REQUIRED"
51+
},
52+
{
53+
"name": "sender",
54+
"type": "STRING",
55+
"mode": "REQUIRED"
56+
},
57+
{
58+
"name": "sender_type",
59+
"type": "STRING",
60+
"mode": "REQUIRED"
61+
},
62+
{
63+
"name": "sender_sdk_version",
64+
"type": "STRING",
65+
"mode": "REQUIRED"
66+
},
67+
{
68+
"name": "recipient",
69+
"type": "STRING",
70+
"mode": "REQUIRED"
71+
},
72+
{
73+
"name": "originator_question_uuid",
74+
"type": "STRING",
75+
"mode": "REQUIRED"
76+
},
77+
{
78+
"name": "parent_question_uuid",
79+
"type": "STRING",
80+
"mode": "NULLABLE"
81+
},
82+
{
83+
"name": "question_uuid",
84+
"type": "STRING",
85+
"mode": "REQUIRED"
86+
},
87+
{
88+
"name": "backend",
89+
"type": "STRING",
90+
"mode": "REQUIRED"
91+
},
92+
{
93+
"name": "backend_metadata",
94+
"type": "JSON",
95+
"mode": "REQUIRED"
96+
}
97+
]
98+
EOF
99+
}

‎google_apis.tf‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
locals {
2+
apis = toset(
3+
[
4+
"artifactregistry.googleapis.com", # Artifact Registry is used to store docker images of the Twined services.
5+
"bigquery.googleapis.com", # BigQuery provides the event store for Twined service events (questions, results, log messages etc.).
6+
"iam.googleapis.com", # IAM provides fine-grained authentication and authorisation to use and access the Twined services and input/output data.
7+
]
8+
)
9+
}
10+
11+
12+
resource "google_project_service" "google_apis" {
13+
for_each = local.apis
14+
service = each.key
15+
disable_dependent_services = true
16+
project = var.google_cloud_project_id
17+
}
18+
19+
20+
resource "time_sleep" "wait_for_google_apis_to_enable" {
21+
depends_on = [google_project_service.google_apis]
22+
create_duration = "1m"
23+
}

‎iam_roles_github_actions.tf‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
resource "google_project_iam_member" "github_actions__roles" {
2+
for_each = toset(
3+
[
4+
"roles/iam.serviceAccountUser",
5+
"roles/pubsub.editor",
6+
"roles/errorreporting.writer",
7+
"roles/artifactregistry.writer",
8+
"roles/storage.objectAdmin",
9+
# Allows the GHA to call "namespaces get" for Cloud Run to determine the resulting run URLs of the services.
10+
# This should also allow a service to get its own name by using:
11+
# https://stackoverflow.com/questions/65628822/google-cloud-run-can-a-service-know-its-own-url/65634104#65634104
12+
"roles/run.developer",
13+
]
14+
)
15+
project = var.google_cloud_project_id
16+
role = each.value
17+
member = "serviceAccount:${google_service_account.github_actions_service_account.email}"
18+
depends_on = [time_sleep.wait_for_google_apis_to_enable]
19+
}

0 commit comments

Comments
 (0)