formancehq · flemzord · Oct 11, 2024 · Oct 10, 2024 · Oct 10, 2024 · Oct 10, 2024
diff --git a/.github/workflows/pull_request.yml b/.github/workflows/pull_request.yml
@@ -25,7 +25,7 @@ jobs:
       - name: Retrieve chart changes
         id: chart_changes
         run: |
-          CHARTS=$(git diff origin/main --stat | grep -oE 'charts/[^/]+/' | sed 's|charts/||;s|/||' | sort -u | jq -R -s -c 'split("\n") | map(select(. != ""))')
+          CHARTS=$(git diff ${{ github.base_ref }} --stat | grep -oE 'charts/[^/]+/' | sed 's|charts/||;s|/||' | sort -u | jq -R -s -c 'split("\n") | map(select(. != ""))')
           echo "Changed charts: $CHARTS"
           echo "::set-output name=changed_charts::$CHARTS"
 
@@ -83,4 +83,10 @@ jobs:
             git status
             echo "There are changes in the repository"
             exit 1
-          fi
+          fi
+      - name: Archive chart
+        uses: actions/upload-artifact@v4
+        with:
+          name: charts_${{ github.ref_type == 'branch' && github.sha }}.tgz
+          path: charts/**/*.tgz
+          retention-days: 3
diff --git a/.gitignore b/.gitignore
@@ -5,4 +5,6 @@
 
 dist
 
+charts/**/LICENSE
+
 .tmp-earthly-out
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -1,67 +1,111 @@
 # Helm Charts
 
-## Validating the PR
+## Working Per Chart
 
-```bash
-earthly +pre-commit
-```
+### Building the Charts
 
-## Working Per Chart
+Each chart must implement the common targets interface with the following minimum required files:
+
+- `Chart.yaml`
+- `Earthfile`
+
+Each `Earthfile` Chart **must** implement the following targets to integrate with the CI:
+
+- `+sources`: Raw sources of the chart without the dependencies.
+> [!IMPORTANT]
+> A LICENSE file is included in every chart through the helper [SOURCE](./charts/Earthfile)
+- `+dependencies`: Raw sources with dependencies updated.
+- `+validate`: Validates the chart, including its dependencies.
+- `+package`: Packages the chart from validated sources.
+- `+readme`: (Optional): Include README when dependendies are validated.
+> The README file is generated with `helm-docs` and included in the chart through the helper [README_GENERATOR](./charts/Earthfile).
+
+> [!TIP]
+> A file named `README.md.gotmpl` can be added to the chart to customize the README generation.
+- `+schema`: (Optional): Generate a values schema from the `values.yaml` and include it in the chart sources. Then it will be validated with the `+validate` target.
 
-### Building the chart
 
-Each chart  must implement the common targets interface with the minimum required files:
-- Chart.yaml
-- Earthfile
-- values.yaml <--- It can be empty with {} as content
+### Core Library
 
-Each chart **must** implement the following targets:
-- `+sources`: raw sources of the chart without the dependendies
-- `+dependencies`: raw sources + dependencies updated
-- `+validate`: validate the chart from the dependencies
-- `+package`: package the chart from validated sources
+Each chart must implement the core Helm library as a dependency to include the common helpers for:
 
-> LICENCE is included in every chart throught the helper [SOURCE](./charts/Earthfile)
+- Improve resource naming
+- [Kubernetes recommended labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/)
+- Values naming
 
-> README is generated with `helm-docs` and it is included in the chart throught the helper [README_GENERATOR](./charts/Earthfile)
+- Global values structure accross all charts, including settings
+  - Monitoring
+    - Traces(OTLP)
+    - Metrics(OTLP)
+    - Logs(JSON)
+  - Storage
+    - PostgreSQL (Bitnami) (Internal or External)
+    - Nats (Nats.io) (Internal)
+
+- (Optionals):
+
+  - AWS IAM
+  - AWS Target Groups
+  - Ingress
+  - PodDisruptionBudget
+  - HorizontalPodAutoscaler
+  - ServiceAccount
+  - RBAC
+  - NetworkPolicy
 
-### Core Dependencies
+## Validate repository changes
 
-Each chart must implement the core helm library as a dependency to include the common helper:
+To validate the all the changes arround the repository, run the following command:
 
-- resources naming 
-- values naming
-- values structure
-- kubernetes recommended labels
-- (optional): aws, tgb, ingress, pdp, hpa
+```bash
+earthly +pre-commit
+```
+
+1. First run
+once running it for the first time, it will build all the dependencies and validate all the charts and the charts.
+
+2. Second run
+It will validate the future changes only where it needs to be validated thanks to caches.
 
-## CI: Github Actions
+## Tests core and charts
 
-The CI is based on github actions and it is triggered on each PR and push to the main branch.
+- Naming conventions for included resource
+- Labels selection
+- Default environment variables bindings
+- Resources disablings
+  - HorizontalPodAutoscaler
+  - Ingress
+  - PodDisruptionBudget
+  - Subchart disabling
+- Secret mapping
+- Configmap mapping
+- Managed Stacks Features: Disable GRPC communication with any type of Agent
 
-The CI is composed by the following workflows:
+## CI: GitHub Actions
 
-- Labeler:
-  - It checks for changes in the `charts/` folder and labels the PR with the chart name
-- Main: 
-  - Validate the PR name
-  - Check if readme has been regenerated
-  - Validate & Package each chart labeled in the PR
-- Release:
-  - Use chart-releaser to release the charts where the PR has been merged on main. It will create a new tag with the version of the chart and release it to the helm repo.
+The CI is based on GitHub Actions, triggered on each PR and the main branch. It is composed of the following workflows:
 
-## CD
+- **Pull Request**: PR
+  - Validates the PR name.
+  - Labels the PR with the charts who have changed in the `charts/` directory.
+  - Lint, Template, Generate Readmes for any charts who has changed. And Test Requirements accross on all charts `earthly +ci`.
+- **Release**: Main
+    - Lint, Template, Generate Readmes for any charts who has changed. And Test Requirements accross on all charts `earthly +ci`.
+  - Release any `Chart.yaml` `.version` that have been upgraded. 
+    - `chart-releaser` is based on builded Artifact. - It creates a new tag with the chart version and releases on github it to the Helm repository.
 
-External repository can relies on the `+package` target and artifact to be able to deploy from specific branch or tag.
+## CD: from sources
+
+External repositories can rely on the `+package` target and artifact to deploy from a specific branch or tag.
 
 ```bash
 earthly github.com/formancehq/helm/charts/cloudprem+package
 ```
 
-<!-- Each chart are published to the [Artifact HUB](https://artifacthub.io/packages/search?repo=formancehq) and can be installed with helm:
+## CD: from Helm repository
 
-```bash
-helm repo add formancehq https://formancehq.github.io/helm
-helm install formancehq/cloudprem
-``` -->
+The helm repository is `ghcr.io/formancehq/helm` and can be used to deploy the charts.
 
+```bash
+helm upgrade --install regions ghcr.io/formancehq/helm/regions --version v2.0.18
+```
diff --git a/Earthfile b/Earthfile
@@ -3,31 +3,27 @@ VERSION 0.8
 IMPORT ./charts AS charts
 IMPORT github.com/formancehq/earthly:tags/v0.16.2 AS core
 
-license:
+sources:
+  ARG --required PATH
   FROM core+base-image
   WORKDIR /src
-  COPY ./LICENSE .
-  SAVE ARTIFACT LICENSE
-
-formance-runner:
-  FROM earthly/earthly:latest
-  RUN apk add --no-cache bash
-  ARG REPOSITORY=ghcr.io
-  ARG tag=latest
-  ENTRYPOINT /bin/bash
-  DO core+SAVE_IMAGE --COMPONENT=formance-runner --REPOSITORY=${REPOSITORY} --TAG=$tag
+  COPY --dir ./${PATH} .
+  SAVE ARTIFACT /src
 
 readme:
-  FROM core+builder-image
+  FROM core+base-image
+  RUN apk add go
+  RUN touch README.md
   COPY --dir charts /charts
   COPY (./tools/readme+sources/*) /src
   WORKDIR /src
-  RUN touch README.md
-  RUN go run ./ readme --chart-dir /charts  >> README.md
+  RUN --mount=type=cache,id=gomod,target=${GOPATH}/pkg/mod \
+      --mount=type=cache,id=gobuild,target=/root/.cache/go-build \ 
+    go run ./ readme --chart-dir /charts  >> README.md
   SAVE ARTIFACT README.md AS LOCAL README.md
 
 validate:
-  LOCALLY
+  FROM core+base-image
   FOR chart IN $(ls -d ./charts/*/)
     BUILD $chart+validate
   END
@@ -36,7 +32,7 @@ tests:
   BUILD ./test/helm+tests
 
 package:
-  LOCALLY
+  FROM core+base-image
   FOR chart IN $(ls -d ./charts/*/)
     BUILD $chart+package
   END

diff --git a/README.md b/README.md
@@ -2,7 +2,7 @@
 
 ## How to use Helm charts
 
-| Charts | Chart Version | App Version | Description | Hub |
+| Readme | Chart Version | App Version | Description | Hub |
 |--------|---------------|-------------|-------------|-----|
 | [Cloudprem](./charts/cloudprem/README.md) | v2.0.0-beta.11 |v0.35.3 | Formance control-plane | [![Artifact HUB](https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/cloudprem)](https://artifacthub.io/packages/search?repo=cloudprem) |
 | [Console](./charts/console/README.md) | v1.0.0-beta.1 |9431e5f4b4b1a03cb8f02ef1676507b9c023f2bb | Formance Console | [![Artifact HUB](https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/console)](https://artifacthub.io/packages/search?repo=console) |

diff --git a/charts/Earthfile b/charts/Earthfile
@@ -1,7 +1,7 @@
 VERSION 0.8
 
 IMPORT github.com/formancehq/earthly:tags/v0.16.3 AS core
-# IMPORT ../.. AS root
+IMPORT .. AS root
 
 BASE:
   FUNCTION
@@ -10,10 +10,9 @@ BASE:
 
 SOURCES:
   FUNCTION
-  # COPY (root+license/*) .
+  COPY (root+sources/* --PATH=LICENSE) .
   COPY --if-exists Chart.lock .
   COPY Chart.yaml .
-  COPY --if-exists README.md .
   COPY --if-exists values.yaml .
   COPY --dir --if-exists templates .
   SAVE ARTIFACT ./* AS LOCAL ./
@@ -45,17 +44,6 @@ PACKAGE:
   RUN helm package .
   SAVE ARTIFACT *.tgz AS LOCAL ./
 
-# publish:
-#   ARG --required PROJECT
-#   FROM ./$PROJECT+package
-#   WITH DOCKER
-#     RUN --secret GITHUB_TOKEN echo $GITHUB_TOKEN | docker login ghcr.io -u NumaryBot --password-stdin
-#   END
-#   WITH DOCKER
-#     RUN helm push *.tgz oci://ghcr.io/formancehq/helm
-#   END
-
-
 README_GENERATOR:
   FUNCTION
   RUN helm-docs --chart-search-root=.. --document-dependency-values
@@ -71,6 +59,7 @@ readme-base:
   ARG HELM_DOCS_VERSION=v1.14
   ENV PATH $GOPATH/bin:/usr/local/go/bin:$PATH
   DO --pass-args core+GO_INSTALL --package=github.com/norwoodj/helm-docs/cmd/helm-docs@$HELM_DOCS_VERSION
+
 SCHEMA:
   FUNCTION
   COPY values.yaml .
@@ -82,11 +71,3 @@ schema-base:
   DO --pass-args +BASE
   RUN apk add --no-cache git
   RUN helm plugin install https://github.com/losisin/helm-values-schema-json.git
-
-increment-version:
-  FROM core+base-image
-  ARG --required CHART
-  WORKDIR /src
-  COPY $CHART/Chart.yaml .
-  RUN yq eval '.version = "$VERSION"' -i Chart.yaml
-  SAVE ARTIFACT Chart.yaml AS LOCAL $CHART/Chart.yaml
diff --git a/charts/cloudprem/Earthfile b/charts/cloudprem/Earthfile
@@ -4,22 +4,15 @@ IMPORT ../console AS console
 IMPORT ../membership AS membership
 IMPORT ../portal AS portal
 IMPORT ../core AS core
-IMPORT .. AS root
-
-# svgs:
-#   FROM minlag/mermaid-cli
-#   COPY --dir build /data
-#   RUN mkdir -p /data/svg
-#   RUN /home/mermaidcli/node_modules/.bin/mmdc -p /puppeteer-config.json -i /data/build/mmd/controlplane.mmd -o /data/svg/controlplane.svg --configFile /data/build/config.json
-#   SAVE ARTIFACT /data/svg/controlplane.svg AS LOCAL ./controlplane.svg
+IMPORT .. AS charts
 
 schema:
-  FROM root+schema-base
+  FROM charts+schema-base
   WORKDIR /src
-  DO --pass-args root+SCHEMA
+  DO --pass-args charts+SCHEMA
 
 readme:
-  FROM root+readme-base
+  FROM charts+readme-base
   WORKDIR /src/core
   COPY --dir (core+dependencies/*) .
 
@@ -33,15 +26,15 @@ readme:
   WORKDIR /src/cloudprem
   COPY (+dependencies/*) .
   COPY (+schema/*) .
-  # COPY (+svgs/*) .
   COPY *.gotmpl .
-  DO --pass-args root+README_GENERATOR
+  DO --pass-args charts+README_GENERATOR
+
 sources:
-  DO --pass-args root+BASE
+  DO --pass-args charts+BASE
   WORKDIR /src/cloudprem
   COPY --dir profiles .
   COPY (+schema/*) .
-  DO --pass-args root+SOURCES
+  DO --pass-args charts+SOURCES
 
 dependencies:
   FROM +sources
@@ -52,20 +45,15 @@ dependencies:
   WORKDIR /src/portal
   COPY --dir (portal+dependencies/*) .
   WORKDIR /src/cloudprem
-  DO --pass-args root+DEPENDENCIES
+  DO --pass-args charts+DEPENDENCIES
 
 validate:
   FROM +dependencies
   WORKDIR /src/cloudprem
   COPY (+readme/*) .
-  DO --pass-args root+VALIDATE
+  DO --pass-args charts+VALIDATE
 
 package:
   FROM +validate
   WORKDIR /src/cloudprem
-  DO --pass-args root+PACKAGE
-
-publish:
-  FROM +package
-  WORKDIR /src/cloudprem
-  DO --pass-args root+PUBLISH
+  DO --pass-args charts+PACKAGE
diff --git a/charts/cloudprem/README.md b/charts/cloudprem/README.md
@@ -33,8 +33,6 @@ helm install cloudprem oci://ghcr.io/formancehq/helm/cloudprem \
 - Membership > 0.28.0
 - Dex > 0.28.0
 
-<!-- ![diagram](./controlplane.svg) -->
-
 ## Introduction
 
 This chart bootstraps 5 different components that form the Formance Control Plane, additionally you will need to install the Formance Data Plane composed of a Kubernetes Operator.

diff --git a/charts/cloudprem/README.md.gotmpl b/charts/cloudprem/README.md.gotmpl
@@ -33,8 +33,6 @@ helm install cloudprem oci://ghcr.io/formancehq/helm/cloudprem \
 - Membership > 0.28.0
 - Dex > 0.28.0
 
-<!-- ![diagram](./controlplane.svg) -->
-
 ## Introduction
 
 This chart bootstraps 5 different components that form the Formance Control Plane, additionally you will need to install the Formance Data Plane composed of a Kubernetes Operator.