Documentation & updates for installer plugin (#49)

* rename directory

* document plugin usage

* simplify ATLAS & IceCube configs

* Update helm docs

* add respositories only when corresponding subcharts are enabled

* fix docs

* explain plugin flags

* Update helm docs

---------

Co-authored-by: GitHub Actions <[email protected]>

Author: kondratyevd

README.md

@@ -31,14 +31,19 @@ The main components of SuperSONIC are:
 
 ## Installation
 
+The installation is done via a custom Helm plugin which takes care of
+internal connectivity of the chart components. Standard Helm installation
+is also supported, but requires a lot more manual configuration.
+
 ```
-helm repo add fastml https://fastmachinelearning.org/SuperSONIC
-helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
-helm repo add grafana https://grafana.github.io/helm-charts
-helm repo update
-helm install <release-name> fastml/supersonic --values <your-values.yaml> -n <namespace>
+helm plugin install https://github.com/fastmachinelearning/SuperSONIC/
+helm install-supersonic <release-name> -n <namespace> -f <your-values.yaml>
 ```
 
+The new `helm install-supersonic` command accepts the same flags that can be passed to `helm install` command, and two additional flags:
+- `--local`: if set, the chart will be installed from the local directory specified by `--path` flag; if not set, the latest released version will be installed from the FastML Helm repository.
+- `--path`: optional path to the local chart directory (default if not set: `./helm/supersonic`).
+
 To construct the `values.yaml` file for your application, follow [Configuration guide](http://fastmachinelearning.org/SuperSONIC/configuration-guide.html "Configuration guide").
 
 The full list of configuration parameters is available in the [Configuration reference](http://fastmachinelearning.org/SuperSONIC/configuration-reference.html "Configuration reference").

docs/advanced-monitoring.rst

@@ -51,6 +51,7 @@ will include an interactive server map, where you can study tracing data in deta
 by clicking on graph nodes.
 
 You can also browse and display tracing data in Grafana as follows:
+
 1. Create a new panel
 2. Select "Tempo" as the data source
 3. Select "Search" or "TraceQL" as the query type and switch panel to "Table view".
@@ -64,6 +65,8 @@ You can also browse and display tracing data in Grafana as follows:
     :width: 80%
     :alt: Grafana Tracing 1
 
+|
+
 .. image:: https://raw.githubusercontent.com/fastmachinelearning/SuperSONIC/main/docs/img/grafana_tracing_2.png
     :align: center
     :width: 80%

docs/getting-started.rst

@@ -19,20 +19,31 @@ Installation
       - `Configuration reference <configuration-reference>`_
       - `Example values.yaml files <https://github.com/fastmachinelearning/SuperSONIC/tree/main/values>`_
 
-   2. Add Helm repositories for SuperSONIC and its dependencies:
+   2. Install Helm plugin to handle SuperSONIC installation
 
       .. code:: shell
 
-         helm repo add fastml https://fastmachinelearning.org/SuperSONIC
-         helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
-         helm repo add grafana https://grafana.github.io/helm-charts
-         helm repo update
+         helm plugin install https://github.com/fastmachinelearning/SuperSONIC/
+
+
+      The Helm plugin is needed to ensure internal connectivity of the SuperSONIC
+      components. Standard Helm installation without a plugin is also supported,
+      but requires a lot more manual configuration.
 
    3. Modify the following command to install the chart at your cluster:
 
       .. code:: shell
 
-         helm install <release-name> fastml/supersonic --values <your-values.yaml> -n <namespace>
+         helm install-supersonic <release-name> -n <namespace> -f <your-values.yaml>
+
+      The new ``helm install-supersonic`` command accepts the same flags that
+      can be passed to ``helm install`` command, and two additional flags:
+
+      - ``--local``: if set, the chart will be installed from the local directory
+        specified by ``--path`` flag; if not set, the latest released version will
+        be installed from the FastML Helm repository.
+      - ``--path``: optional path to the local chart directory
+        (default if not set: ``./helm/supersonic``).
 
       Use a unique meaningful lowercase value as <release-name>, for example
       ``supersonic-cms-run3``.

helm/supersonic/README.md

@@ -31,14 +31,19 @@ The main components of SuperSONIC are:
 
 ## Installation
 
+The installation is done via a custom Helm plugin which takes care of
+internal connectivity of the chart components. Standard Helm installation
+is also supported, but requires a lot more manual configuration.
+
 ```
-helm repo add fastml https://fastmachinelearning.org/SuperSONIC
-helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
-helm repo add grafana https://grafana.github.io/helm-charts
-helm repo update
-helm install <release-name> fastml/supersonic --values <your-values.yaml> -n <namespace>
+helm plugin install https://github.com/fastmachinelearning/SuperSONIC/
+helm install-supersonic <release-name> -n <namespace> -f <your-values.yaml>
 ```
 
+The new `helm install-supersonic` command accepts the same flags that can be passed to `helm install` command, and two additional flags:
+- `--local`: if set, the chart will be installed from the local directory specified by `--path` flag; if not set, the latest released version will be installed from the FastML Helm repository.
+- `--path`: optional path to the local chart directory (default if not set: `./helm/supersonic`).
+
 To construct the `values.yaml` file for your application, follow [Configuration guide](http://fastmachinelearning.org/SuperSONIC/configuration-guide.html "Configuration guide").
 
 The full list of configuration parameters is available in the [Configuration reference](http://fastmachinelearning.org/SuperSONIC/configuration-reference.html "Configuration reference").

install-supersonic/installer.py renamed to installer-plugin/installer.py

@@ -92,12 +92,15 @@ def main() -> None:
         chart_source = args.path if args.local else REPO_CHART
 
         # Add dependencies
-        repo_commands = [
-            ["helm", "repo", "add", "prometheus-community", "https://prometheus-community.github.io/helm-charts"],
-            ["helm", "repo", "add", "grafana", "https://grafana.github.io/helm-charts"],
-            ["helm", "repo", "add", "opentelemetry", "https://open-telemetry.github.io/opentelemetry-helm-charts"],
-            ["helm", "dependency", "build", chart_source]
-        ]
+        repo_commands = []
+        if merged_values.get("prometheus", {}).get("enabled", False):
+            repo_commands.append(["helm", "repo", "add", "prometheus-community", "https://prometheus-community.github.io/helm-charts"])
+        if merged_values.get("grafana", {}).get("enabled", False):
+            repo_commands.append(["helm", "repo", "add", "grafana", "https://grafana.github.io/helm-charts"])
+        if merged_values.get("opentelemetry-collector", {}).get("enabled", False):
+            repo_commands.append(["helm", "repo", "add", "opentelemetry", "https://open-telemetry.github.io/opentelemetry-helm-charts"])
+        repo_commands.append(["helm", "dependency", "build", chart_source])
+
         for cmd in repo_commands:
             print(f"\nExecuting: {' '.join(cmd)}")
             subprocess.run(cmd, check=True)

install-supersonic/overrides.py renamed to installer-plugin/overrides.py

@@ -2,8 +2,8 @@ name: "install-supersonic"
 version: "0.1.0"
 usage: "Installer for SuperSONIC Helm chart"
 description: "A Helm plugin to install SuperSONIC chart with automatic values merging and template replacement"
-command: "./install-supersonic/installer.py"
+command: "./installer-plugin/installer.py"
 hooks:
   install: |
     #!/bin/sh
-    chmod +x ./install-supersonic/installer.py
+    chmod +x ./installer-plugin/installer.py

install-supersonic/utils.py renamed to installer-plugin/utils.py

@@ -69,37 +69,16 @@ prometheus:
 
 grafana:
   enabled: true
-  datasources:
-    datasources.yaml:
-      apiVersion: 1
-      datasources:
-        - name: prometheus
-          type: prometheus
-          access: proxy
-          isDefault: true
-          url: https://prometheus.nrp-nautilus.io
-          jsonData:
-            timeInterval: "5s"
-            tlsSkipVerify: true
+  host: https://grafana-atlas.nrp-nautilus.io
   ingress:
     enabled: true
-    hosts:
-      - grafana-atlas.nrp-nautilus.io
-    tls:
-      - hosts:
-          - grafana-atlas.nrp-nautilus.io
     ingressClassName: haproxy
     annotations:
       haproxy-ingress.github.io/cors-enable: "true"
-      haproxy-ingress.github.io/backend-protocol: "h2"
       haproxy-ingress.github.io/proxy-body-size: "512m"
-  grafana.ini:
-    server:
-      root_url: https://grafana-atlas.nrp-nautilus.io
-
+      haproxy-ingress.github.io/timeout-http-request: "5m"
 metricsCollector:
   enabled: true
-
   ingress:
     enabled: true
     hostName: metrics-collector-atlas.nrp-nautilus.io
@@ -115,3 +94,4 @@ metricsCollector:
       haproxy-ingress.github.io/proxy-body-size: "512m"
       haproxy-ingress.github.io/ssl-redirect: "true"
       haproxy-ingress.github.io/secure-backends: "false"
+

plugin.yaml

@@ -87,34 +87,11 @@ prometheus:
 
 grafana:
   enabled: true
-  datasources:
-    datasources.yaml:
-      apiVersion: 1
-      datasources:
-        - name: prometheus
-          type: prometheus
-          access: proxy
-          isDefault: true
-          url: https://prometheus.nrp-nautilus.io
-          jsonData:
-            timeInterval: "5s"
-            tlsSkipVerify: true
-
-  dashboardsConfigMaps:
-    default: supersonic-grafana-default-dashboard
-
+  host: https://grafana-icecube.nrp-nautilus.io
   ingress:
     enabled: true
-    hosts:
-      - grafana-icecube.nrp-nautilus.io
-    tls:
-      - hosts:
-          - grafana-icecube.nrp-nautilus.io
     ingressClassName: haproxy
     annotations:
       haproxy-ingress.github.io/cors-enable: "true"
       haproxy-ingress.github.io/backend-protocol: "h2"
-      haproxy-ingress.github.io/proxy-body-size: "512m"
-  grafana.ini:
-    server:
-      root_url: https://grafana-icecube.nrp-nautilus.io
+      haproxy-ingress.github.io/proxy-body-size: "512m"