Helm Chart Value Inheritance, Templating, and Documentation Issues

[con5cience]

I am not and will not use the tobs CLI because I can't put any of that in my Infrastructure-As-Code repo to manage and version the config. I'm using Pulumi to (try to) deploy this chart:

https://www.pulumi.com/registry/packages/kubernetes/api-docs/helm/v3/chart/

My code looks like this at the moment:

import { helm } from '@pulumi/kubernetes'

new helm.v3.Chart(
            "prometheus",
            {
                chart: "tobs",
                repo: "timescale",
                namespace: "prometheus",
                version: "0.8.0",
                values: {
                    namespaceOverride: "prometheus",
                    "timescaledb-single": {
                        enabled: false
                    },
                    promscale: {
                        connectionSecretName: "prometheus-stage",
                        connection: {
                            uri: "", // just in case it isn't clear to the template
                            host: "redacted.cloud.timescale.com",
                            port: redacted,
                            database: "redacted",
                            user: "redacted",
                            password: "redacted"
                        }
                    },
                    "kube-prometheus-stack": {
                        enabled: true,
                        grafana: {
                            enabled: true,
                            timescale: {
                                database: {
                                    enabled: true,
                                    host: "redacted.tsdb.cloud.timescale.com",
                                    user: "redacted",
                                    pass: "redacted",
                                    port: redacted,
                                    dbName: "redacted",
                                },
                                adminUser: "redacted",
                                adminPassSecret: "prometheus-stage"
                            }
                        }
                    }
                }
            }
)

On to the ordeals. I'll try to keep the salt to a minimum, but I've burned a non-trivial amount of time trying to reverse-engineer how this chart works so I can provide an excellent monitoring and metric visualization experience for my stakeholders, and I am not at all happy with the experience I've had with tobs.

1. Apparently, when you configure promscale.connectionSecretName, the Helm chart pulls all connection strings from the configured Secret and completely ignores everything in the promscale.connection block. This is not clear from the documentation, nor is it clear how to properly structure the keyvals in the Secret from the documentation alone.

Ok. Decipher how everything should be named and shove all the connection strings into the Secret for now, then, because trying to use promscale.connection.uri also triggers errors for a completely new path of undocumented values that the chart seems to expect (starting with tobs.fullname).

2. Next, kube-prometheus-stack.grafana.timescale.adminPassSecret doesn't populate PGPASSWORD on the Pod that the grafana-db Job launches. PGPASSWORD: <null> on the Pod description. The Secret is properly configured, there's no RBAC in the way, everything else depending on the Secret populates and executes fine up to this point, but this Job never gets the password and throws an error accordingly in the logs (no password supplied).

Ok. Override the password directly at runtime and we'll figure out security later.

3. Next, the prometheus-promscale Pods don't inherit any of the port overrides properly with promscale.connection.port or promscale.connectionSecretName. So I can't point Promscale at my hosted TimescaleDB instance which isn't running on 5432, and I don't have the ability to specify the port at provision time in the Timescale hosted service to force 5432 on the server side.

guess_ill_die.jpeg

The documentation is severely lacking. Populate a Secret with a keyvals, it says, but it doesn't specify how to structure the data. There are malformed template paths all over the place that do not at all match how the template logic actually works. I have to dig around in the template files to figure out how to properly form objects/template paths to pass overrides for values.

Ok. I'll just not use this chart then and instead string together individual deployments of Prometheus, the Promscale Connector, and Grafana.

Wish me luck.

[RupertBothma]

Having the same issue using the default values for tobs.

A describe on the job gives me the following.
Name: tobs-grafana-db
Namespace: taco
Selector: controller-uid=57b44921-b3d2-448e-bed1-facacc3a66ca
Labels: app=tobs
app.kubernetes.io/managed-by=Helm
chart=tobs-0.8.0
heritage=Helm
release=tobs
Annotations: meta.helm.sh/release-name: tobs
meta.helm.sh/release-namespace: taco
Parallelism: 1
Completions: 1
Start Time: Thu, 10 Feb 2022 12:05:59 +0200
Pods Statuses: 0 Running / 0 Succeeded / 1 Failed
Pod Template:
Labels: controller-uid=57b44921-b3d2-448e-bed1-facacc3a66ca
job-name=tobs-grafana-db
Init Containers:
init-db:
Image: busybox:1.28
Port:
Host Port:
Command:
sh
/wait-for-ts.sh
Environment:
PGHOST: tobs.taco.svc.cluster.local
Mounts:
/wait-for-ts.sh from sql-volume (rw,path="wait-for-ts.sh")
Containers:
tobs-grafana-db:
Image: postgres:12-alpine
Port:
Host Port:
Command:
psql
-d
postgres
-f
/add-users.sql
Environment:
PGPORT: 5432
PGUSER: postgres
PGPASSWORD:
PGHOST: tobs.taco.svc.cluster.local
Mounts:
/add-users.sql from sql-volume (rw,path="add-users.sql")
Volumes:
sql-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: tobs-grafana-db
Optional: false

And the logs for the pod give me:
psql: error: fe_sendauth: no password supplied

This is by using the default values file. What needs to be changed for it to pass the right password to PGPASSWORD: ?

[VineethReddy02]

Thanks for sharing this invaluable feedback @con5cience and @RupertBothma this definitely helps us in making tobs better.

We are currently working on stabilizing tobs i.e. fixing all the issues reported as the primary objective (due to some dependency changes in the recent 0.8.0 release of tobs. We see ambiguity and issues in the project). I will make sure to get them fixed for the upcoming release.

The team is currently trying to understand the end-user use-cases in deploying and managing the Observability stack to further understand how to shape the tobs as a project at the moment. Today tobs has two paths the CLI and helm-chart, we are re-thinking the architecture of the project.

@con5cience, @RupertBothma would be interested in helping us understand your use-cases and how you intend tobs to better fit your use-cases by joining a quick feedback call? Your feedback will definitely help us to better understand the priorities and future of tobs.

Thanks again for all the detailed feedback, The team will definitely work on the feedback the earliest!

[RupertBothma]

We are using the helm install command with the default values. The cli install method works but the most recent helm chart and default values give the password errors. I've gone through the commits and seen that the grafana db job has had a couple of changes. We are using terraform to install the helm chart on EKS.

[con5cience]

My use case is monitoring infrastructure with a TimescaleDB-backed datastore using frameworks that I can integrate into a versionable Infrastructure-As-Code implementation. The tobs CLI is not ideal for that.

I can jump on a call if you want, I'm also happy to asynchronously answer any questions you might have here in the clear.

[VineethReddy02]

We really appreciate the detail you shared in the first post here @con5cience and would really like to set up a call. We aren't able to contact you directly through GitHub so would you be open to dropping me an email with a few options of days and times that might suit you? (vineeth@timescale.com) Alternatively, you'd be more than welcome to pick a day and time that works for you from here (https://calendly.com/vineeth97/tobs-user-feedback).

Thanks again!