Thanos Receiver Endpoints as seen by Querier

[michael-burt]

Hello 👋

I have deployed Thanos to Kubernetes. When I log into the Querier web UI and view the different Stores, I only see a single Receiver replica, even though I have deployed 3 Receiver replicas. Is this expected? I am pointing the Querier at the Receivers using the following Store address:

dnssrv+_grpc._tcp.thanos-receive.monitoring.svc.cluster.local

Thanks,
Michael

[GiedriusS]

It seems more of a Kubernetes question than Thanos but I think Kubernetes services point to load-balancers so in your case as far as I know you get one IP that load-balances to all Thanos Receive instances. You should probably rework your setup so that Query would point to all pods.

[michael-burt]

Thanks @GiedriusS . You are right, this is specific to Kubernetes.

I am following the documentation from here: https://thanos.io/tip/thanos/service-discovery.md/#dns-service-discovery

DNS SRV record discovery also work well within Kubernetes. Consider the following example:

--store=dnssrv+_grpc._tcp.thanos-store.monitoring.svc
This configuration will instruct Thanos to discover all endpoints within the thanos-store service in the monitoring namespace and use the declared port named grpc.

Based on the above documentation, I would expect to see all of the Receiver endpoints, not just a single pod. I would like to understand if this is the expected behavior, or if I have a misconfiguration.

[GiedriusS]

Thanos Store is different from Thanos Receiver because Store is essentially stateless, and you can and should point it to the load-balanced service endpoint whereas it is completely different with Thanos Receiver. Just like with Sidecars, all Receivers need to be connected to the Thanos Query so that it would be able to properly perform deduplication :P

[michael-burt]

Thanks for the response.

I am not asking about the Thanos Store Gateway. We use a --store= argument on the Querier pod that points to the Receiver because the Receiver component implements the Thanos Store API, we need to be able to query the Head block on the Receiver pods prior to it getting uploaded to S3. You also use a similar --store argument on the Querier pod to talk to all sidecars (--store=dnssrv+_grpc._tcp.thanos-sidecar.monitoring.svc.cluster.local). This CLI command argument properly discovers all Thanos Sidecar Store API endpoints in the monitoring namespace.

From the documentation:

Thanos Receive exposes the StoreAPI so that Thanos Queriers can query received metrics in real-time.

The documentation also states that we should be able to use DNS service discovery to discover a list of IP addresses. I assume we can use the same pattern with the Receiver component:

With DNS SD, a domain name can be specified and it will be periodically queried to discover a list of IPs.

What is the correct way to query the Store API endpoints on the Receiver Pods from the Thanos Querier? Right now I am using dnssrv+_grpc._tcp.thanos-receive.monitoring.svc.cluster.local but I am only seeing a single Receiver pod from the hashring, I would expect to see all of the pods. Is this expected behavior?

[michael-burt]

I figured out the solution. When using a Kubernetes service in front of the Receiver pods, you must specify ClusterIP: 'None' in order to properly attach the Querier to all Receiver IPs.