GMP not working with ASM istioside enabled pod

[sreejesh-radhakrishnan-db]

@pintohutch hope you are still the SME on this area. We are having a similar issue where ISTIO is running on 15090 and scrape away all the metric. but my app exposing on 8084 is not getting through to GPM.
I followed your step above and I could see this error :

Get "http://240.13.206.101:8084/actuator/prometheus": read tcp xxx.xxx.xxx.xxx:37516->xxx.xxx.xx.xxx:8084: read: connection reset by peer

basically collector not able to connect to the Application POD.

any help please?

also observed this error on pod -istio sidecar container

"textPayload": "[2024-05-10T15:48:49.862Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 0 - "-" "-" "-" "-" "-" - - 240.13.xxx.xxx:8084 240.13.xxx.xxx:51396 - -",

similar to #340 but no solution there

[pintohutch]

Hi @sreejesh-radhakrishnan-db - how do you have TLS configured for Istio?

[TheSpiritXIII]

Hi @sreejesh-radhakrishnan-db I've yet to test it out but GMP needs access to the Istio certificates. I'm not sure if you can get extract them easily, but you can give Istio your own certificates: https://istio.io/latest/docs/tasks/security/cert-management/plugin-ca-cert/

By then you should have your Istio certificates in a Kubernetes Secret. Finally, set the fields in the scrape endpoint.

We are working on public documentation for configuring TLS certificates in general!

[maxromanovsky]

@TheSpiritXIII I am trying to use tls config, but getting an error Error from server (BadRequest): error when creating "/xxx/pm.yaml": PodMonitoring in version "v1" cannot be handled as a PodMonitoring: strict decoding error: unknown field "spec.endpoints[0].tls" with the following manifest.

What am I doing wrong?

apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: test
  namespace: test
spec:
  endpoints:
  - interval: 30s
    path: /
    port: metrics
    tls:
      ca:
        secret:
          namespace: xxx
          name: yyy
          key: zzz.pem
  selector:
    matchLabels:
      app: aaa

I see these annotations for collector pods:

components.gke.io/component-name: managed-prometheus
components.gke.io/component-version: 0.8.3-gke.0

Does it need to be upgraded to v0.10.0?
I can't find any docs on how it can be upgraded manually without upgrading GKE to 1.27+.

[TheSpiritXIII]

@maxromanovsky hello! You need GMP v0.12.0, which is currently rolling out to new and upgraded GKE 1.28 clusters. What version are you currently on?

We plan to eventually backport v0.12.0 to GKE 1.27. GKE 1.26 will be end-of-life next month, so we will likely miss the cut-off.

[maxromanovsky]

@TheSpiritXIII we're planning to migrate to GKE 1.27 soon. Any ETA on 1.27 support?
Thanks in advance!

[TheSpiritXIII]

There's no ETA yet. By policy, we have to let it "soak" on 1.28 before we can put it on 1.27. I can circle back with the exact 1.27 version once we've backported!