server returned HTTP status 401 Unauthorized with valid serviceaccount token

[shubhjoshi-dgs]

Running below command gives me metrics for specific job but when the same bearer token is getting used in scrape job throws "server returned HTTP status 401 Unauthorized" error with target down.

This command with same token used gives me all the metrics - curl -k -H "Authorization: Bearer $(cat /etc/prometheus/app-cluster/token/token.json)" https://10.69.192.16:10250/metrics

I have attached the prometheus config.

Also, only below targets (jobs) are failing:-

• serviceMonitor/monitoring/prometheus-kube-prometheus-kubelet/3 (0/8 up)
• serviceMonitor/monitoring/prometheus-kube-prometheus-kubelet/4 (0/8 up)
• serviceMonitor/monitoring/prometheus-kube-prometheus-kubelet/5 (0/8 up)

this is working:-

• serviceMonitor/monitoring/prometheus-kube-state-metrics/1
• serviceMonitor/monitoring/prometheus-prometheus-node-exporter/1
  global.txt