RE: Monitoring configuration Alerts

[gohilamariappan]

Hi Team!
We have configured monitoring services and we are getting alerts as well. In that alerts few of them are notifying Target down but services are running fine.

@santhosh-tg Please suggest.

Regards,
Gohila

[Krishnaj20]

@gandham-santhosh - can you look into this thread?

[gandham-santhosh]

@gohilamariappan which branch you have used to setup monitoring? Can you check what are the targets mentioned in the prometheus.yaml ?
cc: @santhosh-tg

[gohilamariappan]

Hi @gandham-santhosh !

Am facing this issue in two environemnts. below are the versions release 3.8.0 and release 5.2.0.
below one is the prometheus.yaml, which is giving target down. But i have tried the targets manually , its workring.
########

• job_name: availability_service_search
  metrics_path: /probe
  params:
  jsonpath:
  • $.result
    relabel_configs:
  • source_labels:
    • address
      target_label: __param_target
  • source_labels:
    • __param_target
      target_label: instance
  • replacement: json-path-exporter.monitoring.svc.cluster.local:7979
    target_label: address
    static_configs:
  • labels:
    service_name: search
    targets:
    • http://search-service.dev.svc.cluster.local:9000/health
• job_name: availability_service_knowledgemw
  metrics_path: /probe
  params:
  jsonpath:
  • $.result
    relabel_configs:
  • source_labels:
    • address
      target_label: __param_target
  • source_labels:
    • __param_target
      target_label: instance
  • replacement: json-path-exporter.monitoring.svc.cluster.local:7979
    target_label: address
    static_configs:
  • labels:
    service_name: knowledgemw
    targets:
    • http://knowledge-mw-service.dev.svc.cluster.local:5000/health
• job_name: availability_service_content
  metrics_path: /probe
  params:
  jsonpath:
  • $.result
    relabel_configs:
  • source_labels:
    • address
      target_label: __param_target
  • source_labels:
    • __param_target
      target_label: instance
  • replacement: json-path-exporter.monitoring.svc.cluster.local:7979
    target_label: address
    static_configs:
  • labels:
    service_name: content
    targets:
    • http://content-service.dev.svc.cluster.local:9000/health
• job_name: availability_service_lms
  metrics_path: /probe
  params:
  jsonpath:
  • $.result
    relabel_configs:
  • source_labels:
    • address
      target_label: __param_target
  • source_labels:
    • __param_target
      target_label: instance
  • replacement: json-path-exporter.monitoring.svc.cluster.local:7979
    target_label: address
    static_configs:
  • labels:
    service_name: lms
    targets:
    • http://lms-service.dev.svc.cluster.local:9000/health
• job_name: availability_service_learner
  metrics_path: /probe
  params:
  jsonpath:
  • $.result
    relabel_configs:
  • source_labels:
    • address
      target_label: __param_target
  • source_labels:
    • __param_target
      target_label: instance
  • replacement: json-path-exporter.monitoring.svc.cluster.local:7979
    target_label: address
    static_configs:
  • labels:
    service_name: learner
    targets:
    • http://learner-service.dev.svc.cluster.local:9000/health
      ######################

Regards,
Gohila

[santhosh-tg]

Hi @gohilamariappan the 400 bad request could mean something wrong with the endpoint url. Can you check the endpoint url and see what response you are getting ? this is not the target url. The one below Endpoint in the screenshot you shared.

[gohilamariappan]

Hi @santhosh-tg

I have tried json exporter UI and tried the below endpoints from cluster.

/home/sunbird $ curl http://json-path-exporter.monitoring.svc.cluster.local:7979/probe
Unknown module "default"

curl http://json-path-exporter.monitoring.svc.cluster.local:7979/probe?target=http://lms-service.dev.svc.cluster.local:9000/health
Unknown module "default"

am troubleshooting this issue. still couldn't find any proper solution. @santhosh-tg Can you please suggest.

Regards,
Gohila

[santhosh-tg]

@gohilamariappan It looks like the endpoint that's configured isn't correct. If the same endpoints are configured to monitor in blackbox exporter and working, then i would suggest to disable this alerts until you figure out the root cause.

[gohilamariappan]

@santhosh-tg Thank you!