Grafana Tempo - Indicated Issue: Remote Write Receiver Errors

[mrnicegyu11]

On osparc-master.speag.com, we discovered so-far untackled issue raised in the tempo logs.

The logs show critical errors related to Prometheus remote write functionality:

err="server returned HTTP status 404 Not Found: remote write receiver needs to be enabled with --web.enable-remote-write-receiver"

Nature and Origin: These errors indicate that Tempo is attempting to send metrics to a Prometheus instance that doesn't have the remote write receiver enabled. The prometheuscatchall endpoint is returning 404 errors because Prometheus was not started with the necessary flag to accept remote write requests.

Root Cause: The Prometheus instance at http://prometheuscatchall:9090/api/v1/write is not configured to accept remote write requests. This is a common misconfiguration in containerized environments where Prometheus is expected to receive metrics from other services.

Configuration Solutions:

Enable Prometheus Remote Write Receiver: Start Prometheus with the required flag:

bash
prometheus --web.enable-remote-write-receiver --config.file=/etc/prometheus/prometheus.yml

Update Docker Compose Configuration: If using Docker Compose, modify the Prometheus service:

text
services:
  prometheus:
    image: prom/prometheus:latest
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--web.enable-remote-write-receiver'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'

Alternative: Disable Remote Write in Tempo: If metrics export to Prometheus is not required:

text
metrics_generator:
  storage:
    path: /tmp/tempo/generator/wal
  remote_write: []  # Disable remote write