Incorrect peak sample stats

[yeya24]

Query using promql engine with 1h range and 20min step, got peak samples 82678.

curl 'http://localhost:10902/api/v1/query_range?query=sum(up)&start=1748296266.667&end=1748299866.667&step=1200&engine=prometheus&stats=all' | jq .

{
  "status": "success",
  "data": {
    "resultType": "matrix",
    "result": [
      {
        "metric": {},
        "values": [
          [
            1748296266.667,
            "5987"
          ],
          [
            1748297466.667,
            "5982"
          ],
          [
            1748298666.667,
            "6181"
          ],
          [
            1748299866.667,
            "6466"
          ]
        ]
      }
    ],
    "stats": {
      "timings": {
        "evalTotalTime": 4.260582792,
        "resultSortTime": 7.5e-07,
        "queryPreparationTime": 4.0209e-05,
        "innerEvalTime": 4.260507667,
        "execQueueTime": 0,
        "execTotalTime": 4.260590167
      },
      "samples": {
        "totalQueryableSamples": 82678,
        "peakSamples": 82678
      }
    },
    "analysis": {}
  }
}

Same using thanos engine, got peak samples 1. It doesn't need to match what promql engine has but 1 is also incorrect. Step batch size is 10 and peak samples must > 1 in the vector selector.

curl 'http://localhost:10902/api/v1/query_range?query=sum(up)&start=1748296266.667&end=1748299866.667&step=1200&engine=thanos&stats=all' | jq .

{
  "status": "success",
  "data": {
    "resultType": "matrix",
    "result": [
      {
        "metric": {},
        "values": [
          [
            1748296266.667,
            "5987"
          ],
          [
            1748297466.667,
            "5982"
          ],
          [
            1748298666.667,
            "6181"
          ],
          [
            1748299866.667,
            "6466"
          ]
        ]
      }
    ],
    "stats": {
      "timings": {
        "evalTotalTime": 0,
        "resultSortTime": 0,
        "queryPreparationTime": 0,
        "innerEvalTime": 0,
        "execQueueTime": 0,
        "execTotalTime": 0
      },
      "samples": {
        "totalQueryableSamples": 82678,
        "peakSamples": 1
      }
    },
    "analysis": {}
  }
}

The issue could be in https://github.com/thanos-io/promql-engine/blob/main/storage/prometheus/vector_selector.go#L168C6-L168C33, we use samples per step to update peak samples instead of using total samples to update peak.

[yeya24]

A better way would be to separate steps of updating peak samples and incrementing samples at step in https://github.com/thanos-io/promql-engine/blob/main/execution/telemetry/telemetry.go#L101. So that we can use different values as peak and value at each step