huge out throughput

[gimphammer]

I run vllm in RTX-4090 with this parameter:
CUDA_VISIBLE_DEVICES=3 vllm serve Qwen/Qwen2.5-1.5B-Instruct --port 19999

And the parameter I used to start genai-bench is:
genai-bench benchmark --api-backend openai \ --api-base "http://localhost:19999" \ --api-key "test" \ --api-model-name "Qwen/Qwen2.5-1.5B-Instruct" \ --model-tokenizer "Qwen/Qwen2.5-1.5B-Instruct" \ --task text-to-text \ --max-time-per-run 15 \ --max-requests-per-run 300 \ --server-engine "vLLM" \ --server-version "v0.9.2"

But, I got the huge out throughput at the level of 200k tokens/s as screenshot below:

as you can see the metrics in "Out throughput" is about 100k~200k tokens/s.
why it's so huge? it's abnormal

in the sub-dashboard of "Output Latency vs Output Throughput of Server"
the “Output Throughput of Server” value is 126 tokens/sec. This value matches vllm's log well, which is about 131 tokens/sec

so, is it an issue for the sub-dashboard of "Out throughput"?

[gimphammer]

To avoid no metrics by limitation, this code-routine has been modified:

$ gd genai_bench/metrics/aggregated_metrics_collector.py
diff --git a/genai_bench/metrics/aggregated_metrics_collector.py b/genai_bench/metrics/aggregated_metrics_collector.py
index 143e65f..5215b66 100644
--- a/genai_bench/metrics/aggregated_metrics_collector.py
+++ b/genai_bench/metrics/aggregated_metrics_collector.py
@@ -71,7 +71,8 @@ class AggregatedMetricsCollector:
     def _should_filter_metrics(metrics: RequestLevelMetrics) -> bool:
         """Filters metrics based on out of range TPOT/inference speed."""
         inference_speed = metrics.output_inference_speed
-        if inference_speed is not None and inference_speed > 1000:
+        if inference_speed is not None and inference_speed > 500000:
             logger.warning(
                 f"Metric has abnormal inference speed: {inference_speed} tokens/s."
                 " Filtering it out."

otherwise, there are no any metrics result in the dashboard

[YouNeedCryDear]

@gimphammer According to the output latency in the picture, there might be something wrong with the model's generation. The scenario was supposed to output 200 tokens but all the requests somehow finished the generation within 0.02s.
The sub-dashboard is showing the record for the preview scenario, not the current D(2000,200) you are running.
Could you try to turn on the debug logging to see what's going on? GENAI_BENCH_LOGGING_LEVEL=DEBUG

[gimphammer]

@gimphammer According to the output latency in the picture, there might be something wrong with the model's generation. The scenario was supposed to output 200 tokens but all the requests somehow finished the generation within 0.02s. The sub-dashboard is showing the record for the preview scenario, not the current D(2000,200) you are running. Could you try to turn on the debug logging to see what's going on? GENAI_BENCH_LOGGING_LEVEL=DEBUG

let me try again