Performance difference between v2.41.5 and 2.13.0 #3239
Description
Prerequisites
- I am using the latest version of Locust
- I am reporting a bug, not asking a question
Description
Hi,
We tried load testing on our services with v2.41.5, using the following command:
locust -f <...> --host=<...> --users 3000 --spawn-rate 3000 --run-time 60s --headless --processes 4
And then we tried the same load test with v2.13.0 with the following command:
Terminal 1
locust --master -f <...> --expect-workers=3 --headless --users 3000 --spawn-rate 3000 --run-time 60
Terminal 2
for ((i=1;i<=3;i++));
do
locust --worker --master-host=127.0.0.1 -f <...> &
done
We see that that there is a huge difference in reported performance: v2.41.5 shows p95 that is significantly slower (higher number) compared to using v2.13.0.
Is this a known issue, or are we using the new version incorrectly somehow?
Command line
locust -f <...> --host=<...> --users 3000 --spawn-rate 3000 --run-time 60s --headless --processes 4
Locustfile contents
from locust import HttpUser, task, constant_throughput
class TopicClassificationUser(HttpUser):
"""
Simple Locust user class for load testing the topic classification API endpoint.
"""
# Wait 1 second between requests (for 1 RPS per user)
wait_time = constant_throughput(1)
host = "http://....:8000" # purposely hidden IP
@task
def test_topic_classification(self):
"""
Single test task for topic classification API.
"""
# Prepare the payload
payload = {
"inputs": [
{
"name": "INPUT0",
"shape": [1, 1],
"datatype": "BYTES",
"data": ["what are movies"],
"parameters": {"content_type": "str"},
}
]
}
# Headers
headers = {"Content-Type": "application/json"}
# Make the request
with self.client.post(
"/v2/models/topic_classification/infer",
json=payload,
headers=headers,
catch_response=True,
) as response:
if response.status_code == 200:
try:
result = response.json()
# Check response structure
if self.validate_response(result):
response.success()
else:
response.failure("Response format validation failed")
except Exception as e:
response.failure(f"Invalid JSON response: {str(e)}")
else:
response.failure(f"HTTP {response.status_code}: {response.text}")
def validate_response(self, response_data):
"""
Validate that the response matches the expected format.
"""
try:
# Check required top-level fields
if "model_name" not in response_data:
print("Missing model_name field")
return False
if "model_version" not in response_data:
print("Missing model_version field")
return False
if "outputs" not in response_data:
print("Missing outputs field")
return False
# Check model_name and model_version values
if response_data["model_name"] != "topic_classification":
print(
f"Expected model_name 'topic_classification', got '{response_data['model_name']}'"
)
return False
if response_data["model_version"] != "1":
print(
f"Expected model_version '1', got '{response_data['model_version']}'"
)
return False
# Check outputs structure
outputs = response_data["outputs"]
if not isinstance(outputs, list) or len(outputs) == 0:
print("Outputs should be a non-empty list")
return False
output = outputs[0]
required_output_fields = ["name", "datatype", "shape", "data"]
for field in required_output_fields:
if field not in output:
print(f"Missing output field: {field}")
return False
# Check output field values
if output["name"] != "OUTPUT0":
print(f"Expected output name 'OUTPUT0', got '{output['name']}'")
return False
if output["datatype"] != "BYTES":
print(f"Expected datatype 'BYTES', got '{output['datatype']}'")
return False
if output["shape"] != [1]:
print(f"Expected shape [1], got {output['shape']}")
return False
# Check data structure
data = output["data"]
if not isinstance(data, list) or len(data) == 0:
print("Data should be a non-empty list")
return False
# Check that data contains JSON string with labels
data_item = data[0]
if not isinstance(data_item, str):
print("Data item should be a string")
return False
# Try to parse the JSON string in data
import json
try:
parsed_data = json.loads(data_item)
if not isinstance(parsed_data, list) or len(parsed_data) == 0:
print("Parsed data should be a non-empty list")
return False
item = parsed_data[0]
if "labels" not in item:
print("Missing 'labels' field in parsed data")
return False
labels = item["labels"]
if not isinstance(labels, list):
print("Labels should be a list")
return False
# Check for expected labels (Entertainment, Films)
if "Entertainment" not in labels or "Films" not in labels:
print(
f"Expected labels to contain 'Entertainment' and 'Films', got {labels}"
)
return False
except json.JSONDecodeError as e:
print(f"Failed to parse JSON in data: {e}")
return False
print(f"Response validation successful: {response_data}")
return True
except Exception as e:
print(f"Response validation error: {e}")
return FalsePython version
3.12 when using v2.41.5 of locust, and python 3.10.12 when using v2.13.0 of locust
Locust version
v2.41.5
Operating system
Canonical, Ubuntu, 22.04 LTS Minimal, amd64 jammy minimal image built on 2025-10-01