Description
On osparc-master.speag.com, we discovered so-far untackled issue raised in the tempo logs.
TRACE_TOO_LARGE Warnings
The most prevalent issue in the logs is the repeated TRACE_TOO_LARGE warnings, which indicate that incoming traces exceed the configured size limits. These warnings appear consistently throughout the log with messages like:
level=warn caller=rate_limited_logger.go:38 tenant=single-tenant msg=TRACE_TOO_LARGE max=5000000 size=7012
Nature and Origin: This warning occurs when individual traces exceed the maximum trace size limit configured in Tempo. The default maximum trace size is typically 5MB (5,000,000 bytes), and the logs show traces ranging from approximately 5KB to 14KB in size. While these individual traces are not actually exceeding the limit, the high frequency of these warnings suggests either a misconfiguration or an issue with trace size calculation.
Root Cause Analysis: The rate_limited_logger.go component is designed to prevent log spam by rate-limiting repeated messages. The consistent appearance of these warnings indicates that Tempo is either miscalculating trace sizes or there's an issue with the trace size validation logic.
Configuration Solutions:
Adjust Maximum Trace Size: Increase the maximum trace size in your Tempo configuration:
text
server:
http_listen_port: 3200
grpc_listen_port: 9095
distributor:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
limits:
max_trace_size_bytes: 10485760 # Increase to 10MB
Configure Trace Sampling: Implement sampling to reduce trace volume:
text
distributor:
sampling:
strategies_file: /etc/tempo/sampling.json