Metrics: Record incoming request sizes #4495

guscarreon · 2025-08-20T16:10:07Z

Adds a metric to keep track of the size in bytes of requests that hit the /auction, /amp and /video endpoints. Should we add the cookie_sync endpoint?

Prometheus

To anyone who reviews, please let me know if you agree with the buckets (1000, 5000, 10000, 15000, 20000, 40000, 75000, and 100000 bytes respectively)

  pbs_sub_request_queue_time_bucket{request_status="requestRejectedLabel",request_type="video",le="240"} 0
  pbs_sub_request_queue_time_bucket{request_status="requestRejectedLabel",request_type="video",le="300"} 0
  pbs_sub_request_queue_time_bucket{request_status="requestRejectedLabel",request_type="video",le="+Inf"} 0
  pbs_sub_request_queue_time_sum{request_status="requestRejectedLabel",request_type="video"} 0
  pbs_sub_request_queue_time_count{request_status="requestRejectedLabel",request_type="video"} 0
+ # HELP pbs_sub_request_size_bytes Count that keeps track of incoming request size in bytes labeled by endpoint.
+ # TYPE pbs_sub_request_size_bytes histogram
+ pbs_sub_request_size_bytes_bucket{request_endpoint="auction",le="1000"} 1
+ pbs_sub_request_size_bytes_bucket{request_endpoint="auction",le="5000"} 1
+ pbs_sub_request_size_bytes_bucket{request_endpoint="auction",le="10000"} 1
+ pbs_sub_request_size_bytes_bucket{request_endpoint="auction",le="15000"} 1
+ pbs_sub_request_size_bytes_bucket{request_endpoint="auction",le="20000"} 1
+ pbs_sub_request_size_bytes_bucket{request_endpoint="auction",le="40000"} 1
+ pbs_sub_request_size_bytes_bucket{request_endpoint="auction",le="75000"} 1
+ pbs_sub_request_size_bytes_bucket{request_endpoint="auction",le="100000"} 1
+ pbs_sub_request_size_bytes_bucket{request_endpoint="auction",le="+Inf"} 1
+ pbs_sub_request_size_bytes_sum{request_endpoint="auction"} 809
+ pbs_sub_request_size_bytes_count{request_endpoint="auction"} 1
  # HELP pbs_sub_request_time_seconds Seconds to resolve successful Prebid Server requests labeled by type.
  # TYPE pbs_sub_request_time_seconds histogram
  pbs_sub_request_time_seconds_bucket{request_type="amp",le="0.05"} 0
  pbs_sub_request_time_seconds_bucket{request_type="amp",le="0.1"} 0

bsardo

I don't think we need the metric for any other endpoints. Auction, amp and video endpoints are sufficient.

bsardo · 2025-08-21T23:49:06Z

metrics/prometheus/prometheus.go

@@ -168,6 +170,7 @@ func NewMetrics(cfg config.PrometheusMetrics, disabledMetrics config.DisabledMet
 	priceBuckets := []float64{250, 500, 750, 1000, 1500, 2000, 2500, 3000, 3500, 4000}
 	queuedRequestTimeBuckets := []float64{0, 1, 5, 30, 60, 120, 180, 240, 300}
 	overheadTimeBuckets := []float64{0.05, 0.06, 0.07, 0.08, 0.09, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1}
+	requestSizeBuckets := []float64{1000, 5000, 10000, 15000, 20000, 40000, 75000, 100000}


How were these buckets chosen? Are they based off of any real requests? Have we seen at least one request that falls into each of these buckets or at least most of them?

This is difficult to say... My test request that invokes a lot of request-related features (Stored bid responses, FPD, multi-type imp, category mapping, etc) takes ~4500 bytes.
Should we first see how production data will look like and adjust buckets accordingly?

How were these buckets chosen?

Arbitrarily. These have to be discussed and carefully chosen first.

VeronikaSolovei9 · 2025-08-22T04:43:47Z

metrics/prometheus/prometheus.go

+	endpoint := metrics.GetEndpointFromRequestType(labels.RType)
+	m.requestsSize.With(prometheus.Labels{
+		requestEndpointLabel: string(endpoint),
+	}).Observe(float64(labels.RequestSize))
+


I tested this PR with AMP endpoint and found that if stored request was not found and request Len was never measured, we still write 0 bytes to metrics.
It falls into the very first basket and looks like this in result:

# TYPE pbs_sub_request_size_bytes histogram pbs_sub_request_size_bytes_bucket{request_size="amp",le="1000"} 1 pbs_sub_request_size_bytes_bucket{request_size="amp",le="5000"} 1 pbs_sub_request_size_bytes_bucket{request_size="amp",le="10000"} 1 pbs_sub_request_size_bytes_bucket{request_size="amp",le="15000"} 1 pbs_sub_request_size_bytes_bucket{request_size="amp",le="20000"} 1 pbs_sub_request_size_bytes_bucket{request_size="amp",le="40000"} 1 pbs_sub_request_size_bytes_bucket{request_size="amp",le="75000"} 1 pbs_sub_request_size_bytes_bucket{request_size="amp",le="100000"} 1 pbs_sub_request_size_bytes_bucket{request_size="amp",le="+Inf"} 1

It can be misinterpreted that request had less than 1000 bytes, where in fact that was an error.
Should we add the check for the labels.RequestStatus? Only write metric if labels.RequestStatus == metrics.RequestStatusOK?

I tested this PR with AMP endpoint and found that if stored request was not found and request Len was never measured, we still write 0 bytes to metrics.

Really? Let me go in there with the debugger. Do you mind sharing the amp request you used for this test?

Doesn't the AMP endpoint need to be updated to set this label in loadRequestJSONForAmp? I don't think we are recording the size on this endpoint at the moment.

Doesn't the AMP endpoint need to be updated

Yes. Corrected

SyntaxNode · 2025-09-04T06:28:47Z

metrics/prometheus/prometheus.go

@@ -168,6 +170,7 @@ func NewMetrics(cfg config.PrometheusMetrics, disabledMetrics config.DisabledMet
 	priceBuckets := []float64{250, 500, 750, 1000, 1500, 2000, 2500, 3000, 3500, 4000}
 	queuedRequestTimeBuckets := []float64{0, 1, 5, 30, 60, 120, 180, 240, 300}
 	overheadTimeBuckets := []float64{0.05, 0.06, 0.07, 0.08, 0.09, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1}
+	requestSizeBuckets := []float64{600, 700, 800, 900, 1000, 1500, 3000, 5000, 6000, 7500, 10000, 15000}


This bucket range allows for a max request size of ~15kb. I think that's too small. I've seen very large requests in the wild, especially when privacy signals are involved. Everyone's mileage will vary, but how about we start with 100kb as a max size?

Something like...
requestSizeBuckets := []float64{100, 500, 1000, 2000, 5000, 10000, 20000, 50000, 75000, 100000}

Discussed offline.
requestSizeBuckets := []float64{100, 500, 750, 1000, 2000, 4000, 7000, 10000, 15000, 20000, 50000, 75000}

Corrected. New buckets:

pbs_sub_request_queue_time_sum{request_status="requestRejectedLabel",request_type="video"} 0 pbs_sub_request_queue_time_count{request_status="requestRejectedLabel",request_type="video"} 0 + # HELP pbs_sub_request_size_bytes Count that keeps track of incoming request size in bytes labeled by endpoint. + # TYPE pbs_sub_request_size_bytes histogram + pbs_sub_request_size_bytes_bucket{request_size="auction",le="100"} 0 + pbs_sub_request_size_bytes_bucket{request_size="auction",le="500"} 1 + pbs_sub_request_size_bytes_bucket{request_size="auction",le="750"} 1 + pbs_sub_request_size_bytes_bucket{request_size="auction",le="1000"} 1 + pbs_sub_request_size_bytes_bucket{request_size="auction",le="2000"} 1 + pbs_sub_request_size_bytes_bucket{request_size="auction",le="4000"} 1 + pbs_sub_request_size_bytes_bucket{request_size="auction",le="7000"} 1 + pbs_sub_request_size_bytes_bucket{request_size="auction",le="10000"} 1 + pbs_sub_request_size_bytes_bucket{request_size="auction",le="15000"} 1 + pbs_sub_request_size_bytes_bucket{request_size="auction",le="20000"} 1 + pbs_sub_request_size_bytes_bucket{request_size="auction",le="50000"} 1 + pbs_sub_request_size_bytes_bucket{request_size="auction",le="75000"} 1 + pbs_sub_request_size_bytes_bucket{request_size="auction",le="+Inf"} 1 + pbs_sub_request_size_bytes_sum{request_size="auction"} 217 + pbs_sub_request_size_bytes_count{request_size="auction"} 1 + pbs_sub_request_size_bytes_bucket{request_size="video",le="100"} 0 + pbs_sub_request_size_bytes_bucket{request_size="video",le="500"} 0 + pbs_sub_request_size_bytes_bucket{request_size="video",le="750"} 0 + pbs_sub_request_size_bytes_bucket{request_size="video",le="1000"} 0 + pbs_sub_request_size_bytes_bucket{request_size="video",le="2000"} 1 + pbs_sub_request_size_bytes_bucket{request_size="video",le="4000"} 1 + pbs_sub_request_size_bytes_bucket{request_size="video",le="7000"} 1 + pbs_sub_request_size_bytes_bucket{request_size="video",le="10000"} 1 + pbs_sub_request_size_bytes_bucket{request_size="video",le="15000"} 1 + pbs_sub_request_size_bytes_bucket{request_size="video",le="20000"} 1 + pbs_sub_request_size_bytes_bucket{request_size="video",le="50000"} 1 + pbs_sub_request_size_bytes_bucket{request_size="video",le="75000"} 1 + pbs_sub_request_size_bytes_bucket{request_size="video",le="+Inf"} 1 + pbs_sub_request_size_bytes_sum{request_size="video"} 1790 + pbs_sub_request_size_bytes_count{request_size="video"} 1 # HELP pbs_sub_request_time_seconds Seconds to resolve successful Prebid Server requests labeled by type. # TYPE pbs_sub_request_time_seconds histogram pbs_sub_request_time_seconds_bucket{request_type="amp",le="0.05"} 0

bsardo · 2025-09-04T14:51:52Z

endpoints/openrtb2/auction.go

@@ -442,6 +442,7 @@ func (deps *endpointDeps) parseRequest(httpRequest *http.Request, labels *metric
 		errs = []error{err}
 		return
 	}
+	labels.RequestSize = len(requestJson)


It looks like we are capturing the size of the request that comes in over the wire prior to merging in stored requests for both the auction and video endpoints while the amp endpoint captures the request size after stored requests are merged. IMO this is a bit confusing. Do we even need the amp request size since it is just based on stored requests?

Modified to not log these metrics oh the amp endpoint

linux019 · 2025-09-05T06:21:07Z

metrics/prometheus/prometheus.go

@@ -169,6 +171,7 @@ func NewMetrics(cfg config.PrometheusMetrics, disabledMetrics config.DisabledMet
 	priceBuckets := []float64{250, 500, 750, 1000, 1500, 2000, 2500, 3000, 3500, 4000}
 	queuedRequestTimeBuckets := []float64{0, 1, 5, 30, 60, 120, 180, 240, 300}
 	overheadTimeBuckets := []float64{0.05, 0.06, 0.07, 0.08, 0.09, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1}
+	requestSizeBuckets := []float64{100, 500, 750, 1000, 2000, 4000, 7000, 10000, 15000, 20000, 50000, 75000}


This looks to pretty hardcoded sizes
I would use max_request_size / 10 buckets
https://pkg.go.dev/github.com/prometheus/client_golang/prometheus#LinearBuckets

guscarreon added 8 commits August 19, 2025 13:40

step out quick

3fbc9f2

Merge branch 'master' into reqSizeMetric

9be1ec7

Moved GetEdnpoint() func

1444d13

rolled back metric name change

2ce92ba

Registry

3859047

VerifyMetric fix

b19d133

GetEndpointFromRequestType(requestType RequestType) test

eb2fea3

change 'endpoint' for 'size'

2fad1f9

guscarreon changed the title ~~Metric to account for request size~~ Metric for incoming request sizes Aug 20, 2025

bsardo assigned bsardo, hhhjort and VeronikaSolovei9 Aug 21, 2025

Merge branch 'master' into reqSizeMetric

5ed5714

bsardo reviewed Aug 21, 2025

View reviewed changes

VeronikaSolovei9 reviewed Aug 22, 2025

View reviewed changes

guscarreon added 5 commits August 22, 2025 16:20

Video and Amp

8e58660

remove aux files

1346ecf

New buckets

f1cd323

zero check

a85f7ac

Bucket size correction

4ffc915

VeronikaSolovei9 previously approved these changes Aug 25, 2025

View reviewed changes

SyntaxNode reviewed Sep 4, 2025

View reviewed changes

bsardo reviewed Sep 4, 2025

View reviewed changes

modify request size buckets

536f955

guscarreon dismissed VeronikaSolovei9’s stale review via 536f955 September 4, 2025 18:16

guscarreon added 2 commits September 4, 2025 14:23

Merge branch 'master' into reqSizeMetric

49f8105

removed metric for AMP endpoint

c87bd88

bsardo approved these changes Sep 4, 2025

View reviewed changes

bsardo changed the title ~~Metric for incoming request sizes~~ Metrics: Record incoming request sizes Sep 4, 2025

VeronikaSolovei9 approved these changes Sep 4, 2025

View reviewed changes

bsardo merged commit f63f77a into master Sep 4, 2025
4 checks passed

cross-repo-issue-creation bot mentioned this pull request Sep 4, 2025

Port PR from PBS-Go: Metrics: Record incoming request sizes prebid/prebid-server-java#4177

Closed

linux019 reviewed Sep 5, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Metrics: Record incoming request sizes #4495

Metrics: Record incoming request sizes #4495

guscarreon commented Aug 20, 2025

Uh oh!

bsardo left a comment

Uh oh!

bsardo Aug 21, 2025

Uh oh!

VeronikaSolovei9 Aug 22, 2025

Uh oh!

guscarreon Aug 22, 2025

Uh oh!

VeronikaSolovei9 Aug 22, 2025

Uh oh!

guscarreon Aug 22, 2025

Uh oh!

bsardo Aug 22, 2025

Uh oh!

guscarreon Aug 23, 2025

Uh oh!

SyntaxNode Sep 4, 2025

Uh oh!

bsardo Sep 4, 2025 •

edited

Loading

Uh oh!

guscarreon Sep 4, 2025

Uh oh!

bsardo Sep 4, 2025

Uh oh!

guscarreon Sep 4, 2025

Uh oh!

Uh oh!

linux019 Sep 5, 2025

Uh oh!

Uh oh!

Metrics: Record incoming request sizes #4495

Metrics: Record incoming request sizes #4495

Conversation

guscarreon commented Aug 20, 2025

Prometheus

Uh oh!

bsardo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bsardo Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bsardo Sep 4, 2025 •

edited

Loading