[Q] Inferring rr load and limits from Prometheus metrics?

[victor-sudakov]

Is it correct to assume the following (where N is the number of simultaneous requests):

1. rr can handle N <= num_workers. If current N > num_workers, rr begins queueing and delaying requests.

2. If N * requests_duration > allocate_timeout, rr begins rejecting requests.

So we basically have a "soft limit" and a "hard limit" for N (depending on requests processing time of course. The limits will be higher for shorter requests).

If the above statements are correct, then how can I use the rr Prometheus metrics to warn me that 1) queuing has begun and 2) rejecting has begun? It would be great if you could help with a PromQL expression to catch the above situations.

I've tried combining the available metrics but cannot figure out. RoadRunner version 2.5.3

[rustatian]

Please, read: #799 (reply in thread)

[rustatian]

1. queuing has begun

You won't know about that at the time. We don't have metrics about that at the time.

2. rejecting has begun?

Only for the request timeout. You'll see the timeout error. But, no general stats.

[victor-sudakov]

1. queuing has begun

You won't know about that at the time. We don't have metrics about that at the time.

It is a pity. Are you sure we cannot somehow cleverly use one of the rr_http_request_duration_seconds_* metrics to indirectly calculate/predict queuing and rejections?

Do you mind if a create a feature request in roadrunner-plugins for these metrics?

[rustatian]

For sure you can predict that. I sent your a link with a discussion about how to do that. You just can't see the actual number of the queued requests.

[victor-sudakov]

For sure you can predict that. I sent your a link with a discussion about how to do that. You just can't see the actual number of the queued requests.

I remember that discussion, it was with me. It was about calculating rr settings for the expected (theoretical) load, which is of course very useful. But here I mean a different kind of prediction: based on current metrics of a live deployment, we should be able to notice when things will be going wrong (slow or out-of-service) soon, and take some measures (trigger an alarm, scale the deployment etc).

[rustatian]

Do you mind if a create a feature request in roadrunner-plugins for these metrics?

Yeah, sure, will be great.

[victor-sudakov]

Do you mind if a create a feature request in roadrunner-plugins for these metrics?

Yeah, sure, will be great.

Done in https://github.com/spiral/roadrunner-plugins/issues/116

[victor-sudakov]

With the development of new metrics in 2.6, I suggest closing this discussion and moving to #799 for load measurement issues. I cannot follow two discussions, tool old for that.

[rustatian]

Hey) The discussion can not be closed, only deleted or locked. It's not a problem to leave it open)

[victor-sudakov]

It's not a problem to leave it open)

I'm getting a split personality from two discussions on the same topic :-) Kill it!