Memory and Http_push_duration_seconds increase suddenly

[yuanyueping]

Hi，I used pushgateway for metric reporting. During the use of pushgateway, the memory suddenly increased occasionally, at the same time, the pushgateway_http_push_duration_seconds of pushgateway was increased. I had to restart pushgateway to solve the problem.

my question

1. The number of samples the pushgateway exposed hasn't changed significantly. Why memory suddenly increased.
2. The memory of pushgateway has not reached the upper limit, and the cpu usage was stable. Why pushgateway_http_push_duration_seconds was increased

pushgateway version： v1.4.1
cpu limit： 10 cores
memery limit： 15G

Pushgateway is mainly used for flink. The number of flink jobs is 60
The frequency of pushing to the pushgateway is 60s

cpu useage ：

memery useage：

pushgateway_http_push_duration_seconds

pushgateway http qpm had not increased (put request)

pushgateway_http_push_size_bytes p90

pushgateway_http_push_size_bytes p50

pushgateway scrape_samples_scraped: the number of samples the pushgateway exposed hasn't changed significantly

[beorn7]

I can imagine that 20,000 pushed metrics mean that the PGW runs very close to its limits. In that case, even a relatively inconspicuous change (a slight increase in load, but also, perhaps, another process starting on the same machine, the network becoming a bit slower etc.) could push it over the brink, entering a death-spiral of no return.

Note that the PGW is designed for hundreds of metrics, perhaps a few thousand in extreme cases.

However, if you find a bug or a good optimization that doesn't make the life harder for the intended use cases of the PGW, I'll happily accept a PR.

[yuanyueping]

Hi, I found the reason for the delay of the push request. The reason is that the number of push tasks is large, the writing speed of the writeQueue is much faster than the consumption speed, resulting in the accumulation of the writeQueue, the error channel waiting time in the push request is longer, and the request response time is longer.

The response time in the picture is too long because the writeQueue length reaches 1000.

Thank you very much for your answer