Add support to configure min GC intervals for soft and hard limits #12450
Conversation
bogdandrutu
force-pushed
the
add-support-for-gc
branch
from
e77346b to
d3f19cf
Compare
Codecov ReportAttention: Patch coverage is
❌ Your patch check has failed because the patch coverage (88.88%) is below the target coverage (95.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #12450 +/- ##
========================================
Coverage 92.19% 92.19%
========================================
Files 466 465 -1
Lines 25325 25187 -138
========================================
- Hits 23348 23221 -127
+ Misses 1577 1570 -7
+ Partials 400 396 -4 ☔ View full report in Codecov by Sentry. |
Signed-off-by: Bogdan Drutu <[email protected]>
bogdandrutu
force-pushed
the
add-support-for-gc
branch
from
d3f19cf to
1f8a2a5
Compare
| if time.Since(ml.lastGCDone) > ml.minGCIntervalWhenHardLimited { | ||
| ml.logger.Warn("Memory usage is above hard limit. Forcing a GC.", memstatToZapField(ms)) | ||
| ms = ml.doGCandReadMemStats() | ||
| // Check the limit again to see if GC helped. | ||
| aboveSoftLimit = ml.usageChecker.aboveSoftLimit(ms) | ||
| } |
There was a problem hiding this comment.
nit: would be nice to see this factored into a function or closure that is called twice inside section of code.
There was a problem hiding this comment.
2 out of 4 lines of code have different arguments (the if statement and the log message). I though it will not help that much. If you feel strong, I will do it.
Co-authored-by: Joshua MacDonald <[email protected]>
When servers are in a high memory usage mode, the amount of times GC is called creates a high CPU usage which combined with a high ingest rate limits capability to offload existing queued data.
Configuring the minimum interval even for hard limit, allows the system to spend some CPU cycles between GCs to offload old data from queues/batch processor. The most amount of data I've seen accumulated are blocked in the batch processor queue (incoming channel) not in the exporter queue.