Prometheus Counter Zero Initialisation Problem

[atibdialpad]

Hi Folks,
Do we have anything in the Python client lib to circumvent the classic Zero Initialization Problem in Prometheus Counters or is it expected from to implement the recommendation by manually initing all time-series with ZERO

https://prometheus.io/docs/practices/instrumentation/#avoid-missing-metrics
https://groups.google.com/g/prometheus-users/c/w69L7w6Z5y0
https://docs.google.com/document/d/1kakDVn8aP1JerimLeazuMfy2jaB14W2kZDHMRokPcv4/edit
prometheus/client_golang#1313 - Go Client
prometheus/client_ruby#141 - Ruby Client
https://www.section.io/blog/beware-prometheus-counters-that-do-not-begin-at-zero/
https://github.com/goto-opensource/smart-prometheus-client

[csmarchbanks]

Currently it is expected that you manually initialize counters, I believe this can be done just by calling my_counter.labels(...) to initialize to zero.

That said I would not be opposed to building something that makes this easier when the labelset is known ahead of time. An easy solution may also help people who don't know about this issue.

[atibdialpad]

Thanks @csmarchbanks we do have a proposal for this using the zero injection method https://github.com/prometheus/proposals/blob/main/proposals/2023-06-13_created-timestamp.md

Until then, I am planning to manually do the zero init as you recommended i.e. my_counter.labels(**tags).
I have a question in that regard. Is it enough to just zero init the counters or do I need to ensure that any subsequent increments happen after the zero count is scraped ?
For example: app -> prom agent -> prw to mimir,

if __new__timeseries:
  my_counter.labels(**tags) # zero init new timeseries
my_counter.labels(**tags).inc(value) # actual increment by value

Will this work out ? or do I need to delay the actual increment (by value) to ensure it happens after the zero value has been scraped and sent to mimir

I will experiment with this, putting it out here for any interesting insights from you

[csmarchbanks]

It is fine to just zero init the counter, Prometheus will assume the value reset to zero between scrapes if the value of a counter is lower than previously seen.

[atibdialpad]

I was discussing this with folks from Prometheus (the team that has come up with the zero injection solution) here https://cloud-native.slack.com/archives/C167KFM6C/p1698243223038169

The challenge here is that Prometheus needs to scrape the target when the value is zero i.e before the subsequent increment happens and I cannot guarantee that without either a 'delay' or the 'push zero' idea proposed in that thread.

I am still thinking about it.

[csmarchbanks]

If your problem is that increments can happen too quickly, or you need rate to work immediately then you will need to wait for the _created work to land in Prometheus. That is different than the series just completely missing though which is what my_counter.labels(...) will fix. In most scenarios waiting a scrape for rate to work on a brand new instance is ok, but that will depend on your system.

[atibdialpad]

If your problem is that increments can happen too quickly, or you need rate to work immediately then you will need to wait for the _created work to land in Prometheus

Yup, unfortunately thats the case.

I see the Go Client and Prometheus has merged the "created timestamp" fix via prometheus/client_golang#1313 and prometheus/prometheus#12936
by "you will need to wait for the _created work to land in Prometheus" you mean "wait for the Python Client" to implement the fix, right ?
I can tryout the Go Client in that case to get unblocked ?

[csmarchbanks]

The python client already supports _created series (unless you disable them). However, the Prometheus server does not do anything special with them today. You will need to wait for Prometheus to solve the issue, see prometheus/prometheus#6541.

That work is happening now.

[atibdialpad]

Ahh I see... I got confused... I thought the agreed approach is to add timestamps to the metadata. If the agreed solution is to add a new timeseries "<>_created" and use its timestamp and the "starting timestamp", we are good from the Python Client POV. Thanks !