Store Gateway: use different stats for lazy posting series size estimation

[yeya24]

Is your proposal related to a problem?

Lazy expanded postings is an optimization strategy which skips downloading certain postings in large size, if we find it is cheaper to fetch series and then filter out the label matchers. This is extremely useful to skip those expensive matchers where a lot of postings are matched such as env="prod", pod!="".

One of the key information used here is the estimated series size we use to calculate if it is worth downloading postings or series. Today, compactor gathers block max series size and sets it in meta.json file and Store Gateway uses this block max series size for estimation. The problem here is that max series size might be too large, causing some certain queries unable to be optimized by lazy postings so the optimization not really taking effect.

Take a production block we have for example, the block has around 20M series. Its max series size is about 36K but the P99 size is only 735 so 40X difference. That means using max series size probably doesn't work the best for all the scenarios and maybe using a different statistics can more accurately represent most of the series in the block.

thanos tools bucket analyze --id=01JCJDDN6G6ST8SKKNM4CA5W9S
series max size: 36175
series min size: 48
series avg size: 255
series p75 size: 288
series p90 size: 464
series p99 size: 735
series p999 size: 831
series p9999 size: 911
series p99999 size: 2064
series p999999 size: 3344
series p9999999 size: 10687
series p99999999 size: 10767

If we use p9999, then for a 20M block, only 20M * 0.0001 = 2000 series exceeds this size so the impact is very small.

Describe the solution you'd like

Similar to how we support estimated max series size today, compactor will try to gather more statistics about block series size (from P90 to P9999) and configure them in meta.json. It can look like below:

{
  "index_stats": {
    "series_max_size": 30000,
    "series_p90_size": 3000,
    "series_p99_size": 3000,
    "series_p999_size": 3000,
    "series_p9999_size": 3000
  }
}

Those percentiles are hardcoded to be P90, P99, P999, P9999. I am open to make them configurable or open to different formats in meta.json.

In store gateway, there is a flag estimated-series-size-stats to choose which statistic you want to use for lazy posting optimization. By default, it is MAX. Users can pick from P90 to P9999 and if the required statistic doesn't exist it can fallback to MAX.

This configured strategy is specific to lazy posting series size optimization. We also need to estimate series size when fetching series from object store and that will keep using max statistics.