Fetching external metrics from Prometheus

[faderskd]

No description provided.

[pitagoras3]

question: It seems that we're only interested in vector metrics, right?

[pitagoras3]

follow-up suggestion: If we are interested only in vector metrics, then maybe we can change this record name to VectorResult? Accordingly, values (L26) could be renamed to vector.

[faderskd]

yes, we are only interested in vectors, good idea to change its name.

[pitagoras3]

potential issue: Default cache size is 100 000. Isn't it too much (for Prometheus metrics)?

Prometheus cached metrics take more memory compared to Graphite cached metrics.

Graphite cached metric:

public class MetricDecimalValue {
    private final boolean available;
    private final String value;
    ...
}

Prometheus cached metric:

public class MonitoringMetricsContainer {
    private final Map<String, MetricDecimalValue> metrics; // <- A Map - it can store multiple elements per one cache entry
    private final boolean isAvailable;
    ...
}

What do you think about it?

[faderskd]

The difference should not be huge, because:

1. CachingGraphiteClient fetches multiple metrics by readMetrics(String... multipleMetricPaths) and dispatches them into multiple enitres:

cache[singleMetricPath1] = MetricDecimalValue(...)
cache[singleMetricPath2] = MetricDecimalValue(...)

2. CachingPromertheusClient fetches multiple metrics with one query readMetrics(String query) and put them into single element

cache[query] = Map<String, MetricsDecimalValue(...)>

The cache hit will be the same in both cases despite the fact that CachingGraphiteClient stores metrics more granularly: because we always query in the context of single subscription/topic so no metrics paths are shared between different queries.

[faderskd]

right now we don't have any cache metrics. but we should probably add a few in the future.

[pitagoras3]

question: One more thing - what about the documentation? :) Would you like to approach it in separate PR, or you can add docs about the Prometheus metrics in this PR?

[moscicky]

suggestion: we should probably name this one VictoriaMetricsMetricsProvider

[moscicky]

It would be nice to be able to disable the whole monitoring but we can do that in the future