Skip to content

Commit eae79a7

Browse files
QuentinBissonclayton-cornell
QuentinBisson
and
clayton-cornell
authored
enhancement: add support for extra query matchers in loki.rules.kubernetes (#3373)
* enhancement: add support for extra query matchers in loki.rules.kubernetes * fix linting * fix existing test * Update docs/sources/reference/components/loki/loki.rules.kubernetes.md Co-authored-by: Clayton Cornell <[email protected]> * Update docs/sources/reference/components/loki/loki.rules.kubernetes.md * Update docs/sources/reference/components/loki/loki.rules.kubernetes.md Co-authored-by: Clayton Cornell <[email protected]> * Update docs/sources/reference/components/loki/loki.rules.kubernetes.md Co-authored-by: Clayton Cornell <[email protected]> * Update docs/sources/reference/components/loki/loki.rules.kubernetes.md Co-authored-by: Clayton Cornell <[email protected]> * address comments --------- Co-authored-by: Clayton Cornell <[email protected]>
1 parent dc8f5b7 commit eae79a7

File tree

6 files changed

+342
-40
lines changed

6 files changed

+342
-40
lines changed

‎CHANGELOG.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,9 @@ Main (unreleased)
4747

4848
- Pretty print diagnostic errors when using `alloy run` (@kalleep)
4949

50+
- The `loki.rules.kubernetes` component now supports adding extra label matchers
51+
to all queries discovered via `PrometheusRule` CRDs. (@QuentinBisson)
52+
5053
- Add optional `id` field to `foreach` block to generate more meaningful component paths in metrics by using a specific field from collection items. (@harshrai654)
5154

5255
- Fix validation logic in `beyla.ebpf` component to ensure that either metrics or traces are enabled. (@marctc)

‎docs/sources/reference/components/loki/loki.rules.kubernetes.md‎

Lines changed: 50 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -81,23 +81,27 @@ You can use the following blocks with `loki.rules.kubernetes`:
8181

8282
| Block | Description | Required |
8383
| ------------------------------------------------------------------ | ---------------------------------------------------------- | -------- |
84-
| [`authorization`][authorization] | Configure generic authorization to the endpoint. | no |
85-
| [`basic_auth`][basic_auth] | Configure `basic_auth` for authenticating to the endpoint. | no |
86-
| [`rule_namespace_selector`][label_selector] | Label selector for `Namespace` resources. | no |
87-
| `rule_namespace_selector` > [`match_expression`][match_expression] | Label match expression for `Namespace` resources. | no |
88-
| [`rule_selector`][label_selector] | Label selector for `PrometheusRule` resources. | no |
89-
| `rule_selector` > [`match_expression`][match_expression] | Label match expression for `PrometheusRule` resources. | no |
90-
| [`oauth2`][oauth2] | Configure OAuth 2.0 for authenticating to the endpoint. | no |
91-
| `oauth2` > [`tls_config`][tls_config] | Configure TLS settings for connecting to the endpoint. | no |
92-
| [`tls_config`][tls_config] | Configure TLS settings for connecting to the endpoint. | no |
84+
| [`authorization`][authorization] | Configure generic authorization to the endpoint. | no |
85+
| [`basic_auth`][basic_auth] | Configure `basic_auth` for authenticating to the endpoint. | no |
86+
| [`extra_query_matchers`][extra_query_matchers] | Additional label matchers to add to each query. | no |
87+
| `extra_query_matchers` > [`matcher`][matcher] | A label matcher to add to each query. | no |
88+
| [`rule_namespace_selector`][label_selector] | Label selector for `Namespace` resources. | no |
89+
| `rule_namespace_selector` > [`match_expression`][match_expression] | Label match expression for `Namespace` resources. | no |
90+
| [`rule_selector`][label_selector] | Label selector for `PrometheusRule` resources. | no |
91+
| `rule_selector` > [`match_expression`][match_expression] | Label match expression for `PrometheusRule` resources. | no |
92+
| [`oauth2`][oauth2] | Configure OAuth 2.0 for authenticating to the endpoint. | no |
93+
| `oauth2` > [`tls_config`][tls_config] | Configure TLS settings for connecting to the endpoint. | no |
94+
| [`tls_config`][tls_config] | Configure TLS settings for connecting to the endpoint. | no |
9395

9496
The > symbol indicates deeper levels of nesting.
9597
For example, `oauth2` > `tls_config` refers to a `tls_config` block defined inside an `oauth2` block.
9698

9799
[authorization]: #authorization
98100
[basic_auth]: #basic_auth
101+
[extra_query_matchers]: #extra_query_matchers
99102
[label_selector]: #rule_selector-and-rule_namespace_selector
100103
[match_expression]: #match_expression
104+
[matcher]: #matcher
101105
[oauth2]: #oauth2
102106
[tls_config]: #tls_config
103107

@@ -109,6 +113,26 @@ For example, `oauth2` > `tls_config` refers to a `tls_config` block defined insi
109113

110114
{{< docs/shared lookup="reference/components/basic-auth-block.md" source="alloy" version="<ALLOY_VERSION>" >}}
111115

116+
### `extra_query_matchers`
117+
118+
The `extra_query_matchers` block has no attributes.
119+
It contains zero or more [matcher][] blocks.
120+
These blocks allow you to add extra label matchers to all queries that are discovered by the `loki.rules.kubernetes` component.
121+
The algorithm for adding the label matchers to queries is the same as the one used by the [`promtool promql label-matchers set` command](https://prometheus.io/docs/prometheus/latest/command-line/promtool/#promtool-promql).
122+
It's adapted to work with the LogQL parser.
123+
124+
### `matcher`
125+
126+
The `matcher` block describes a label matcher that's added to each query found in `PrometheusRule` CRDs.
127+
128+
The following arguments are supported:
129+
130+
| Name | Type | Description | Default | Required |
131+
| ------------ | -------- | -------------------------------------------------- | ------- | -------- |
132+
| `match_type` | `string` | The type of match. One of `=`, `!=`, `=~` or `!~`. | | yes |
133+
| `name` | `string` | Name of the label to match. | | yes |
134+
| `value` | `string` | Value of the label to match. | | yes |
135+
112136
### `rule_selector` and `rule_namespace_selector`
113137

114138
The `rule_selector` and `rule_namespace_selector` blocks describe a Kubernetes label selector for rule or namespace discovery.
@@ -228,6 +252,23 @@ Replace the following:
228252
* _`<GRAFANA_CLOUD_API_KEY>`_: Your Grafana Cloud API key.
229253
* _`<GRAFANA_CLOUD_API_KEY_PATH>`_: The path to the Grafana Cloud API key.
230254

255+
This example adds label matcher `{cluster=~"prod-.*"}` to all the queries discovered by `loki.rules.kubernetes`.
256+
257+
```alloy
258+
loki.rules.kubernetes "default" {
259+
address = "loki:3100"
260+
extra_query_matchers {
261+
matcher {
262+
name = "cluster"
263+
match_type = "=~"
264+
value = "prod-.*"
265+
}
266+
}
267+
}
268+
```
269+
270+
If a query in the form of `{app="my-app"}` is found in `PrometheusRule` CRDs, it will be modified to `{app="my-app", cluster=~"prod-.*"}` before sending it to Loki.
271+
231272
The following example is an RBAC configuration for Kubernetes. It authorizes {{< param "PRODUCT_NAME" >}} to query the Kubernetes REST API:
232273

233274
```yaml

‎internal/component/loki/rules/kubernetes/events.go‎

Lines changed: 96 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -6,12 +6,16 @@ import (
66
"regexp"
77
"time"
88

9-
"github.com/grafana/alloy/internal/component/common/kubernetes"
10-
"github.com/grafana/alloy/internal/runtime/logging/level"
9+
"github.com/grafana/loki/v3/pkg/logql/syntax"
1110
"github.com/hashicorp/go-multierror"
1211
promv1 "github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring/v1"
12+
"github.com/prometheus/prometheus/model/labels"
1313
"github.com/prometheus/prometheus/model/rulefmt"
14+
"github.com/prometheus/prometheus/promql/parser"
1415
"sigs.k8s.io/yaml" // Used for CRD compatibility instead of gopkg.in/yaml.v2
16+
17+
"github.com/grafana/alloy/internal/component/common/kubernetes"
18+
"github.com/grafana/alloy/internal/runtime/logging/level"
1519
)
1620

1721
const eventTypeSyncLoki kubernetes.EventType = "sync-loki"
@@ -102,16 +106,16 @@ func (c *Component) reconcileState(ctx context.Context) error {
102106
}
103107

104108
diffs := kubernetes.DiffRuleState(desiredState, c.currentState)
105-
var result error
109+
var errs error
106110
for ns, diff := range diffs {
107111
err = c.applyChanges(ctx, ns, diff)
108112
if err != nil {
109-
result = multierror.Append(result, err)
113+
errs = multierror.Append(errs, err)
110114
continue
111115
}
112116
}
113117

114-
return result
118+
return errs
115119
}
116120

117121
func (c *Component) loadStateFromK8s() (kubernetes.RuleGroupsByNamespace, error) {
@@ -127,33 +131,111 @@ func (c *Component) loadStateFromK8s() (kubernetes.RuleGroupsByNamespace, error)
127131
return nil, fmt.Errorf("failed to list rules: %w", err)
128132
}
129133

130-
for _, pr := range crdState {
131-
lokiNs := lokiNamespaceForRuleCRD(c.args.LokiNameSpacePrefix, pr)
132-
133-
groups, err := convertCRDRuleGroupToRuleGroup(pr.Spec)
134+
for _, rule := range crdState {
135+
lokiNs := lokiNamespaceForRuleCRD(c.args.LokiNameSpacePrefix, rule)
136+
groups, err := convertCRDRuleGroupToRuleGroup(rule.Spec)
134137
if err != nil {
135138
return nil, fmt.Errorf("failed to convert rule group: %w", err)
136139
}
137140

141+
if c.args.ExtraQueryMatchers != nil {
142+
for _, ruleGroup := range groups {
143+
for i := range ruleGroup.Rules {
144+
query := ruleGroup.Rules[i].Expr.Value
145+
newQuery, err := addMatchersToQuery(query, c.args.ExtraQueryMatchers.Matchers)
146+
if err != nil {
147+
level.Error(c.log).Log("msg", "failed to add labels to PrometheusRule query", "query", query, "err", err)
148+
}
149+
ruleGroup.Rules[i].Expr.Value = newQuery
150+
}
151+
}
152+
}
153+
138154
desiredState[lokiNs] = groups
139155
}
140156
}
141157

142158
return desiredState, nil
143159
}
144160

161+
func addMatchersToQuery(query string, matchers []Matcher) (string, error) {
162+
var err error
163+
for _, s := range matchers {
164+
query, err = labelsSetLogQL(query, s.MatchType, s.Name, s.Value)
165+
if err != nil {
166+
return "", err
167+
}
168+
}
169+
return query, nil
170+
}
171+
172+
// Inspired from the labelsSetPromQL function from the mimir.rules.kubernetes component
173+
// this function was modified to use the logql parser instead
174+
func labelsSetLogQL(query, labelMatchType, name, value string) (string, error) {
175+
expr, err := syntax.ParseExpr(query)
176+
if err != nil {
177+
return query, err
178+
}
179+
180+
var matchType labels.MatchType
181+
switch labelMatchType {
182+
case parser.ItemType(parser.EQL).String():
183+
matchType = labels.MatchEqual
184+
case parser.ItemType(parser.NEQ).String():
185+
matchType = labels.MatchNotEqual
186+
case parser.ItemType(parser.EQL_REGEX).String():
187+
matchType = labels.MatchRegexp
188+
case parser.ItemType(parser.NEQ_REGEX).String():
189+
matchType = labels.MatchNotRegexp
190+
default:
191+
return query, fmt.Errorf("invalid label match type: %s", labelMatchType)
192+
}
193+
expr.Walk(func(e syntax.Expr) {
194+
switch concrete := e.(type) {
195+
case *syntax.MatchersExpr:
196+
var found bool
197+
for _, l := range concrete.Mts {
198+
if l.Name == name {
199+
l.Type = matchType
200+
l.Value = value
201+
found = true
202+
}
203+
}
204+
if !found {
205+
concrete.Mts = append(concrete.Mts, &labels.Matcher{
206+
Type: matchType,
207+
Name: name,
208+
Value: value,
209+
})
210+
}
211+
}
212+
})
213+
214+
return expr.String(), nil
215+
}
216+
145217
func convertCRDRuleGroupToRuleGroup(crd promv1.PrometheusRuleSpec) ([]rulefmt.RuleGroup, error) {
146218
buf, err := yaml.Marshal(crd)
147219
if err != nil {
148220
return nil, err
149221
}
150222

223+
var errs error
151224
groups, _ := rulefmt.Parse(buf)
152-
153-
// Disable looking for errors, loki queries won't be valid prometheus queries, but still want the similar information
154-
//if len(errs) > 0 {
155-
// return nil, multierror.Append(nil, errs...)
156-
//}
225+
for _, group := range groups.Groups {
226+
for _, rule := range group.Rules {
227+
if _, err := syntax.ParseExpr(rule.Expr.Value); err != nil {
228+
if rule.Record.Value != "" {
229+
errs = multierror.Append(errs, fmt.Errorf("could not parse expression for record '%s' in group '%s': %w", rule.Record.Value, group.Name, err))
230+
} else {
231+
errs = multierror.Append(errs, fmt.Errorf("could not parse expression for alert '%s' in group '%s': %w", rule.Alert.Value, group.Name, err))
232+
}
233+
}
234+
}
235+
}
236+
if errs != nil {
237+
return nil, errs
238+
}
157239

158240
return groups.Groups, nil
159241
}

0 commit comments

Comments
 (0)