-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore(dataobj): predicate pushdown metadata label filters #16846
base: main
Are you sure you want to change the base?
Conversation
Outer: | ||
for i, stage := range pipelineExpr.MultiStages { | ||
switch stage := stage.(type) { | ||
case *syntax.LabelFmtExpr, *syntax.LineParserExpr, *syntax.LogfmtParserExpr, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe pattern here too? That can add to the label set
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
loki/pkg/logql/syntax/syntax.y.go
Lines 1371 to 1390 in 21bf5ee
case 100: | |
syntaxDollar = syntaxS[syntaxpt-1 : syntaxpt+1] | |
{ | |
syntaxVAL.stage = newLabelParserExpr(OpParserTypeJSON, "") | |
} | |
case 101: | |
syntaxDollar = syntaxS[syntaxpt-2 : syntaxpt+1] | |
{ | |
syntaxVAL.stage = newLabelParserExpr(OpParserTypeRegexp, syntaxDollar[2].str) | |
} | |
case 102: | |
syntaxDollar = syntaxS[syntaxpt-1 : syntaxpt+1] | |
{ | |
syntaxVAL.stage = newLabelParserExpr(OpParserTypeUnpack, "") | |
} | |
case 103: | |
syntaxDollar = syntaxS[syntaxpt-2 : syntaxpt+1] | |
{ | |
syntaxVAL.stage = newLabelParserExpr(OpParserTypePattern, syntaxDollar[2].str) | |
} |
LineParserExpr should cover it
21bf5ee
to
7a4a514
Compare
What this PR does / why we need it:
Follow-up to #16802, adds support to build dataobj reader predicates from label filters that refer to structured metadata columns. This is done by looking up the metadata columns of each dataobj section and applying filters that refer to any of these columns, so the predicates and expressions for readers that operate on different sections on an object need not be the same.
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Not part of this pr, but some though on applying stream filters: I think they can only be applied by looking at the labels of the matching streams for an object.
For streams containing the label, we can look at the value to either discard streams that do not match or discard the label filter from expression to avoid redundant work if there is a match
Checklist
CONTRIBUTING.md
guide (required)feat
PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.docs/sources/setup/upgrade/_index.md
deprecated-config.yaml
anddeleted-config.yaml
files respectively in thetools/deprecated-config-checker
directory. Example PR