Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(dataobj): predicate pushdown metadata label filters #16846

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

ashwanthgoli
Copy link
Contributor

@ashwanthgoli ashwanthgoli commented Mar 20, 2025

What this PR does / why we need it:

Follow-up to #16802, adds support to build dataobj reader predicates from label filters that refer to structured metadata columns. This is done by looking up the metadata columns of each dataobj section and applying filters that refer to any of these columns, so the predicates and expressions for readers that operate on different sections on an object need not be the same.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Not part of this pr, but some though on applying stream filters: I think they can only be applied by looking at the labels of the matching streams for an object.
For streams containing the label, we can look at the value to either discard streams that do not match or discard the label filter from expression to avoid redundant work if there is a match

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

Base automatically changed from predicate-pushdown to main March 20, 2025 13:57
Outer:
for i, stage := range pipelineExpr.MultiStages {
switch stage := stage.(type) {
case *syntax.LabelFmtExpr, *syntax.LineParserExpr, *syntax.LogfmtParserExpr,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe pattern here too? That can add to the label set

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

case 100:
syntaxDollar = syntaxS[syntaxpt-1 : syntaxpt+1]
{
syntaxVAL.stage = newLabelParserExpr(OpParserTypeJSON, "")
}
case 101:
syntaxDollar = syntaxS[syntaxpt-2 : syntaxpt+1]
{
syntaxVAL.stage = newLabelParserExpr(OpParserTypeRegexp, syntaxDollar[2].str)
}
case 102:
syntaxDollar = syntaxS[syntaxpt-1 : syntaxpt+1]
{
syntaxVAL.stage = newLabelParserExpr(OpParserTypeUnpack, "")
}
case 103:
syntaxDollar = syntaxS[syntaxpt-2 : syntaxpt+1]
{
syntaxVAL.stage = newLabelParserExpr(OpParserTypePattern, syntaxDollar[2].str)
}

LineParserExpr should cover it

@ashwanthgoli ashwanthgoli force-pushed the label-filter-predicate-pushdown branch from 21bf5ee to 7a4a514 Compare March 21, 2025 05:18
@ashwanthgoli ashwanthgoli marked this pull request as ready for review March 21, 2025 08:05
@ashwanthgoli ashwanthgoli requested a review from a team as a code owner March 21, 2025 08:05
@ashwanthgoli ashwanthgoli changed the title chore(dataobj): predicate pushdown label filters chore(dataobj): predicate pushdown metadata label filters Mar 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants