Support API for "pre-image" for pruning predicate evaluation #19722

sdf-jkl · 2026-01-09T21:34:58Z

Which issue does this PR close?

closes Support "pre-image" for pruning predicate evaluation #18320

Rationale for this change

Splitting the PR to make it more readable.

What changes are included in this PR?

Adding the udf_preimage logic without date_part implementation.

Are these changes tested?

Added unit tests for a test specific function

Are there any user-facing changes?

No

alamb

Thank you @sdf-jkl -- reviewed this PR carefully this morning and it looks great (thank you for splitting up the work), I found it well commented and well designed and a joy to read

I do think we need to add unit tests tests to for this feature, which I know you have lined up in #18789 but I think writing the unit tests in for the rewrite will make it easiest to validate.

I also have some questions about the rewrite for = (aka the boundary conditions)

datafusion/expr/src/udf.rs

alamb · 2026-01-18T12:31:33Z

datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs

+            // NOTE: we only consider immutable UDFs with literal RHS values
+            Expr::BinaryExpr(BinaryExpr { left, op, right }) => {
+                use datafusion_expr::Operator::*;
+                let is_preimage_op = matches!(


it might be nice (as a follow on PR) to mention this list in the docs for preimage -- e.g. that it only applies to predicates =, !=, ...

datafusion/optimizer/src/simplify_expressions/udf_preimage.rs

Add tests for additional cases

sdf-jkl · 2026-01-20T15:53:57Z

Wow, this is much cleaner, thanks!

alamb · 2026-01-20T17:21:36Z

I think this PR needs two more things:

Fi the NULL handling (probably by not calling preimage with null constants)
Update the API to have only a single method

(I am trying to keep my review context under control, so trying to focus on getting stuff through before starting more)

sdf-jkl · 2026-01-20T19:02:19Z

Both done. Re-requested a review.

alamb

Thank you very much @sdf-jkl

This looks good to me. I would like to change the signature to use Interval rather than Box<Interval> and there are a few other small comments, but we can also do this as a follow on PR (or I can push some commits to this PR)

Thank you for hanging with this one

FYI @colinmarc -- once we get this in, I think @sdf-jkl plans to implement preimage for date_part. Perhaps you are interested in something similar for date_trunc

Also, FYI @jonahgao and @xudong963 / @zhuqi-lucas in case you are interested in this PR (the primary usecase is improving the handling of date/timestamp predicates)

datafusion/expr/src/preimage.rs

datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs

alamb · 2026-01-20T21:21:53Z

datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs

+    let Expr::ScalarFunction(ScalarFunction { func, args }) = left_expr else {
+        return Ok((None, None));
+    };
+    if !is_literal_or_literal_cast(right_expr) {


I think this is still an open question, but it is ok to handle as a follow on PR (aka widen the expressions)

alamb · 2026-01-20T21:24:24Z

datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs

+    if !is_literal_or_literal_cast(right_expr) {
+        return Ok(PreimageResult::None);
+    }
+    if func.signature().volatility != Volatility::Immutable {


Also for a follow on PR, I think it would be safe to rewrite stable functions (whose values don't change during the statement)

datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs

datafusion/optimizer/src/simplify_expressions/udf_preimage.rs

alamb · 2026-01-20T21:27:17Z

datafusion/optimizer/src/simplify_expressions/udf_preimage.rs

+        Operator::LtEq => expr.lt(upper),
+        // <expr> = x ==> (<expr> >= lower) and (<expr> < upper)
+        //
+        // <expr> is not distinct from x ==> (<expr> is NULL and x is NULL) or ((<expr> >= lower) and (<expr> < upper))


Suggested change

// <expr> is not distinct from x ==> (<expr> is NULL and x is NULL) or ((<expr> >= lower) and (<expr> < upper))

// <expr> is not distinct from x ==> (<expr> is NULL) or ((<expr> >= lower) and (<expr> < upper))

I am not sure this IS NOT DISTICNT rewrite is correctas it is rewritten to just the range predicate. If expr is NULL and the literal is non-NULL, the original expression is FALSE, but the rewrite evaluates to NULL (x >= lower AND x < upper), which is not equivalent and violates the “same nullability” expectation for simplified expressions.

@alamb In a WHERE clause, both FALSE and NULL might behave similarly (both filter out the row), so here may be safety?

If we want to keep false:

Operator::IsNotDistinctFrom => { // expr IS NOT DISTINCT FROM x => must return FALSE if expr is NULL // because we know x is NOT NULL. expr.clone().is_not_null().and( and(expr.clone().gt_eq(lower), expr.lt(upper)) ) }

@xudong963 this solves the issue. Thanks!

xudong963 · 2026-01-21T02:55:07Z

I'll have a look at the PR today

xudong963 · 2026-01-21T13:36:28Z

datafusion/optimizer/src/simplify_expressions/udf_preimage.rs

+/// [preimage]: https://en.wikipedia.org/wiki/Image_(mathematics)#Inverse_image
+///
+pub(super) fn rewrite_with_preimage(
+    _info: &SimplifyContext,


Do we need this arg?

@alamb mentioned that we should keep it in #18789 (comment), but it was a while ago.

I think it is important to pass to ScalarUDFImpl::preimage but probably it can be removed from this method call

Since I want to merge this PR up from main anyways before merge, I'll clean it up too

xudong963 · 2026-01-21T13:49:00Z

datafusion/optimizer/src/simplify_expressions/udf_preimage.rs

+        Operator::LtEq => expr.lt(upper),
+        // <expr> = x ==> (<expr> >= lower) and (<expr> < upper)
+        //
+        // <expr> is not distinct from x ==> (<expr> is NULL and x is NULL) or ((<expr> >= lower) and (<expr> < upper))


@alamb In a WHERE clause, both FALSE and NULL might behave similarly (both filter out the row), so here may be safety?

If we want to keep false:

Operator::IsNotDistinctFrom => { // expr IS NOT DISTINCT FROM x => must return FALSE if expr is NULL // because we know x is NOT NULL. expr.clone().is_not_null().and( and(expr.clone().gt_eq(lower), expr.lt(upper)) ) }

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

xudong963

Thank you for the work!

…tafusion into smaller-preimage-pr-1

alamb · 2026-01-22T21:49:44Z

Thank you @sdf-jkl and @xudong963 -- I took a final look through this and it looks really nice. Thank you.

I merged up from main and I'll plan to merge when the tests pass. Let's get this one in to keep things moving forward

alamb · 2026-01-22T23:01:58Z

FYI @rcurtin

alamb · 2026-01-22T23:02:06Z

Thanks again @sdf-jkl and @xudong963

rcurtin · 2026-01-22T23:10:52Z

Thank you for keeping me in the loop! 👍 Glad to see this.

Add udf_preimage logic

d94889a

github-actions bot added logical-expr Logical plan and expressions optimizer Optimizer rules labels Jan 9, 2026

sdf-jkl and others added 7 commits January 9, 2026 20:37

Cargo fmt

4aa7f4e

Fix err in rewrite_with_preimage

2329c12

Rewrite the preimage_in_comparison

7ac8325

cargo fmt

7a3e8b3

Fix ci

fbd5dcc

Fix GtEq, Lt logic

d920735

Merge branch 'main' into smaller-preimage-pr-1

5ffb704

sdf-jkl mentioned this pull request Jan 10, 2026

Support "pre-image" for pruning predicate evaluation #18789

Closed

8 tasks

sdf-jkl mentioned this pull request Jan 18, 2026

Optimize the evaluation of DATE_TRUNC(<col>) == <constant>) when pushed down #18319

Open

alamb reviewed Jan 18, 2026

View reviewed changes

sdf-jkl force-pushed the smaller-preimage-pr-1 branch from f308662 to 5ffb704 Compare January 18, 2026 18:21

github-actions bot removed the documentation Improvements or additions to documentation label Jan 18, 2026

Merge pull request #1 from alamb/alamb/more_tests

47a18dc

Add tests for additional cases

sdf-jkl added 2 commits January 20, 2026 11:40

Add rhs Null guard

01b254b

Fix comment

d8b4f0f

sdf-jkl added 4 commits January 20, 2026 12:50

Update API

116d6e2

clippy

c0ed63c

Fix null handling unit test

5856150

Fix null handling test

c53a9fc

sdf-jkl requested a review from alamb January 20, 2026 19:01

alamb approved these changes Jan 20, 2026

View reviewed changes

xudong963 reviewed Jan 21, 2026

View reviewed changes

sdf-jkl and others added 4 commits January 21, 2026 09:27

Update datafusion/expr/src/preimage.rs

9b32843

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

Fix docs

53f72ed

Fix comment

ba5be8a

Fix is_not_distinct_from rewrite

46a941f

xudong963 approved these changes Jan 22, 2026

View reviewed changes

alamb added 3 commits January 22, 2026 16:39

Merge remote-tracking branch 'apache/main' into smaller-preimage-pr-1

798d88f

Simplify the API

fb155f6

Merge branch 'smaller-preimage-pr-1' of https://github.com/sdf-jkl/da…

a070246

…tafusion into smaller-preimage-pr-1

alamb mentioned this pull request Jan 22, 2026

Support "pre-image" for pruning predicate evaluation #18320

Closed

alamb added this pull request to the merge queue Jan 22, 2026

Merged via the queue into apache:main with commit c2f3d65 Jan 22, 2026
32 checks passed

	// <expr> is not distinct from x ==> (<expr> is NULL and x is NULL) or ((<expr> >= lower) and (<expr> < upper))
	// <expr> is not distinct from x ==> (<expr> is NULL) or ((<expr> >= lower) and (<expr> < upper))

Support API for "pre-image" for pruning predicate evaluation #19722

Support API for "pre-image" for pruning predicate evaluation #19722

Conversation

sdf-jkl commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sdf-jkl commented Jan 20, 2026

Uh oh!

alamb commented Jan 20, 2026

Uh oh!

sdf-jkl commented Jan 20, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xudong963 commented Jan 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xudong963 left a comment

Choose a reason for hiding this comment

Uh oh!

alamb commented Jan 22, 2026

Uh oh!

alamb commented Jan 22, 2026

Uh oh!

alamb commented Jan 22, 2026

Uh oh!

Uh oh!

rcurtin commented Jan 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sdf-jkl commented Jan 9, 2026 •

edited

Loading