Implementation for regex_instr #15928

nirnayroy · 2025-05-02T20:02:33Z

Which issue does this PR close?

Closes Add regexp function - regexp_instr() #13009

Rationale for this change

Implements a regex SQL standard function in datafusion

What changes are included in this PR?

Implementation, tests, benches and docs for the regexp_instr function

Are these changes tested?

Yes

Are there any user-facing changes?

Yes

No

2010YOUY01 · 2025-05-04T03:01:13Z

Thank you. I'm wondering what's the reference system for this function's behavior (like postgres or others)

nirnayroy · 2025-05-04T06:56:41Z

Thank you. I'm wondering what's the reference system for this function's behavior (like postgres or others)

The reference system for this function's behaviour is postgres.

alamb · 2025-05-07T20:37:15Z

Thank you for this PR @nirnayroy

Can you please resolve the CI error: https://github.com/apache/datafusion/actions/runs/14820525339/job/41754009017?pr=15928

If you encounter an error, run './dev/update_function_docs.sh' and commit

nirnayroy · 2025-05-19T12:46:25Z

Thank you for this PR @nirnayroy

Can you please resolve the CI error: https://github.com/apache/datafusion/actions/runs/14820525339/job/41754009017?pr=15928

If you encounter an error, run './dev/update_function_docs.sh' and commit

I ran the bash script, but I’m not sure if the workflow succeeded.

nirnayroy · 2025-05-19T21:53:06Z

fixed the cippy errors showing up in the workflow

nirnayroy · 2025-05-20T05:40:21Z

fixed formatting error in workflow

alamb · 2025-05-21T18:46:37Z

@Omega359 I wonder if you might have time to review this PR?

Omega359 · 2025-05-21T19:35:08Z

Of course @alamb, not sure how I missed this one. It may be a day or two though

blaginin

Thank you for the PR!

I've left some comments - a few are just nits, so feel free to dismiss them or raise separate issues on top of this PR.

My main concern right now is that I’d expect this function to be a superset of strpos, but it currently behaves differently in some edge cases. For example:

SELECT regexp_instr('😀abcdef', 'abc');
SELECT strpos('😀abcdef', 'abc');

SELECT strpos(NULL, 'abc');
SELECT regexp_instr(NULL, 'abc');

SELECT strpos('abc', NULL);
SELECT regexp_instr('abc', NULL);

Do you think we should unify them? 🙏

datafusion/functions/src/regex/regexpinstr.rs

nirnayroy · 2025-06-12T07:10:36Z

Hi @blaginin, thanks for the review and regret the delay in reply. I think I have rectified a majority of the concerns raised. Please have a look again.

datafusion/functions/src/regex/regexpcount.rs

Copilot

Pull Request Overview

This PR adds the regexp_instr SQL function, including documentation, SQL logic tests, benchmarking, and integration into the function registry.

Introduce regexp_instr in docs and user guide
Add regexp_instr SQL tests and benches
Expose regex compilation helpers and register the new UDF

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
docs/source/user-guide/sql/scalar_functions.md	Add docs for `regexp_instr`, including signature and examples
datafusion/sqllogictest/test_files/regexp/regexp_instr.slt	Add SQL logic tests covering `regexp_instr` behavior
datafusion/functions/src/regex/regexpcount.rs	Make `compile_and_cache_regex` and `compile_regex` public
datafusion/functions/src/regex/mod.rs	Register `regexp_instr` UDF and define its expression builder
datafusion/functions/benches/regx.rs	Add benchmark cases for `regexp_instr`

Comments suppressed due to low confidence (4)

docs/source/user-guide/sql/scalar_functions.md:1844

The start argument description is duplicated (- **start**: - **start**:). Remove the extra label for clarity.

- **start**: - **start**: Optional start position (the first position is 1) to search for the regular expression. Can be a constant, column, or function. Defaults to 1

docs/source/user-guide/sql/scalar_functions.md:1837

The function signature is missing the subexpr (and return_option) parameters implemented in code. Update the signature to match the actual argument order.

regexp_instr(str, regexp[, start[, N[, flags]]])

datafusion/functions/src/regex/mod.rs:71

[nitpick] The parameter name endoption is unclear. Consider renaming it to return_option or another descriptive name consistent with SQL terminology.

        endoption: Option<Expr>,

datafusion/sqllogictest/test_files/regexp/regexp_instr.slt:1

There are no tests covering the subexpr argument. Add test cases that use the subexpr parameter to validate capture-group position behavior.

# Licensed to the Apache Software Foundation (ASF) under one

datafusion/functions/benches/regx.rs

datafusion/functions/src/regex/regexpinstr.rs

blaginin

Thanks for your work, @nirnayroy!

The PR is much better now, but I think there's still room for improvement:

Tests are failing. If that helps, you can run the CI command locally to debug:

The code around handling literals and arrays feels a bit complex. I think arrow/df does this in a simpler and more concise way – I’ve attached some links for inspiration :)

I haven’t reviewed the tests yet since I assume things might change as you iterate. But I can see you're covering a lot of cases, which is great ☘️

datafusion/functions/src/regex/regexpcount.rs

datafusion/functions/src/regex/regexpinstr.rs

add tests for subexp correct function signature for benches

nirnayroy · 2025-06-28T16:00:32Z

Hi @blaginin , thanks for the help and suggestions for improvement.
I have addressed the requested changes. Please have another look.

Tests are failing. If that helps, you can run the CI command locally to debug:

I have tried running it and the tests are passing on my local.

Omega359 · 2025-07-01T19:57:24Z

datafusion/functions/src/regex/mod.rs

+            args.push(subexpr);
+        };
+        super::regexp_instr().call(args)
+    }
    /// Returns true if a has at least one match in a string, false otherwise.


Suggested change

/// Returns true if a has at least one match in a string, false otherwise.

/// Returns true if a regex has at least one match in a string, false otherwise.

Omega359 · 2025-07-01T19:59:09Z

datafusion/functions/src/regex/mod.rs

+        Some(flags) => {
+            if flags.contains("g") {
+                return Err(ArrowError::ComputeError(
+                    "regexp_count() does not support global flag".to_string(),


Suggested change

"regexp_count() does not support global flag".to_string(),

"regexp_count()/regexp_instr() does not support the global flag".to_string(),

datafusion/functions/src/regex/regexpinstr.rs

Omega359 · 2025-07-01T20:07:23Z

datafusion/functions/src/regex/regexpinstr.rs

+    let (start_array, _is_start_scalar) = start_array.map_or((None, true), |start| {
+        let (start, is_start_scalar) = start.get();
+        (Some(start), is_start_scalar)
+    });


Suggested change

let (start_array, _is_start_scalar) = start_array.map_or((None, true), |start| {

let (start, is_start_scalar) = start.get();

(Some(start), is_start_scalar)

});

let start_array = start_array.map_or(None, |start| {

let (start, _is_start_scalar) = start.get();

Some(start)

});

???

addressed, removed the unused variables

Omega359 · 2025-07-01T20:36:21Z

I'll pull this branch later this week and run the tests but in general this PR is looking pretty good! I left a few comments/suggestions for a few things I found from a quick review.

blaginin

I think all my points are resolved, so approving. Thank you!
Will merge after the review round from @Omega359 ☺️

Omega359 · 2025-07-02T19:42:53Z

Run extended tests

alamb · 2025-07-02T19:45:56Z

Run extended tests

It works!

Omega359 · 2025-07-02T20:24:04Z

Clippy failures related to rand update (I think #16062)

Edit: looks like the usages of rand for the benchmark was updated in the above commit ... I'm thinking the additions in this PR do not reflect that change.

Omega359 · 2025-07-02T20:58:46Z

Run extended tests

Implementation for regex_instr

45b45c3

github-actions bot added documentation Improvements or additions to documentation sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels May 2, 2025

linting and typo addressed in bench

4fb0cca

prettier formatting

cff8d42

scalar_functions_formatting

2e70881

alamb and others added 2 commits May 19, 2025 13:49

Merge branch 'main' into complete-regexp-instr

0b12a21

linting format macros

c3131ed

formatting

eeeac8d

Merge branch 'main' into complete-regexp-instr

4bd7eb6

blaginin self-requested a review May 21, 2025 16:54

blaginin reviewed May 21, 2025

View reviewed changes

Nirnay Roy added 3 commits June 12, 2025 02:23

address comments to PR

89e2315

formatting

fad6e7b

clippy

93c35c8

fmt

990a653

blaginin self-requested a review June 17, 2025 20:25

blaginin reviewed Jun 17, 2025

View reviewed changes

datafusion/functions/src/regex/regexpcount.rs Outdated Show resolved Hide resolved

blaginin requested a review from Copilot June 17, 2025 20:35

Copilot AI reviewed Jun 17, 2025

View reviewed changes

datafusion/functions/benches/regx.rs Show resolved Hide resolved

blaginin reviewed Jun 17, 2025

View reviewed changes

datafusion/functions/src/regex/regexpinstr.rs Outdated Show resolved Hide resolved

address docs typo

1120043

blaginin requested changes Jun 18, 2025

View reviewed changes

Nirnay Roy and others added 4 commits June 28, 2025 02:24

remove unnecessary struct and comment

ef43f28

delete redundant lines

ec71d2b

add tests for subexp correct function signature for benches

refactor get_index

265db0e

Merge branch 'main' into complete-regexp-instr

d25d21d

Omega359 reviewed Jul 1, 2025

View reviewed changes

datafusion/functions/src/regex/regexpinstr.rs Show resolved Hide resolved

Omega359 reviewed Jul 1, 2025

View reviewed changes

blaginin approved these changes Jul 1, 2025

View reviewed changes

Nirnay Roy added 2 commits July 2, 2025 23:05

comments addressed

94df4c8

update doc

5d1044a

clippy upgrade

581af20

	/// Returns true if a has at least one match in a string, false otherwise.
	/// Returns true if a regex has at least one match in a string, false otherwise.

	"regexp_count() does not support global flag".to_string(),
	"regexp_count()/regexp_instr() does not support the global flag".to_string(),

Implementation for regex_instr #15928

Are you sure you want to change the base?

Implementation for regex_instr #15928

Conversation

nirnayroy commented May 2, 2025

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

2010YOUY01 commented May 4, 2025

Uh oh!

nirnayroy commented May 4, 2025

Uh oh!

alamb commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nirnayroy commented May 19, 2025

Uh oh!

nirnayroy commented May 19, 2025

Uh oh!

nirnayroy commented May 20, 2025

Uh oh!

alamb commented May 21, 2025

Uh oh!

Omega359 commented May 21, 2025

Uh oh!

blaginin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nirnayroy commented Jun 12, 2025

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

blaginin left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nirnayroy commented Jun 28, 2025

Uh oh!

Omega359 Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

Omega359 Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Omega359 Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

nirnayroy Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

Omega359 commented Jul 1, 2025

Uh oh!

blaginin left a comment

Choose a reason for hiding this comment

Uh oh!

alamb commented May 7, 2025 •

edited

Loading

blaginin left a comment •

edited

Loading

Omega359 commented Jul 2, 2025 •

edited

Loading