[FEATURE] Delta Lake Tests Not Triggered in CI for Relevant PRs

[razajafri]

Summary:
Our CI should detect PRs that modify Delta Lake modules and automatically run the relevant Delta Lake tests before merge. This minor change would close a significant coverage gap, improving project stability and developer experience.

Problem Statement:
Currently, our CI does not automatically run Delta Lake tests before a pull request is merged, even when the PR includes changes to Delta Lake–related components (such as files in delta-xxx modules). This adds risk of bugs or regressions being merged, reduces confidence in PR validation, and creates additional overhead if failures are only discovered after the merge by the nightly jobs failing.

Suggested Solution:
We can significantly improve our test coverage and reliability by updating our CI configuration to:

Automatically detect PRs that affect any delta-xxx module

This can be done by checking the file paths of modified files in the PR for patterns like delta-*. We are already doing something similar when triggering Databricks pipelines when we detect a Databricks change in a PR

Trigger Delta Lake tests in CI for the relevant Spark/Delta version(s) if such changes are detected.

Benefits:
Prevents regressions: Always verifies Delta Lake compatibility before merge.

Increases confidence: Contributors can see test results in their PR.

Reduces follow-up work: Avoids post-merge fixes and revert cycles.

[pxLi]

Trigger Delta Lake tests in CI for the relevant Spark/Delta version(s) if such changes are detected.

@razajafri @liurenjie1024 Please help provide a change detector (such as a script or regex) that can identify changes related to delta_lake (not just in test files, but across all associated code), thanks

cc @yinqingh to help follow up on this once the detector/regex is provided, thanks

[razajafri]

We can detect the change based on the modules, e.g., delta-lake/delta-24x should trigger a delta lake tests for Spark 3.4.x. I remember there was a script that automatically detected databricks changes and triggered databricks as part of the premerge even if the dev didn't provide [databricks] as part of the title. I can't find it now but will post here if I do.

Can you provide a little bit more information on what you will be running the regex against?

[pxLi]

We can detect the change based on the modules, e.g., delta-lake/delta-24x should trigger a delta lake tests for Spark 3.4.x. I remember there was a script that automatically detected databricks changes and triggered databricks as part of the premerge even if the dev didn't provide [databricks] as part of the title. I can't find it now but will post here if I do.

Can you provide a little bit more information on what you will be running the regex against?

We have a func here for pre-merge jenkinsfile https://github.com/NVIDIA/spark-rapids/blob/branch-25.08/jenkins/Jenkinsfile-blossom.premerge#L439-L451, please provide the command for local run, we can help add some CI funcs later, thanks