CI: add `clippy::needless_pass_by_value` rule #18468

2010YOUY01 · 2025-11-03T10:12:54Z

Which issue does this PR close?

An initial attempt towards #18467

Rationale for this change

Rationale for the additional lint rule `clippy::needless_pass_by_value`

There is a clippy lint rule that is not turned on by the current strictness level in CI: https://rust-lang.github.io/rust-clippy/master/index.html#needless_pass_by_value
Note it has the Clippy category pedantic, and its description is lints which are rather strict or have occasional false positives from https://doc.rust-lang.org/nightly/clippy

It seems we have been suffering from the excessive copying issue for quite some time, and @alamb is on the front line now #18413. I think this extra lint rule is able to help.

Implementation plan

This PR only enables this rule in datafusion-common package, and apply #[allow(clippy::needless_pass_by_value)] for all violations.
If this PR makes sense, we can open a tracking issue and roll out this check to the remaining workspace packages. At least this can help prevent new inefficient patterns and identify existing issues that we can fix gradually.

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

alamb

Thank you @2010YOUY01 -- I think this looks like an improvement to me

If we are going to add this lint I think we should also update the various APIs to pass (not just add #[allow(clippy::needless_pass_by_value)]

If this PR makes sense, we can open a tracking issue and roll out this check to the remaining workspace packages. At least this can help prevent new inefficient patterns and identify existing issues that we can fix gradually.

I think it makes a lot of sense

alamb · 2025-11-03T22:02:25Z

datafusion/common/src/column.rs

    use std::sync::Arc;

+    #[allow(clippy::needless_pass_by_value)]
    fn create_qualified_schema(qualifier: &str, names: Vec<&str>) -> Result<DFSchema> {


we could change this to impl IntoIterator<Item = &str> for example

Suggested change

fn create_qualified_schema(qualifier: &str, names: Vec<&str>) -> Result<DFSchema> {

fn create_qualified_schema(qualifier: &str, names: impl IntoIterator<Item = &str>) -> Result<DFSchema> {

alamb · 2025-11-03T22:03:08Z

datafusion/common/src/hash_utils.rs

 /// If `rehash==true` this combines the previous hash value in the buffer
 /// with the new hash using `combine_hashes`
 #[cfg(not(feature = "force_hash_collisions"))]
+#[allow(clippy::needless_pass_by_value)]


Why is this needed? What is clippy's alternate suggestion?

(I am wondering if this has found a good potential for optimization...)

The arg array can safely be taken by reference.

However, I believe all current call sites already pass it by moving, so they’re fine.

That said, there might be optimization opportunities elsewhere — this needless_pass_by_value warning could indicate that some callers are cloning unnecessarily just to make the rust compiler happy.

alamb · 2025-11-03T22:07:04Z

ci/scripts/rust_clippy.sh

 cargo clippy --all-targets --workspace --features avro,pyarrow,integration-tests,extended_tests -- -D warnings
+
+# Update packages incrementally for stricter Clippy checks
+# TODO: add tracking issue for the remaining workspace packages like `datafusion-catalog`


I think this would be better to add at the module level (rather than in the CI script) so that it would also be flagged locally when people ran clippy.

Similar to this:

datafusion/datafusion/common/src/lib.rs

Line 25 in e4f2b49

#![deny(clippy::clone_on_ref_ptr)]

I reverted the CI script, and update the datafusion/common/cargo.toml instead in 2736b39, since we can update the whole datafusion-common crate at once.

When updating a larger crate, I think it's possible to configure like this to update module by module.

2010YOUY01 · 2025-11-04T10:07:08Z

Thank you @2010YOUY01 -- I think this looks like an improvement to me

If we are going to add this lint I think we should also update the various APIs to pass (not just add #[allow(clippy::needless_pass_by_value)]

If this PR makes sense, we can open a tracking issue and roll out this check to the remaining workspace packages. At least this can help prevent new inefficient patterns and identify existing issues that we can fix gradually.

I think it makes a lot of sense

Thank you for the feedbacks!

For the following cases, I kept the #[allow(clippy::needless_pass_by_value)] to suppress it

Tests. I think this rule do seem a bit annoying for tests, and I couldn't find a way to turn it off for all test modules...however we can apply the suppressing macro for individual test modules.
Public APIs
Intentional moves

For future works, I think we could start with crates that are performance critical or currently suffering from unnecessary clones.

xudong963 · 2025-11-05T10:03:45Z

performance critical

One place is the logical optimizer I think

alamb

Thanks @2010YOUY01 -- this looks like an improvement to me

I think changing allow to expect would probably be a good improvement

I would also suggest as follow on PRs:

Consolidate the clippy configuration into Cargo.toml (there are a bunch of configs in library functions too -- for example

datafusion/datafusion/common/src/lib.rs

Lines 23 to 25 in 7b5685b

    
           // Make sure fast / cheap clones on Arc are explicit: 
        
           // https://github.com/apache/datafusion/issues/11143 
        
           #![deny(clippy::clone_on_ref_ptr)]

Change existing allow to expect to clean up our clippy linting

alamb · 2025-11-05T14:08:17Z

datafusion/common/src/column.rs

    use arrow::datatypes::{DataType, SchemaBuilder};
    use std::sync::Arc;

+    #[allow(clippy::needless_pass_by_value)]


I personally prefer using

Suggested change

#[allow(clippy::needless_pass_by_value)]

#[expect(clippy::needless_pass_by_value)]

Which will error if the lint is no longer needed

allow will sit there silently even when the the lint is fixed

fixed in dac1004

alamb · 2025-11-05T14:08:56Z

datafusion/common/src/scalar/mod.rs

-            ScalarValue::List(arr) => fmt_list(arr.to_owned() as ArrayRef, f)?,
-            ScalarValue::LargeList(arr) => fmt_list(arr.to_owned() as ArrayRef, f)?,
-            ScalarValue::FixedSizeList(arr) => fmt_list(arr.to_owned() as ArrayRef, f)?,
+            ScalarValue::List(arr) => fmt_list(arr.as_ref(), f)?,


2010YOUY01 · 2025-11-06T01:22:21Z

Consolidate the clippy configuration into Cargo.toml (there are a bunch of configs in library functions too -- for example

datafusion/datafusion/common/src/lib.rs

Lines 23 to 25 in 7b5685b

// Make sure fast / cheap clones on Arc are explicit:

// https://github.com/apache/datafusion/issues/11143

#![deny(clippy::clone_on_ref_ptr)]

fixed in dac1004

2010YOUY01 · 2025-11-06T01:38:17Z

Thank you @alamb and @xudong963 for the review and suggestions.

I have opened #18503 to track the follow-up tasks for enforcing this lint rule globally in DataFusion. I’ve opened issues for only two smaller packages first, to see how things go.

2010YOUY01 · 2025-11-06T02:14:30Z

datafusion/common/src/lib.rs

-#![deny(clippy::clone_on_ref_ptr)]
+// This lint rule is enforced in `../Cargo.toml`, but it's okay to skip them in tests
+// See details in https://github.com/apache/datafusion/issues/18503
+#![cfg_attr(test, allow(clippy::needless_pass_by_value))]


To turn off this lint rule for all tests

add clippy::needless_pass_by_value to datafusion-common

665c453

github-actions bot added development-process Related to development process of DataFusion common Related to common crate labels Nov 3, 2025

alamb reviewed Nov 3, 2025

View reviewed changes

review: fix clippy error instead of suppressing

2736b39

github-actions bot added substrait Changes to the substrait crate proto Related to proto crate labels Nov 4, 2025

revert public interface changes

f2811e9

github-actions bot removed substrait Changes to the substrait crate proto Related to proto crate labels Nov 4, 2025

alamb mentioned this pull request Nov 4, 2025

Andrew Lamb Weekly-ish Open Source plan - 2025-11-03 #18486

Open

34 tasks

alamb approved these changes Nov 5, 2025

View reviewed changes

review: use expect + move existing rule from lib.rs -> clippytoml

dac1004

github-actions bot added the logical-expr Logical plan and expressions label Nov 6, 2025

2010YOUY01 mentioned this pull request Nov 6, 2025

[Tracking] Rollout of new lint clippy::needless_pass_by_value in all datafusion crates #18503

Open

2 tasks

disable for test modules using cfg_attr

c66bee8

2010YOUY01 commented Nov 6, 2025

View reviewed changes

jizezhang mentioned this pull request Nov 6, 2025

Enforce lint rule clippy::needless_pass_by_value to datafusion-optimizer #18505

Open

	fn create_qualified_schema(qualifier: &str, names: Vec<&str>) -> Result<DFSchema> {
	fn create_qualified_schema(qualifier: &str, names: impl IntoIterator<Item = &str>) -> Result<DFSchema> {

	// Make sure fast / cheap clones on Arc are explicit:
	// https://github.com/apache/datafusion/issues/11143
	#![deny(clippy::clone_on_ref_ptr)]

	#[allow(clippy::needless_pass_by_value)]
	#[expect(clippy::needless_pass_by_value)]

CI: add clippy::needless_pass_by_value rule #18468

Are you sure you want to change the base?

CI: add clippy::needless_pass_by_value rule #18468

Conversation

2010YOUY01 commented Nov 3, 2025

Which issue does this PR close?

Rationale for this change

Rationale for the additional lint rule clippy::needless_pass_by_value

Implementation plan

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

2010YOUY01 commented Nov 4, 2025

Uh oh!

xudong963 commented Nov 5, 2025

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

2010YOUY01 commented Nov 6, 2025

Uh oh!

2010YOUY01 commented Nov 6, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

CI: add `clippy::needless_pass_by_value` rule #18468

CI: add `clippy::needless_pass_by_value` rule #18468

Rationale for the additional lint rule `clippy::needless_pass_by_value`