Skip to content

WIP: Upgrade DataFusion to arrow-rs/parquet 58.0.0 #19728

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
alamb wants to merge 17 commits into
base: main
Choose a base branch
from
Draft

WIP: Upgrade DataFusion to arrow-rs/parquet 58.0.0 #19728

Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
120d2b0
Update to pre-release arrow
alamb Dec 16, 2025
40a6a71
Fix clippy due to UnionFields::new is deprecated
alamb Dec 16, 2025
f50b9c3
Fix real bug in test
alamb Dec 16, 2025
87563d8
Update test for fixed issue upstream
alamb Dec 16, 2025
4afd46f
Update tests for new feature
alamb Dec 18, 2025
70e2841
Update tests for struct casting
alamb Dec 18, 2025
b29b662
Update to latest
alamb Jan 7, 2026
6a2e631
Merge remote-tracking branch 'apache/main' into alamb/update_arrow_57…
alamb Jan 7, 2026
ca11966
Merge branch 'main' into alamb/update_arrow_57.2.0
alamb Jan 8, 2026
63a85eb
Update to pre-release main
alamb Jan 10, 2026
277aac4
Update expected value of make_interval
alamb Jan 10, 2026
bb29b0c
Update expected output
alamb Jan 10, 2026
35b97fa
Update to latest arrow
alamb Jan 10, 2026
d2e8da6
Update rev
Dandandan Jan 16, 2026
1002408
update outputs
Dandandan Jan 16, 2026
c062756
Merge
Dandandan Jan 16, 2026
917a464
Update rev
Dandandan Jan 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
126 changes: 39 additions & 87 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

43 changes: 36 additions & 7 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,19 +91,19 @@ ahash = { version = "0.8", default-features = false, features = [
"runtime-rng",
] }
apache-avro = { version = "0.21", default-features = false }
arrow = { version = "57.1.0", features = [
arrow = { version = "57.2.0", features = [
"prettyprint",
"chrono-tz",
] }
arrow-buffer = { version = "57.1.0", default-features = false }
arrow-flight = { version = "57.1.0", features = [
arrow-buffer = { version = "57.2.0", default-features = false }
arrow-flight = { version = "57.2.0", features = [
"flight-sql-experimental",
] }
arrow-ipc = { version = "57.1.0", default-features = false, features = [
arrow-ipc = { version = "57.2.0", default-features = false, features = [
"lz4",
] }
arrow-ord = { version = "57.1.0", default-features = false }
arrow-schema = { version = "57.1.0", default-features = false }
arrow-ord = { version = "57.2.0", default-features = false }
arrow-schema = { version = "57.2.0", default-features = false }
async-trait = "0.1.89"
bigdecimal = "0.4.8"
bytes = "1.11"
Expand Down Expand Up @@ -166,7 +166,7 @@ log = "^0.4"
num-traits = { version = "0.2" }
object_store = { version = "0.12.4", default-features = false }
parking_lot = "0.12"
parquet = { version = "57.1.0", default-features = false, features = [
parquet = { version = "57.2.0", default-features = false, features = [
"arrow",
"async",
"object_store",
Expand Down Expand Up @@ -271,3 +271,32 @@ incremental = false
inherits = "release"
debug = true
strip = false

## Temporary arrow-rs patch until 58 is released

[patch.crates-io]
arrow = { git = "https://github.com/apache/arrow-rs.git", branch = "main" }
arrow-array = { git = "https://github.com/apache/arrow-rs.git", branch = "main" }
arrow-buffer = { git = "https://github.com/apache/arrow-rs.git", branch = "main" }
arrow-cast = { git = "https://github.com/apache/arrow-rs.git", branch = "main" }
arrow-data = { git = "https://github.com/apache/arrow-rs.git", branch = "main" }
arrow-ipc = { git = "https://github.com/apache/arrow-rs.git", branch = "main" }
arrow-schema = { git = "https://github.com/apache/arrow-rs.git", branch = "main" }
arrow-select = { git = "https://github.com/apache/arrow-rs.git", branch = "main" }
arrow-string = { git = "https://github.com/apache/arrow-rs.git", branch = "main" }
arrow-ord = { git = "https://github.com/apache/arrow-rs.git", branch = "main" }
arrow-flight = { git = "https://github.com/apache/arrow-rs.git", branch = "main" }
parquet = { git = "https://github.com/apache/arrow-rs.git", branch = "main" }

#arrow = { path= "/Users/andrewlamb/Software/arrow-rs2/arrow" }
#arrow-array = { path= "/Users/andrewlamb/Software/arrow-rs2/arrow-array" }
#arrow-buffer = { path= "/Users/andrewlamb/Software/arrow-rs2/arrow-buffer" }
#arrow-cast = { path= "/Users/andrewlamb/Software/arrow-rs2/arrow-cast" }
#arrow-data = { path= "/Users/andrewlamb/Software/arrow-rs2/arrow-data" }
#arrow-ipc = { path= "/Users/andrewlamb/Software/arrow-rs2/arrow-ipc" }
#arrow-schema = { path= "/Users/andrewlamb/Software/arrow-rs2/arrow-schema" }
#arrow-select = { path= "/Users/andrewlamb/Software/arrow-rs2/arrow-select" }
#arrow-string = { path= "/Users/andrewlamb/Software/arrow-rs2/arrow-string" }
#arrow-ord = { path= "/Users/andrewlamb/Software/arrow-rs2/arrow-ord" }
#arrow-flight = { path= "/Users/andrewlamb/Software/arrow-rs2/arrow-flight" }
#parquet = { path= "/Users/andrewlamb/Software/arrow-rs2/parquet" }
8 changes: 4 additions & 4 deletions datafusion-cli/src/main.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -617,8 +617,8 @@ mod tests {
| filename | file_size_bytes | metadata_size_bytes | hits | extra |
+-----------------------------------+-----------------+---------------------+------+------------------+
| alltypes_plain.parquet | 1851 | 8882 | 2 | page_index=false |
| alltypes_tiny_pages.parquet | 454233 | 269266 | 2 | page_index=true |
| lz4_raw_compressed_larger.parquet | 380836 | 1347 | 2 | page_index=false |
| alltypes_tiny_pages.parquet | 454233 | 269074 | 2 | page_index=true |
Copy link
Contributor Author

@alamb alamb Jan 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this reduction in metadata size is a direct consequence of @WaterWhisperer's PR to improve PageEncoding representation

| lz4_raw_compressed_larger.parquet | 380836 | 1339 | 2 | page_index=false |
+-----------------------------------+-----------------+---------------------+------+------------------+
");

Expand Down Expand Up @@ -648,8 +648,8 @@ mod tests {
| filename | file_size_bytes | metadata_size_bytes | hits | extra |
+-----------------------------------+-----------------+---------------------+------+------------------+
| alltypes_plain.parquet | 1851 | 8882 | 5 | page_index=false |
| alltypes_tiny_pages.parquet | 454233 | 269266 | 2 | page_index=true |
| lz4_raw_compressed_larger.parquet | 380836 | 1347 | 3 | page_index=false |
| alltypes_tiny_pages.parquet | 454233 | 269074 | 2 | page_index=true |
| lz4_raw_compressed_larger.parquet | 380836 | 1339 | 3 | page_index=false |
+-----------------------------------+-----------------+---------------------+------+------------------+
");

Expand Down
Loading