Sketch / MVP: Upgrade to arrow-rs 55.0.0 / parquet 55.0.0 / object_store 0.12.0

[alamb]

• Related to feat: upgrade to DataFusion 47.0.0 delta-rs#3378

This is part of testing delta.rs with DataFusion 37 (which will, among other things, use a new version of arrow-rs and object_store)

I don't really intend this PR to be merged as is -- it is simply enough modifications to get the code to compile to unblock me testing delta.rs

Notes for anyone who finds this PR:

1. I could not figure out how to make chrono work correctly with 53/54/55. I suspect what is needed is to feature gate a chrono crate the way you feature gate arrow and pick the right version when needed
2. the HDFS object store crate uses object_store 0.11.0 so that woudl need to be updated to 0.12.0 before updating this PR

Note this is based on the v0.8.0 tag

[alamb]

I am pretty sure that simply deleting support for the older arrow releases is not the right approach for this crate but it was the most expedient for my purposes

[zachschuermann]

awesome, thanks @alamb! just saw arrow 55 is out, thanks for starting this - ill pick it up and see if we can support arrow 55 soon! regarding your questions:

1. I could not figure out how to make chrono work correctly with 53/54/55. I suspect what is needed is to feature gate a chrono crate the way you feature gate arrow and pick the right version when needed

yea we have pinned chrono so that we can support arrow 53 (some context here) - perhaps we can just get by with dropping 53 support now and just do 54/55?

2. the HDFS object store crate uses object_store 0.11.0 so that woudl need to be updated to 0.12.0 before updating this PR

ah yep - I can try to follow up over there to see if they can bump too!

[alamb]

yea we have pinned chrono so that we can support arrow 53 (some context #784) - perhaps we can just get by with dropping 53 support now and just do 54/55?

I don't think I am in a position to judge this 🤷 -- that will be on the maintainers of this crate I think

[zachschuermann]

did the same thing for the hdfs-native-object-store crate to update to object_store 0.12: datafusion-contrib/hdfs-native-object-store#15

[rtyler]

@alamb Thanks for taking a crack at this, I noticed 55 yesterday when doing
some unrelated work too and queued this up on my todo list 😄

Personally I think the hdfs-native-object-store is a boat anchor that should be
jettisoned as a dependency. I begrudgingly update the deltalake-hdfs package
because somebody donated that code a while ago and it has been nothing but
trouble.

I'll see what I can do with the chrono juggling to ensure that kernel try to keep support for multiple arrow versions via feature flags. If we can get that to work, then the arrow_55 flag could go into a 0.9.1 without much trouble 🤞

[Kimahriman]

Personally I think the hdfs-native-object-store is a boat anchor that should be jettisoned as a dependency. I begrudgingly update the deltalake-hdfs package because somebody donated that code a while ago and it has been nothing but trouble.

Donator of the code here. What trouble does it cause? First I'm hearing of any issues. Just something with transitive dependencies?

I didn't notice there was a new object store release, but happy to push a new release out asap for it.