[FEA] Reduce arrow library dependencies in cudf

[GregoryKimball]

Is your feature request related to a problem? Please describe.
Arrow is in general a difficult dependency to work with, increasing build system complexity and fragility on its own while simultaneously expanding the full dependency tree, which particularly complicates use cases like conda where it leads to meaningful constraints on core system packages like protobuf, abseil, or the AWS SDK. This often hinders developer velocity when builds or CI are broken, but can also have far-reaching impacts when it creates problems with installation or running in specific environments. To prevent this, we would like to reduce or remove our dependence on Arrow libraries entirely.

Currently cudf makes use of Arrow in various ways at different levels of the stack. The primary uses of Arrow boil down to interop with host Arrow data and I/O with specific types of files. This involves interaction at both the Python layer via pyarrow, at the Cython layer (also via pyarrow), and in C++. Both Cython and C++ interactions are particularly problematic because they involve C-level interactions, which sets ABI-level constraints that are significantly tighter than we would like while also significantly complicating build (CMake, Python builds) and packaging (narrow Arrow version support ranges leading to limited support of other packages in the dependency tree). Python interactions are generally less difficult to work around, especially since Python code can be written to dynamically adapt to the pyarrow version.

Describe the solution you'd like

We should look to remove the Arrow dependencies from the various layers of cudf (Java, Python, Cython, C++) to the greatest extent possible, ideally entirely.

For Arrow Array interop code, this can be accomplished by using the Arrow C Data Interface (see #5097), which provides an ABI-stable way to interchange Arrow data without directly using Arrow libraries. To make this even easier, the nanoarrow library was created to support clients that wish to produce or interpret Arrow C Data and Arrow C Streams structures, without having to include a dependency on libarrow. We can make use of that (see also #13678 which discusses this in depth). For Python interaction we can use Arrow's pycapsule interface, which provides a standard way to interchange this data from Python. We can write Cython code leveraging this interface to get Arrow C Data from pyarrow objects without relying directly on pyarrow's Cython, therefore also allowing us to remove this dependency from the Cython layer.

For I/O, the question is a bit trickier. We currently have limited usage of libarrow headers in our C++, and those features largely exist only for Python support for reading Arrow's NativeFiles. We could in principle remove those from the C++ entirely, which would in turn allow us to remove libarrow as a dependency of libcudf. However, libcudf tests would still need libarrow (removing that dependency would require significant additional work). Moreover, those features would still be used by cudf Cython, so we would just be limiting the dependency. However, this could at least allow us to remove Arrow as a build-time dependency for both libcudf and the low-level pylibcudf Python API (#13921) that we are currently developing, which would still be a significant improvement since it would avoid imposing the Arrow dependency on low-level consumers of our APIs at the Python level. Then we could come back to working on replacing the cudf Cython usage.

Based on the above, the current plan is the following:

1. Remove libarrow as a dependency of libcudf/pylibcudf:
   a. Remove the compiled parts of arrow_io_source.cpp and make arrow_io_source.hpp a standalone header not compiled by anything in libcudf.
   b. Rewrite cudf Cython to use the arrow headers directly.
   c. Add new interop code that uses the Arrow C Data interface (see Add to_arrow_device function to cudf interop using nanoarrow #15047)
   d. Rewrite Python interop code to call through to the new interfaces
   e. Remove the old Cython bindings for interop
2. Remove pyarrow Cython linkages from cudf Cython
   a. This will require some exploration as to how we can maintain performant file reading. We may have to implement our own minimal version of something like Arrow's NativeFile reader interface.
   b. Once the above is done, we'll need to rewrite cuIO C++ to consume this interface and remove the current functions.
3. Rewrite libcudf tests to remove libarrow dependence.
   a. This will require further investigation into how tests could be rewritten without Arrow. One possibility would be rewriting these tests as pylibcudf tests (see [FEA] Add tests of pylibcudf #15133) that use pyarrow instead (only the Python API, no Cython). That would give us access to the same functionality without tying us to linking to the libarrow library

Additional context

Code pointers where libarrow is used in 24.04

Source file	Arrow include	Notes
detail/interop.hpp	api.h	to_arrow_array uses many array classes: arrow::*Array, arrow::TimeUnit::*, arrow::*Type also arrow::MemoryPool, arrow::Scalar, arrow::Table. I believe all of these are covered by nanoarrow
include/cudf/interop.hpp	api.h	uses arrow::Table, arrow::MemoryPool, arrow::default_memory_pool, arrow::Scalar. I believe all of these are covered by nanoarrow
include/cudf/io/arrow_io_source.hpp	filesystem/filesystem.h io/interfaces.h	uses arrow::io:RandomAccessFile, arrow::fs::FileSystem. See #13698 for the work to refactor arrow_io_source out of datasource
include/cudf/io/arrow_io_source.cpp	buffer.h filesystem/filesystem.h result.h	uses arrow::Buffer, arrow::fs::FileSystemFromUri,
src/io/utilities/datasource.cpp	io/memory.h	to be solved by #15189

Test file	Arrow include	Notes
tests/interop/arrow_utils.hpp	util/bitmap_builders.h for arrow::internal::BytesToBits	Also uses many arrow types such as: arrow::Array, arrow:DictionaryArray, arrow::dictionary, arrow::Table, arrow::Decimal128Builder, arrow::decimal, arrow::default_memory_pool, arrow::ListArray, arrow::list , arrow::Buffer, arrow::StringBuilder, arrow::StringArray , arrow::BooleanArray, arrow::BooleanBuilder needs research - can all of these references be migrated to nanoarrow?
tests/io/arrow_io_source_test.cpp	io/api.h filesystem/filesystem.h filesystem/s3fs.h util/config.h	uses arrow::fs::FileSystemFromUri, arrow::fs::EnsureS3Finalized
tests/io/json_test.cpp	io/api.h	Uses arrow::io::ReadableFile as part of a test for reading from an ArrowFileSource
tests/io/csv_test.cpp	io/api.h	uses arrow::io::ReadableFile
tests/quantiles/percentile_approx_test.cpp	util/tdigest.h	uses arrow::internal::TDigest. presumably we could replace this with our own limited implementation

[vyasr]

Other relevant issues:

• [QST] Would the new Arrow C Data Interface be useful for CuDF? #5097
• [QST] Evaluate nanoarrow as part of any future Arrow C Data Interface exploration #13678

[vyasr]

One important correction to the table above, nanoarrow does not support scalars and neither does the Arrow C data interface, which focuses only on arrays. Previously this was a significant concern due to concerns with Python interoperability, but it looks like pyarrow 13 patched in some critical support for scalar->array conversion that ought to make this concern moot for us now. See #15213

[vyasr]

I've substantially updated this issue with a more holistic discussion of how we can stage the removal of Arrow at different layers of our library (C++, Cython, Python).

[zeroshade]

Just wanted to point out that a big step towards this is the work started in #15047 and discussed in #14926. Once the to_arrow_device and from_arrow_device functionality is crafted, it should be pretty simple to re-implement several of the interoperability areas in terms of the Arrow C Data and Device interfaces, making strides towards eliminating the need for libarrow as a dependency in favor of nanoarrow

EDIT: Just re-read and saw this was already mentioned in the OP.... oops 😄

[vyasr]

Yup! Your PR was top of mind while writing up this issue!

[jakirkham]

Assuming this lets us drop NumPy from cuDF's build dependencies, would make sure to drop these lines (with comments) when that happens

cudf/dependencies.yaml

Lines 395 to 399 in 508bdea

	- output_types: pyproject
	packages:
	# Hard pin the version used during the build.
	# Sync with conda build constraint & wheel run constraint.
	- numpy==2.0.*

cudf/conda/recipes/cudf/meta.yaml

Line 67 in 508bdea

- numpy 2.0

Edit - Reflects current state since NumPy 2 changes in PR: #16300