[BUG] Shared-memory from_arrow no longer zero-copy

[bryevdv]

cc @trxcllnt @galipremsagar

Looking into implementing zero-copy from_arrow for node-rapids for Arrow arrays stored in IPC format in shared memory, we noticed that Python cudf from_arrow delegates to libcudf.interop.from_arrow which will always result in a copy. Evidently previously it was possible to avoid a copy when and arrow table had been created via GpuArrowReader from GPU memory.

[beckernick]

@bryevdv it looks like we've been delegating to from_arrow for quite a while based on the git blame history (two separate hyperlinks). Is this a recent change in behavior?

[trxcllnt]

@beckernick no, the change happened when we switched Python to using libcudf.from_arrow. Previously all the from_arrow logic was handled in Python, and we zero-copy moved the Arrow buffers into cuDF buffers via buffers_from_pyarrow.

[github-actions]

This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.

[bryevdv]

Still needs to be addressed, AFAIK.

[shwina]

Is it not possible for C++ to also zero copy similarly where applicable? The reason we moved to the C++ impl was to avoid duplicates logic between C++ and Python.

[trxcllnt]

@shwina yeah it would be possible, but the current implementation returns a unique_ptr<cudf::table>, meaning it must copy the input data so the return can be a valid owning type. For true zero-copy, we'd need a version that returns a cudf::table_view (from which a cudf::table could be constructed if a copy was desired).

[shwina]

cc @jrhemstad for advice on the API here, but maybe we could return something like a pair<table_view, memory_owner>? In the IPC case, the memory_owner would own nothing?

[vyasr]

I think the new solutions to this issue will come from work on #14926.

[vyasr]

With the closing of #14926 this is resolved. We now use the C Data Interface for all Arrow interop, so no extraneous copies will be made.