feat(python): Native implementation of dataframe interchange protocol

[stinodego]

Closes #10035

Changes:

• Implement DataFrame.__dataframe__ natively, rather than relying on pyarrow.
• Move existing from_dataframe implementation to the new interchange module. For this function, we still rely on pyarrow's implementation (for now).

[stinodego]

@MarcoGorelli from your experience with the DataFrame API, could you give me a hint as to where the nan_as_null option should be implemented? I imagine we could do a fill_null of the data buffer when Column.get_buffers is called? Or should I be thinking about this differently?

[stinodego]

Do we allocate a validity buffer if there are no missing values?

@ritchie46 Could you answer this one for me?

[ritchie46]

Do we allocate a validity buffer if there are no missing values?

@ritchie46 Could you answer this one for me?

No, we don't.

[MarcoGorelli]

How to implement nan_as_null option? Initial idea would be to call fill_null on the data buffer

I think that's the idea, but in practice it looks like for both pyarrow and for pandas it's not implemented

https://github.com/pandas-dev/pandas/blob/b864b8eb54605124827e6af84aa7fbc816cfc1e0/pandas/core/interchange/dataframe.py#L30-L42

https://github.com/apache/arrow/blob/7cbbd3ee95bc0f7d20bb358e2978e2eb18a05304/python/pyarrow/interchange/dataframe.py#L56-L63

cc @AlenkaF - do you know about how nan_as_null is meant to be implemented?

[rgommers]

I imagine we could do a fill_null of the data buffer when Column.get_buffers is called?

That sounds right to me, assuming that that is the call where you ensure the data is accessible in memory. In case calling __dataframe__ will already trigger materialization for whole columns, then I'd suggest doing it at that point.

I'm not sure why it isn't implemented in other libraries yet, probably just skipped for convenience. It shouldn't be a big burden to do so I'd think though. And at least in the case of pandas with numpy floating-point dtypes, it is in principle useful - nan is used as a sentinel for missing data there, and those dtypes don't have separate NA/null support.

[stinodego]

Thanks for the input.

I guess setting nan_as_null=True and allow_copy=False should fail if there are any missing values to fill?

Implementing this for our Float types is trivial. I guess we can do this for Integer types as well though it will trigger a cast to Float64 and no longer be zero copy. Utf8 types can be filled with a string "NaN", and for Categorical I guess we can add a category called "NaN".

I don't know what to do with Boolean types or Datetime types though. Your input is appreciated!

Though now I am thinking I should probably save it for a separate PR. It's not trivial and this PR is already huge.

[rgommers]

I guess setting nan_as_null=True and allow_copy=False should fail if there are any missing values to fill?

I think so, yes.

Implementing this for our Float types is trivial. I guess we can do this for Integer types as well though it will trigger a cast to Float64 and no longer be zero copy. Utf8 types can be filled with a string "NaN", and for Categorical I guess we can add a category called "NaN".

My first thought here was that it should only apply to columns with floating-point dtypes. However, after doing some digging to confirm that, I instead found data-apis/dataframe-api#125. So perhaps don't do anything here, and if we confirm on that issue that no one needs it, we should instead follow through and remove the keyword.

[stinodego]

So perhaps don't do anything here

Great, thanks for the insights. In that case, I'll not implement anything for this right now.

[stinodego]

Discussed with Ritchie and this is going in 🚀

Next up: from_dataframe

[AlenkaF]

cc @AlenkaF - do you know about how nan_as_null is meant to be implemented?

We have a default set to False in PyArrow implementation, otherwise it results in an error. I saw you have already found a connected issue data-apis/dataframe-api#125 and submitted a PR for it data-apis/dataframe-api#226. Great!

And sorry for late response!