Support Aggregation serialization in pylibcudf
#17469
Conversation
pentschev
added
feature request
pentschev
changed the title
pentschev
changed the title
Aggregation serialization in pylibcudf
|
The CI errors from custreamz look related to this PR but I don't think they are in fact. I can locally reproduce them only with upstream Dask/Distributed, but not with |
| cdef correlation_aggregation *correlation_cast | ||
| cdef covariance_aggregation *covariance_cast | ||
|
|
||
| if self.kind() is Kind.SUM: |
There was a problem hiding this comment.
Can we replace this cascade with a singledispatch helper function? It's a lot of clauses to fail through in the worst case.
| @@ -170,3 +174,66 @@ cdef extern from "cudf/aggregation.hpp" namespace "cudf" nogil: | |||
| null_policy null_handling, | |||
| null_order null_precedence, | |||
| rank_percentage percentage) except +libcudf_exception_handler | |||
|
|
|||
| cdef extern from "cudf/detail/aggregation/aggregation.hpp" \ | |||
| namespace "cudf::detail" nogil: | |||
There was a problem hiding this comment.
Hmm we don't expose pretty much any cudf detail APIs to pylibcudf, and I don't want to start here. Can we open an issue about these? If these are attributes that are absolutely necessary to reconstruct the serialized types, then we should discuss exposing them publicly in libcudf.
There was a problem hiding this comment.
Thanks for the comment Vyas, I've opened #17630. Please let me know if there's anything else I can do.
Description
Support
Aggregationserialization in pylibcudf. This is required to provide distributed support for cudf-polars.Unfortunately, serialization of the
Aggregationclass requires access to implementation details in libcudf, so those needed to be exposed to Cython. Fortunately, all the attributes required are already public so there are no changes required in libcudf.Checklist