GH-38558: [C++] Add support for null sort option per sort key

[Taepper]

See #38584 for original PR. Will be quoted for this PR description.

Rationale for this change

support multi sortkey nulls first.

order by i nulls first, j, k nulls first;

The current null sorting only supports all sortkeys, not a certain sortkey, so NullPlacement is extended to the SortKey field. Since the underlying framework is very well written, when modifying this function, you only need to pass the null_placement of each SortKey in. That’s it.

What changes are included in this PR?

1.SortKey structure, NullPlacemnt transfer logic, sorting logic and Ording related, test related
2.Substriait related.
3.c_glib related.
4.SelectK related.
5.RankOptions related.

Are these changes tested?

yes, I changed the code inside vector_sort_test.cc and performed additional tests.

Are there any user-facing changes?

yes, pg database include null sorting of multiple sort keys.

This PR includes breaking changes to public APIs. (If there are any breaking changes to public APIs, please explain which changes are breaking. If not, you can remove this.)

I amended the original PR to be less breaking in public APIs.

Still Ordering, SortOptions, RankOptions, and RankQuantileOptions now accept a std::optional<NullPlacement> instead of NullPlacement, which did lead to some changes in downstream APIs and bindings. I also need some help with fixing the c_glib bindings.

• GitHub Issue: [C++][Compute] Fix null sorting of multiple sort keys #38558

[Taepper]

Note that I fixed the failing CI runs on my fork

[pitrou]

Hi @Taepper , thanks for submitting this.

Design-wise, I think there are two possible APIs here:

1. (As done in this PR) Add a required NullPlacement in SortKey, and make the NullPlacement in SortOptions optional and deprecated
2. Add an optional NullPlacement in SortKey, and keep the required NullPlacement in SortOptions as fallback

Option 1 has the advantage that it's conceptually simpler once the deprecation period is over, but it comes with a minor API change and a slightly complicated deprecation period.

Option 2 is conceptually a bit more complicated (per-key + global fallback) but avoids breaking the current API and doesn't introduce any deprecation.

I'm not sure which one is better. @zanmato1984 @felipecrv Thoughts?

[Taepper]

Thank you for your comments!

I had the same considerations. I opted for option 1 because it provides the clear path forward for the post-deprecation API.

Maybe option 2 is worth more consideration for the missing API breakage.

Should the SelectKOptions (which did not have a NullPlacement before) receive an additional global fallback, or should it implement their fallback internally and non-configurable?

[pitrou]

Well, the API breakage isn't critical either IMHO. I'd just like to have opinions from other core developers. @lidavidm Perhaps?

[lidavidm]

As long as we're still doing major releases, then I think the slight breakage is acceptable in return for a nicer API in the end.

[Taepper]

Alright, also note that this tries to be analogous to the deprecation of skip_nulls in SetLookupOptions:

https://github.com/apache/arrow/pull/36739/files#diff-6bc7ecec6a4f7bcefc2511cde3bd809340ad0d94bb8f7cc5f4994063c798f2faR313

[pitrou]

Alright, also note that this tries to be analogous to the deprecation of skip_nulls in SetLookupOptions:

https://github.com/apache/arrow/pull/36739/files#diff-6bc7ecec6a4f7bcefc2511cde3bd809340ad0d94bb8f7cc5f4994063c798f2faR313

That's a good comparison point, thank you.

[zanmato1984]

I prefer a conceptually simpler model and sacrifice the API compatibility:

1. (As done in this PR) Add a required NullPlacement in SortKey, and make the NullPlacement in SortOptions optional and deprecated

[pitrou]

Ok, it seems everyone agrees with this approach, so let's go for it.