add user-defined handling of NaN and inf to RDataFrame statistics

[ynikitenko]

Feature description

I analysed a tree in PyROOT with RDataFrame. Statistics for most variables could be calculated correctly. However, for one column I got nan-s. After further inspection it turned out that one event from 8 millions contained a NaN in that column. I tried Stats and Mean, and they both returned NaN. NaN is the correct answer for that data, but it is not practical.

The possible solutions could be:

1. RDataFrame could optionally filter irregular numbers (like NaN or inf).
2. Statistics functions of RDataFrame optionally support that. Tree Drawer simply ignored that value and calculated the mean (already noted here). One could provide an option to Stats() or Mean() to ignore NaN-s.
3. Documentation is added for user to handle that. There is a similar section Working with missing values in the dataset in the RDataFrame reference. NaNs are common in statistical analysis, so they would be worth explanation too.

The approach with RDataFrame looks cleaner; at the same time, another method like "FilterNaNs" for RDataFrame looks like feature creep. In Python that could be solved with a keyword argument (option); for C++ it is less clear to me (5 overloaded functions neither look good). An improved guideline is always possible.

Alternatives considered

I could solve it in PyROOT with

ROOT.gInterpreter.Declare("auto myisnan = static_cast<bool (*)(float)>(&isnan);") dr_cmx_no_cut = df.Filter("!myisnan(displacement.dr_proj_cmx_cm)")\ .Stats("displacement.dr_proj_cmx_cm")

However, that is a) non-evident, especially for Python users, b) filtering the same column with a dedicated function looks redundant (maybe there is a better syntax for that).

Additional context

Not sure whether it is connected, but there was a similar topic on the ROOT Forum.

[ferdymercury]

In Scipy they solve this with an extra arg: https://docs.scipy.org/doc/scipy/dev/api-dev/nan_policy.html

[dpiparo]

Thanks for taking the time to share your user story: this approach helps us greatly!

In general, I am inclined to say that if NaNs appear in a dataset which is supposed to be represent physical quantities, something already went wrong before the analysis. If these NaNs are well understood, and they can be skipped without ignoring any fundamental issue, I therefore see the option you illustrated above, the filtering, as a fine way to proceed. The filtering could even be further simplified by using the std::isnan function template (e.g. std::isnan<float>(myfloat)). In your case (correct me if I am wrong!) that would look like:

dr_cmx_no_cut = df.Filter("!std::isnan<float>(displacement.dr_proj_cmx_cm)").Stats("displacement.dr_proj_cmx_cm")

I let the experts comment further.

[hageboeck]

filtering could even be further simplified by using the std::isnan function template (e.g. std::isnan<float>(myfloat)). In your case (correct me if I am wrong!) that would look like:

dr_cmx_no_cut = df.Filter("!std::isnan<float>(displacement.dr_proj_cmx_cm)").Stats("displacement.dr_proj_cmx_cm")

This solution looks to be the best for me. I believe NaN should show up in the statistics if it's in the input dataset, because that's what you would expect in any "standard" floating-point computation.
The filter expression mentioned above should be the most concise way to implement the required filter, and could easily be extended to inf.

[ynikitenko]

I think that isnan is not a template, but an overloaded function. This does not work for me in ROOT:

root [0] std::isnan<float>
ROOT_prompt_0:1:1: error: reference to overloaded function could not be resolved; did you mean to call it?
std::isnan<float>
^~~~~~~~~~~~~~~~~
/usr/include/c++/14.2.1/cmath:1196:5: note: possible target for call
isnan(_Tp)
^`

[hageboeck]

I think that isnan is not a template, but an overloaded function. This does not work for me in ROOT:

You are right, the most concise I could do was:

rdf.Filter("!std::isnan(x)").Mean("x")

assuming that there is a column called "x".

[ynikitenko]

Thank you. I added the third solution of improving documentation. At some moment it was difficult for me to use C++ templated functions, even though I read the PyROOT manual. Common patterns could be explained in RDataFrame docs.