Auto encoding for categorical data during inference.

[trivialfis]

We are working on automatic re-encoding for categorical features during inference. This teaches the booster to handle data encoded differently than the training dataset and eliminates the need for a scikit-learn pipeline for data encoding when using DataFrame inputs.

• Python native
• Dask
• R
• PySpark
• Scala/Spark
• Thread safety
• https://github.com/scikit-learn/scikit-learn/blob/5b0ca3939854a3823beee6840b415a32ef16deb2/sklearn/utils/_tags.py#L36
• Demos
• cudf.pandas
• Polars
• Refresh updater.
• Training continuation.
• Validation datasets.
• Documents. Update the existing requirement on the ordinal encoder.
• Plot. Display name instead of code (optional).

Removed the spark variants, its dataframe doesn't have encoding. Use the StringIndexer instead.

Related:

• Strange behaviour with Xgboost when dealing with categorical data type, Possibly a bug #9676

Notes:
Looking into the Arrow CPU implementation, its compute module dispatches based on whether a null mask is present. If true, it tries to find consecutive valid values (called a run) and then iterates on this run. This way, it avoids having a predicate for every element for the validity check. The consecutive valid values are found using compiler builtins with leading nnz counting.

Related:

• XGBoost Fails with Polars DataFrames Containing Categorical Columns #10554

Tracking PRs:

• [breaking] [py] Drop support for datatable. #11070
• Extract array interface handlers. #11089
• Support all integer types in ubjson. #11094
• Various small cleanups. #11097
• [R] Ensure ProxyDMatrix creation keeps data until next iteration #11092
• Initial implementation of the ordinal recoder. #11098
• Small cleanups for DMatrix constructor. #11107
• Add datagen for testing string-based categorical data. #11114
• Cleanup CPU predict function. #11139
• Implement the container for categories. #11297
• Store categories from pandas. #11303
• Test dataset for dask dataframe with str columns. #11310
• Store categories from cuDF. #11311
• Store categories from iterator. #11313
• Auto re-coding for the CPU predictor. #11315
• Implement ordinal recoder for the GPU predictor. #11347
• [enc] Add tests for re-coding validation datasets. #11561
• Support categorical features from polars. #11565
• [enc] Add a cat accessor to the booster. #11568

[trivialfis]

@david-cortes May I ask how the tmp is kept alive during the construction of QDM?

xgboost/R-package/R/xgb.DMatrix.R

Line 572 in f4f3bd4

rm(tmp)

The setData from the proxy DMatrix only holds a reference (a C pointer) to the input data, if the data is released early then it's a use-after-free error. But testing the qdm.from_iterator seems fine.

[david-cortes]

@trivialfis In that function, the DMatrix is set in the line right before tmp gets deleted. Or are you saying that the data still needs to be kept after the DMatrix has been set? If so, at which point is it safe to deallocate the data?

Regarding the feature: since the idea is to have this feature in different interfaces, how would it work behind the scenes?

Would be ideal if the categorical encodings could get saved in the booster and be used in plots/trees-to-tables/jsons/etc. (#9927). Better yet if it's a standardized C-level attribute so that the encodings could survive transfers from one interface to another.

I see some potential difficulties though:

• Pandas allows arbitrary python object for categorical encodings (not json-friendly).
• Different data formats (pandas, arrow, polars, etc.) might have different limitations for what kind of values they allow as categories (e.g. strings-only vs. also integers).

[trivialfis]

Or are you saying that the data still needs to be kept after the DMatrix has been set

It needs to be kept until the next next call or reset call of the iterator. I got a bit confused since the data is deleted right after the setData call, but no access error. So, in theory, it should be kept after set, and until the next iteration.

Regarding the feature: since the idea is to have this feature in different interfaces, how would it work behind the scenes?

We will store the levels in the booster as you suggested. Things will be handled in C++, we might allow users to optionally disable the encoder for performance reasons (searching through levels is not cheap in the context of inference, especially with strings).

Better yet if it's a standardized C-level attribute so that the encodings could survive transfers from one interface to another.

Currently, I'm returning the categories in the arrow columnar format with the help of pyarrow. Haven't decided the exact return type yet, in my experimental code, it's just a Python map (dictionary) from feature names to arrow arrays.

Pandas allows arbitrary python object for categorical encodings

We accept only strings and some other primitive types like integers. Still working on the typing part. The pandas.Index (used for representing categories) should have the same type of feature column, so it can't be arbitrary and has to be something XGBoost can understand.

[trivialfis]

This has taken way too long due to the new year holiday. I will leave it to the next release.