Q: Does EvoTrees use ordered boosting?

[tecosaur]

Hello again 👋

Comparing the python boosted decision tree libraries, one of the design aspects of CatBoost that sounds rather appealing is it's use of ordered boosting to mitigate prediction shift (https://proceedings.neurips.cc/paper_files/paper/2018/file/14491b756b3a51daac41c24863285549-Paper.pdf).

Since EvoTrees development started after CatBoost started using ordered boosting, I'm wondering if this design made its way into EvoTrees?

[jeremiedb]

Hello! Yes I'm aware of these ordered boosting methods used in CatBoost. However, no such approach has been integrated in EvoTrees at the moment.
I've however evaluated the potential caveat from bias as described in the CatBoost paper.
The approach I took was that of out-of-sample leaf prediction: build distinct histograms and statisticcs for a set of observations used to develop the tree split, and use the distinct subset of observations for deriving the leaf predictions. Such approach appeared to have some benefit for small datasets, but no added value for large ones. This appear aligned with the intuition as for large dataset, one can expect the out-of-sample estimate to be fairly close to the one that would have been used to derive the split.
Also, AFAIK, CatBoost only uses that methodology for small datasets.

Below the quick assessment/observations from out-of-sample approach with EvoTrees and the associated brach:

• https://github.com/Evovest/MLBenchmarks.jl/tree/oos-pred?tab=readme-ov-file#oos-branch
• main...oos-pred

My interest has been more around proper application of regularization methodology, for example the work around credibility-based loss: https://evovest.github.io/EvoTrees.jl/dev/tutorials/cred-loss/

I have integrated much support for Categorical features, other than an efficient handling of Categorical features where the number of categories is limited (<= nbins). CatBoost target encoders utilities would be nice to have, but it hasn't been a priority so far given my use cases.