Performance Issue with Combiner Function on Large Dataset (2M Rows, 56 Columns)

[IbtesamKh]

I am currently using Toad version 0.1.4 and encountered a performance issue when using the combiner function on a dataset with 2 million rows and 56 columns. The execution time is taking approximately 3 days to complete, which is significantly longer than expected.

Parameters used:
combiner.fit(X=x_train,
y=x_train['label'],
method='chi',
min_samples = 0.05,
exclude=exclude_list)

I wanted to inquire if:

• This performance issue has been addressed in a later version of Toad.
• There are any recommended methods or optimizations to speed up the combiner function for large datasets.
• If there are any specific configurations or parameters I can adjust to improve performance.

[biginflation]

Here is my two cents. By checking chi merge code, you may find the stopping rule for this funciton is

if n_bins is None and min_samples is None and min_threshold is None:
    n_bins = DEFAULT_BINS

if min_samples and min_samples < 1:
    min_samples = len(feature) * min_samples


...


while(True):
        # break loop when reach n_bins
        if n_bins and len(grouped) <= n_bins:
            break

        # break loop if min samples of groups is greater than threshold
        if min_samples and c_min(c_sum_axis_1(grouped)) > min_samples:
            break

So I believe in your case even for one feature, the chi merge will stop only when the number of observations in each bin is nearly 2M*0.02 = 100k (Supppose this feature has unqiue 2M values). Which is very time consuming to compress nearly 2M observations into ~20 bins