MLforecast - test several features

[ssatti0]

I'm aware of how to try several models in MLforecast, and then choose the "best" one based on metrics. However, I'd also like to try several combinations of features and then be able to choose the "best combination" in the final forecast. How can I do that in Nixtla? Thank you,

[jmoralez]

Hey. Do you mean using AutoMLForecast?

[ssatti0]

@jmoralez thanks for the pointer. AutoMLForecast works for this purpose, but I also noticed that the models come with pre_decided hyperparameters e.g. AutoRandomForest has n_estimators:731 et.c These are different from the defaults in MLforecast.

1. Is there a way to change these in AutoMLForecast?
2. Is there an example somewhere on how to CrossValidate after Hyperparameter Tunning of features in AutoMLForecast?

[jmoralez]

1. Yes, that guide shows how you can define your search space here.

2. The resulting score from each trial is the average of the loss across the CV folds, you can get it with auto_mlf.results_[your_model].best_trial.value

[ssatti0]

So, I've run my trials for feature tuning, and now I need to refit the models with the best trial results. How can I do that. Below is the code I used for feature tuning.

def my_init_config(trial: optuna.Trial):
# Suggest parameters for ExponentiallyWeightedMean
alpha = trial.suggest_float('alpha', 0.1, 0.9)

# Suggest parameters for RollingMean
window_size = trial.suggest_int('window_size', 1, 12)
min_samples = trial.suggest_int('min_samples', 1, window_size)

lag_transforms = [
    ExponentiallyWeightedMean(alpha=alpha),
    RollingMean(window_size=window_size, min_samples=min_samples),
]

lag_to_transform = trial.suggest_categorical('lag_to_transform', [1, 2])

# Suggest a list of lags to tune
lags = trial.suggest_categorical('lags', [
    [i for i in range(1, 4)],  # Example: [1, 2, 3]
    [2 * i for i in range(1, 7)],  # Example: [2, 4, 6, 8, 10, 12]
    [i for i in range(1, 12, 2)],  # Example: [1, 3, 5, 7, 9, 11]
    [i for i in range(1, 13)],  # Example: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
])

return {
    'lags': lags,
    'lag_transforms': {lag_to_transform: lag_transforms},
}

Define the models

models = [
AutoLinearRegression(config=None),
AutoLightGBM(config=None),
AutoXGBoost(config=None),
AutoRandomForest(config=None)
]

Example usage

auto_mlf = AutoMLForecast(
models=models,
freq='MS', # Monthly frequency
season_length=12, # Yearly seasonality
init_config=my_init_config,
).fit(
train_df,
n_windows=2,
h=12,
num_samples=2,
)

[jmoralez]

The final models are automatically trained with the best parameters using all of the data, so if you save them or call predict it'll use those models and the forecasts will start after your train end.

[ssatti0]

Thanks Jose, Let me ask my question another way. I'm running both StatsForecast and MLForecast(with different models), then I merge both and find the best model based on the best metric. All I need now is a way to find the best combination of features to use. How can I do this? Sara     On Thursday, October 24, 2024 at 05:07:55 PM EDT, José Morales ***@***.***> wrote: - Yes, that guide shows how you can define your search space here. - The resulting score from each trial is the average of the loss across the CV folds, you can get it with auto_mlf.results_[your_model].best_trial.value — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>