Time-based cross-validation with consistent cutoff date

[laxhammar]

Hi,

I have a use-case where I want to do cross-validation over a set of time-series with different start and end dates, enforcing strict separation in time between training data and test data for each window. The current cross-validation function (MLForecast.cross_validation) will AFAIU, split the time-series into training data and test data based on the same ordinal index for current window? This is fine if time-series are perfectly aligned in time, ie they have the same end date. But in my case, where end dates are not the same across all time-series, the cutoff date will not be the same for all time-series for current window? Ie the training set for a particular window may include data points from the same time period as the test set? I believe this is a form of data leakage and hence will bias the CV results?

I tried using fill_gaps from utilsforecast.preprocessing to extend all the time-series to have the same end date, filling the missing dates with NaN values for y. Hoping that MLForecast.cross_validation would simply ignore (remove) any data points where y is nan during training and prediction? But this does not seem to work?

Any thoughts on this? How would you approach this?

[jmoralez]

Hey. Since the cross validation method is implemented with a fixed forecasting horizon, splitting by the same date when some series don't span the full horizon wouldn't work. The CV method pretty much only does fit + predict repeatedly, so you can manually compute your splits and run fit + predict for each of them. You can probably use the same forecasting horizon for all series and just perform an inner join against the validation set in each fold (mlforecast checks that the inner join doesn't loose any rows, that's why it wouldn't work in your case).

Please let us know if you have further doubts.

[laxhammar]

Hi José
Thanks for your reply. Okay, I will implement a procedure for the cross-validation with manual splits, shouldn't be to complicated.