Save train/val/test splits

[valosekj]

Imagine we want to compare models trained with this framework with models trained on the same data using another tool. To make the comparison transparent and reproducible, both models need to be trained with the same train/val/test splits.

Feature request: allow saving train/val/test splits for each replicate, trial, and cross-validation.

[SomeoneInParticular]

We already do this implicitly, actually; so long as your study and data config don't change, all splits are identical no matter the model being run. This is accomplished by via forcing the splitters to always reset their RNG state so abide by one we define during init (which is itself determined by the RNG seed given by the user).

Specifically, this function prepares the entire study before its run, including ensuring RNG replication.

I do still need to add some automated tests to ensure this remains the case though; mind if I hijack this issue for that?

[valosekj]

We already do this implicitly, actually;

Nice! Can we easily save/export the splits into a JSON/YML file? (To be reusable by other tools)

I do still need to add some automated tests to ensure this remains the case though; mind if I hijack this issue for that?

Sure!

[SomeoneInParticular]

Nice! Can we easily save/export the splits into a JSON/YML file? (To be reusable by other tools)

Will look into it once the documentation pass is completed. I doubt it'd be too hard, just want to focus on that for the time being,