[ENH] re-implement the logic of data loading to ensure good performance

[workhours]

the current implementation performance is poor. dataload process consumed 99% performance time while fitting.
please don't use traditional way of handling large group of time series, which organize multi-dimensions data in a 2-dimensions pd.DataFrame and manupulate it. use tensor please!
use DataFrame as input data type only, and manupulate it always by tensor or np.array, restore every usefull dimension from df columns. if DataFrame manupulation is inevitable, please use AI to find a vectorized way
never pre-process data on time series wise, thounds and thounds times of time cost there.

[fkiraly]

Yes, we know of the problem and are addressing it here, design issue for v2 data loaders:
#1736

We will release a number of versions with beta designs that you can try out - also, feedback on the issue is appreciated!

[jobs-git]

the current implementation performance is poor. dataload process consumed 99% performance time while fitting. please don't use traditional way of handling large group of time series, which organize multi-dimensions data in a 2-dimensions pd.DataFrame and manupulate it. use tensor please! use DataFrame as input data type only, and manupulate it always by tensor or np.array, restore every usefull dimension from df columns. if DataFrame manupulation is inevitable, please use AI to find a vectorized way never pre-process data on time series wise, thounds and thounds times of time cost there.

yes exactly this, it pre-process data real-time which causes the slowness. I prepared a PR that solves training slowness, and it does not need to do all of these micro-optimizations: #1850