Adding neural network models to Safe-DS library

[Gerhardsa0]

What the user should be able to do:

model = new NN_Model()
model.add(RNN_Layer(output_size))
model.add(FNN_Layer(second_output_size))
model.train(TaggedTable,...)

What are Layers in Safe-DS:

• A layer is one layer of neurons, which can be combined in the NN_Modell to create bigger neural networks
• the output size should be given, but the input size should be figured out dynamically
• A layer knows how to create itself, but doesn't hold an entity of an PyTorch Model
• The layer type is defined by its class type
• all layers share similar behavior in the abstract Layer-class

What is the NN_Modell in Safe-DS:

• The NN_Modell creates the neural network by one or more layers
• the class realizes all neural network operations
• the class contains the PyTorch Model

new functionalities:

• to_dataloader() function in Tagged Table, which creates the input for the PyTorch Models from the tagged table

Open Questions:

• where do create the windows, and how do we handle them with other layers except RNN?
• the train function is overloaded (because we define the windows there)
• should the user be able to modify the activation functions?
• how do we track the input sizes of the different models?

What are your thoughts on this @lars-reimann?

[lars-reimann]

Is into_Dataloader only supposed to be used internally? You marked it as protected in the UML diagram but also mentioned it prominently in your description.

should the user be able to modify the activation functions?

Yes.

• where do create the windows, and how do we handle them with other layers except RNN?
• how do we track the input sizes of the different models?
• the train function is overloaded (because we define the windows there)

How about having an explicit input layer? Then define the window sizes etc. there.

[Gerhardsa0]

Is into_Dataloader only supposed to be used internally? You marked it as protected in the UML diagram but also mentioned it prominently in your description.

Yes

should the user be able to modify the activation functions?

Yes.

Okay, then we could add "function types", so the user gets autofill for the functions.

How about having an explicit input layer? Then define the window sizes etc. there.

This is interesting, this could avoid overloading the Table-creation or the training function.
In my opinion, Keras has this issue, while loading windows on a low level of abstraction(NumpyArrays) and a complicated way of using the function.
(see here the docs of the function: https://www.tensorflow.org/api_docs/python/tf/keras/utils/timeseries_dataset_from_array
see here the function in example: https://keras.io/examples/timeseries/timeseries_traffic_forecasting/)

Meanwhile, PyTorch doesn't have the ability in their API and only in PyTorch Lightning, and the function there is meant for experts.

With one input layer, we could create an abstract interface for windows, which would take the work of the user, because he does not need to do the low level data wrangling.

[Gerhardsa0]

pending ToDos:

• Adding early_stopping function for training
• implement callabacks
• creating windows by reference
• loss functions?
• baseline "models", static models to beat