[BUG Report]: Optimizer is null after loading a saved model

[bhairav-thakkar]

Description

When loading a previously saved sequential model with optimizer, the loaded model has optimizer = null. This leads to the training NOT continuing from the state before saving.

Reproduction Steps

I created a minimal LSTM network with Adam optimizer. I followed the steps outlined in #1246 (comment) to get model.save working. However, when I tried to load the model, I found the Optimizer to be null. My relevant code blocks are:

public void BuildModel()
{
    var seqArgs = new Tensorflow.Keras.ArgsDefinition.SequentialArgs()
    {
        Name = "LSTMModel",
        InputShape = new Tensorflow.Shape(TimeSteps, 1),
    };
    model = new Sequential(seqArgs);

    for (int i = 0; i < NumberOfLayers; i++)
    {
        var lstmArgs = new Tensorflow.Keras.ArgsDefinition.LSTMArgs()
        {
            Units = NumberOfCells,
            ReturnSequences = i < NumberOfLayers - 1,
            InputShape = i == 0 ? new Tensorflow.Shape(TimeSteps, 1) : null,
            Activation = new Tensorflow.Keras.Activations().Tanh,
            RecurrentActivation = new Tensorflow.Keras.Activations().Sigmoid,
            ActivityRegularizer = new Tensorflow.Keras.Regularizers().l2(L2Regularization)
        };
        var lstm = new LSTM(lstmArgs);
        model.add(lstm);

        if (i < NumberOfLayers - 1)
        {
            var args = new Tensorflow.Keras.ArgsDefinition.DropoutArgs()
            {
                Rate = DropoutRate,
            };
            var dropout = new Dropout(args);
            model.add(dropout);
        }
    }

    var denseArgs = new Tensorflow.Keras.ArgsDefinition.DenseArgs()
    {
        Units = 1,
        KernelRegularizer = new Tensorflow.Keras.Regularizers().l2(L2Regularization)
    };
    var dense = new Dense(denseArgs);
    model.add(dense);
}

        public void SaveModel()
        {
            var tf = Tensorflow.Binding.tf;
            string dir = @"C:\Users\bhair\Desktop\tmp";
            var modelPath = Path.Combine(dir, "LSTMModel");
            model.save(filepath: modelPath, overwrite: true, include_optimizer: true, save_format: "tf");

            Adam a = (Adam)optimizer;


        }


        public void LoadModel()
        {
            var tf = Tensorflow.Binding.tf;
            string dir = @"C:\Users\bhair\Desktop\tmp";
            var modelPath = Path.Combine(dir, "LSTMModel");
            model = (Sequential)tf.keras.models.load_model(modelPath, compile: true);

            this.optimizer = model.Optimizer;
        }

after loading, this.optimizer is null.

Known Workarounds

No response

Configuration and Other Information

Download the Tensorflow.Net master repo on 03 January 2025 and referenced the Tensorflow.Keras and Tensorflow.Bindings project from my project. Using SciSharp.Tensorflow.Redist 2.16.0.

[yangmijun]

I try to load a saved model, and "tf.keras.models.load_model" throw out a "System.NotImplementendedException", also I still want to know how to reload a saved model then use it to predict or train in TensorFlow.net.
I can not find any saved model load sample code in "SciSharp STACK examples", only find sample about "graph.Import", but I think it's a different format.

[yangmijun]

I tracked source code, and found that, the saved model folder has no "keras_metadata.pb", that cause the "System.NotImplementendedException", but the saved model folder generated by tf 2.18, do not have "keras_metadata.pb". any more.

[bhairav-thakkar]

I downloaded the code and followed the steps outlined in #1246 (comment). I used.Net framework 4.8. Then, when you use model.save, it does produce various files including keras_metadata.pb. Note that you have to use the "tf" format only, otherwise it is not completely implemented yet.

After saving, when we load the model back, the weights are loaded, but the optimizer is not. I thought that if we load the file including optimizer, we should be able to continue training from where we saved.

However, this did not happen. I saved the model, on file and saved the optimizer (Adam) in a variable. Then, I loaded the model back from the files (without optimizer as it was anyway null). And then applied the Adam optimizer I had saved. Still, the training did NOT continue from the point when I had saved the model.