Under the same model, same input, and same initialization, the gradients of the model exhibit differences.

[BiophiliaSWDA]

Environment

Software Environment:

• MindSpore version (source or binary):2.2.14
• Ubuntu20.04, Python 3.8, CUDA 11.6, cuDNN 8

Describe the current behavior

These two models have the same architecture, identical input, and the same initialization. However, there is a significant difference in their gradients after backpropagation. When we experiment with each API individually, this phenomenon does not occur. Currently, this issue is only observed with the LSTM API.

Describe the expected behavior

They should be same.

Steps to reproduce the issue

class Model_cDf5CgkzsFaikw3L4C7KgcmSun0p4zhi(mindspore.nn.Cell):
    def __init__(self):
        super(Model_cDf5CgkzsFaikw3L4C7KgcmSun0p4zhi, self).__init__()
        self.rnn1 = mindspore.nn.RNN(input_size=2048, hidden_size=2031, batch_first=True)
        self.rnn2 = mindspore.nn.LSTM(input_size=2031, hidden_size=1779, batch_first=True)
        self.rnn3 = mindspore.nn.LSTM(input_size=1779, hidden_size=1236, batch_first=True)
        self.linear = mindspore.nn.Dense(in_channels=1236, out_channels=10)

    def construct(self, x):
        x, _ = self.rnn1(x)
        x, _ = self.rnn2(x)
        _, (x, _) = self.rnn3(x)
        x = self.linear(x[-1])

        x = x
        return x


class Model_1748971508(nn.Layer):
    def __init__(self):
        super(Model_1748971508, self).__init__()
        self.rnn1 = paddle.nn.SimpleRNN(input_size=2048, hidden_size=2031, time_major=False)
        self.rnn2 = paddle.nn.LSTM(input_size=2031, hidden_size=1779, time_major=False)
        self.rnn3 = paddle.nn.LSTM(input_size=1779, hidden_size=1236, time_major=False)
        self.linear = paddle.nn.Linear(in_features=1236, out_features=10)


    def forward(self, x):
        x, _ = self.rnn1(x)
        x, _ = self.rnn2(x)
        _, (x, _) = self.rnn3(x)
        x = self.linear(x[-1])

        x = x
        return x

[zhouyifeng888]

The difference is likely due to the model's initialization weights. Different frameworks may use different default initialization methods. Even if the same type of initialization method is used, the default parameters and internal implementations may differ slightly. Or even if the implementation and default parameters are exactly the same, these initialization methods—such as Kaiming Normal—are usually random and follow the same probability distribution, but the actual values are still randomly generated. I previously compared MindSpore and PyTorch and found differences in their specific implementations. There are also slight differences between TensorFlow and PyTorch. I'm not very familiar with the Paddle framework, but I’ve heard that Paddle is quite similar to PyTorch in many aspects, such as having the same implementation for differentiation. So, Paddle may be consistent with PyTorch and somewhat different from MindSpore.

[wyqmath]

To unify model initialization, you can first initialize a source model and extract its weight state dictionary (state_dict), and then manually load these weights into the target model. For example, after loading and mapping the weights from a PaddlePaddle model to a MindSpore model, the correctness of the weight transfer can be verified by comparing the forward pass outputs of their intermediate layers.