anomalous training results with Keras > 2.2.x

[aparrish]

Validation accuracy gets stuck around ~0.64 when training with Keras 2.3.x. This is training the phon2orth model with Keras 2.3.1 (128 hidden dims for encoder and decoder):

Train on 108072 samples, validate on 27019 samples
Epoch 1/50
108072/108072 [==============================] - 136s 1ms/step - loss: 0.4864 - accuracy: 0.5375 - val_loss: 0.4675 - val_accuracy: 0.5743
Epoch 2/50
108072/108072 [==============================] - 135s 1ms/step - loss: 0.2915 - accuracy: 0.7124 - val_loss: 0.4381 - val_accuracy: 0.6047
Epoch 3/50
108072/108072 [==============================] - 135s 1ms/step - loss: 0.2485 - accuracy: 0.7530 - val_loss: 0.4347 - val_accuracy: 0.6164
Epoch 4/50
108072/108072 [==============================] - 135s 1ms/step - loss: 0.2258 - accuracy: 0.7744 - val_loss: 0.4297 - val_accuracy: 0.6240
Epoch 5/50
108072/108072 [==============================] - 135s 1ms/step - loss: 0.2114 - accuracy: 0.7876 - val_loss: 0.4288 - val_accuracy: 0.6285
Epoch 6/50
108072/108072 [==============================] - 134s 1ms/step - loss: 0.2004 - accuracy: 0.7975 - val_loss: 0.4276 - val_accuracy: 0.6322

With 2.2.5, validation accuracy follows the expected path (slightly leading training accuracy, I think because validation doesn't use dropout):

108072/108072 [==============================] - 136s 1ms/step - loss: 1.4452 - acc: 0.5640 - val_loss: 0.8284 - val_acc: 0.7504
Epoch 2/50
108072/108072 [==============================] - 133s 1ms/step - loss: 0.7985 - acc: 0.7512 - val_loss: 0.6364 - val_acc: 0.8042
Epoch 3/50
108072/108072 [==============================] - 133s 1ms/step - loss: 0.6632 - acc: 0.7910 - val_loss: 0.5551 - val_acc: 0.8259
Epoch 4/50
108072/108072 [==============================] - 133s 1ms/step - loss: 0.5933 - acc: 0.8114 - val_loss: 0.5042 - val_acc: 0.8402
Epoch 5/50
108072/108072 [==============================] - 134s 1ms/step - loss: 0.5478 - acc: 0.8245 - val_loss: 0.4676 - val_acc: 0.8513

Not sure what's happening here—could have something to do with Keras 2.3's changing the "default recurrent activation to sigmoid (from hard_sigmoid) in all RNN layers."