Model training performed well, but good test results could not be obtained

[lfxuan]

Hello! The training accuracy is very high, and the image reasoning directly using the verification set or training set is still very poor. There was no such problem before, but it has always appeared now. Look forward to your reply!

[fg-mindee]

Hi @lfxuan 👋

Thanks for reporting this! We would need a bit more information to help you on this 😅

• are you using PyTorch or TensorFlow?
• for which task did you train your model? did you use our reference scripts?
• what did you use for training and validation sets?

Cheers ✌️

[lfxuan]

Hello! @fg-mindee

• I used the Tensorflow.
• I have used your recognition script in the doctr/references/recognition/train_tensorflow.py and use the test script in the doctr/doctr/models/recognition/zoo.py(Some modifications have been made as needed).
• I have used my train data and valid data to train the model 'master'.
• I have uploaded my training log and test results.
• Application Description:

1. It is mainly used for verification code identification
2. vocab——string. digits + string. ascii_ letters（pretrained=False）
3. The results of the previous training process are good, and the corresponding test results will also be very good. Recently, the phenomenon that the training accuracy is very high, but the test is ineffective (the code has been updated yesterday).
4. There's two questions: (1)why do you need crop when testing with the identified tes
   t script (zoo. py)?(2)Why is 'cleanup call' output in the process of training the 'master' model?relative_files.zip

[fg-mindee]

Thanks for the specifics!

Regarding MASTER, we're already debugging this: #802
I'd suggest going for a CRNN model 👍

About your 3rd point, do you mean that the training performances have decreased due to a recent codebase change?

Regarding your last point:

(1)why do you need crop when testing with the identified test script (zoo. py)?

I don't understand your question, you wrote this custom script so are you asking about the existing test script from this repo? If so, it's because we use an OCR dataset, so we only pass word crops to the recognition model.

(2)Why is 'cleanup call' output in the process of training the 'master' model?

I don't have this on my end when using the model. My suspect would have been a clear_session call (cf. https://www.tensorflow.org/api_docs/python/tf/keras/backend/clear_session) but docTR only calls this in its unittests. What command did you run to get those?

[lfxuan]

Thanks your reply!
(1)What I want to say is that my code base is up-to-date, but I still encounter the problem of high training accuracy (validation loss: 0.0176707 (exact: 98.30% | partial: 98.92%) and test accuracy of 0.（used model: sar_resnet31 / master ）
(2)I use the commandnohup python references/recognition/train_tensorflow.py master --train_path ./train_data/rec_match/captcha/train_n --val_path ./train_data/rec_match/captcha/val_n/ --epoch 30 --lr 0.0001 --vocab 'la' --batch_size 80 --input_size 40 > capt.log 2>&1 &,and the cleanup callsometimes appears, sometimes doesn't.

[fg-mindee]

1. I understand that this is the situation right now. But you seemed to suggest that this wasn't the case before, can you confirm this?
2. Mmmh that's odd, I'll try to see if I can reproduce this but I haven't seen that so far!

[lfxuan]

1. After many attempts, it is found that as long as the input_size of the training process is modified will lead to high training accuracy and poor test accuracy; Keep default input_ Size, there is little difference between the training and test results.
2. The appearance of cleanup seems to be triggered by the image suffix PNG. It will not appear when I resave the image in JPG format.The image causing cleanup has been given.
   cleanup_images.zip