Loss didn't decrease in pwcnet_train_sm-6-2-multisteps-chairsthingsmix #30
Description
some output infos:
2019-10-31 21:27:13 Iter 49000 [Train]: loss=184.37, epe=15.17, lr=0.000100, samples/sec=24.4, sec/step=0.655, eta=8 days, 17:27:22
2019-10-31 21:27:21 Iter 49000 [Val]: loss=141.03, epe=11.58
Saving model...
INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-49000 is not in all_model_checkpoint_paths. Manually adding it.
... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-49000
2019-10-31 21:39:36 Iter 50000 [Train]: loss=184.73, epe=15.20, lr=0.000100, samples/sec=27.7, sec/step=0.578, eta=7 days, 16:41:55
2019-10-31 21:39:44 Iter 50000 [Val]: loss=140.98, epe=11.57
Saving model...
INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-50000 is not in all_model_checkpoint_paths. Manually adding it.
... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-50000
2019-10-31 21:50:47 Iter 51000 [Train]: loss=184.02, epe=15.14, lr=0.000100, samples/sec=28.3, sec/step=0.566, eta=7 days, 12:37:08
2019-10-31 21:50:56 Iter 51000 [Val]: loss=140.30, epe=11.52
Saving model...
INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-51000 is not in all_model_checkpoint_paths. Manually adding it.
... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-51000
2019-10-31 22:03:22 Iter 52000 [Train]: loss=184.69, epe=15.20, lr=0.000100, samples/sec=26.0, sec/step=0.616, eta=8 days, 4:35:21
2019-10-31 22:03:33 Iter 52000 [Val]: loss=141.20, epe=11.60
Saving model...
INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-52000 is not in all_model_checkpoint_paths. Manually adding it.
... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-52000
2019-10-31 22:16:15 Iter 53000 [Train]: loss=184.40, epe=15.17, lr=0.000100, samples/sec=25.4, sec/step=0.629, eta=8 days, 8:18:36
2019-10-31 22:16:25 Iter 53000 [Val]: loss=140.56, epe=11.53
Saving model...
INFO:tensorflow:./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-53000 is not in all_model_checkpoint_paths. Manually adding it.
... model saved in ./pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-53000