the training loss is quiet high(0.7) when using U-net and U-net_ae

[yunclouding]

2019-01-04 12:57:31,217 - InfoLog - INFO - Epoch :[21/22]
2019-01-04 12:57:32,304 - InfoLog - INFO - Train: [0/1323] | Time: 145m 34s | loss: 0.2563 | oa:0.9428 | ma: 0.5084 | fa: 0.8927 | miou: 0.3969
2019-01-04 12:58:32,274 - InfoLog - INFO - Train: [200/1323] | Time: 146m 34s | loss: 0.7213 | oa:0.7850 | ma: 0.3646 | fa: 0.6490 | miou: 0.2838
2019-01-04 12:59:32,480 - InfoLog - INFO - Train: [400/1323] | Time: 147m 34s | loss: 0.7265 | oa:0.7848 | ma: 0.3623 | fa: 0.6477 | miou: 0.2872
2019-01-04 13:00:32,482 - InfoLog - INFO - Train: [600/1323] | Time: 148m 34s | loss: 0.7195 | oa:0.7871 | ma: 0.3608 | fa: 0.6521 | miou: 0.2858
2019-01-04 13:01:32,458 - InfoLog - INFO - Train: [800/1323] | Time: 149m 34s | loss: 0.7247 | oa:0.7862 | ma: 0.3592 | fa: 0.6503 | miou: 0.2841
2019-01-04 13:02:32,531 - InfoLog - INFO - Train: [1000/1323] | Time: 150m 35s | loss: 0.7310 | oa:0.7843 | ma: 0.3584 | fa: 0.6475 | miou: 0.2836
2019-01-04 13:03:32,648 - InfoLog - INFO - Train: [1200/1323] | Time: 151m 35s | loss: 0.7314 | oa:0.7842 | ma: 0.3571 | fa: 0.6474 | miou: 0.2826
2019-01-04 13:04:09,282 - InfoLog - INFO - Train: [1322/1323] | Time: 152m 11s | loss: 0.7309 | oa:0.7841 | ma: 0.3576 | fa: 0.6468 | miou: 0.2829
2019-01-04 13:04:09,333 - InfoLog - INFO - val starts...
2019-01-04 13:04:26,716 - InfoLog - INFO - Val: [181/182] | Time: 152m 29s | loss: 0.5527 | oa:0.8382 | ma: 0.3950 | fa: 0.7325 | miou: 0.2990
2019-01-04 13:04:26,717 - InfoLog - INFO - ------
2019-01-04 13:04:27,137 - InfoLog - INFO - Saving model
2019-01-04 13:04:27,137 - InfoLog - INFO - per class iou: {0: 0.8689010639169004, 1: 0.578765685997748, 2: 0.20360704910793018, 3: 0.191739763291252, 4: 0.023346662627081405, 5: 0.002138736852459477, 6: 0.3903862621014857, 7: 0.5442177579282417, 8: 0.4707609219879637, 9: 0.017006589661997945, 10: 0.0004230102815512101, 11: 0.11381994406156236, 12: 0.3213647095875155, 13: 0.2210694392970857, 14: 0.4312723662164119, 15: 0.6251583876268879, 16: 0.021866156063586918, 17: 0.2671511962509249, 18: 0.16121951454400624, 19: 0.43448701641267573, 20: 0.3903983331489073}

[yunclouding]

and this is the config
{
"model_params": {
"unet": {
"filter_scale": 1
},
"unet_ae": {
"backend": "resnet101",
"pretrained": "imagenet"
},
"dlv3plus": {
"backend": "mobilenet_v2",
"os": 16,
"pretrained": "imagenet"
},
"pspnet": {
"backend": "resnet101",
"pool_scales": [
1,
2,
3,
6
],
"pretrained": "imagenet"
}
},
"loss_params": {
"ce": {
"weight": null,
"ignore_index": 255
},
"dice": {
"smooth": 1
},
"focal": {
"weight": null,
"gamma": 2,
"alpha": 0.5
},
"lovasz": {
"multiclasses": true
}
},
"model": "unet",
"loss": "ce",
"lr": 0.01,
"max_iters": 30000,
"epochs": null,
"batch_size": 8,
"distributed": true,
"gpuid": "0",
"num_workers": 8,
"ckpt_dir": "./checkpoints/voc2012-unet-0-ce",
"resume": false,
"train_image_size": 256,
"val_image_size": [
256,
256
],
"in_channels": 3,
"data_type": "voc2012",
"image_root": "/Downloads/datasets/voc/VOCdevkit/VOC2012",
"train_list": "/Downloads/datasets/voc/VOCdevkit/VOC2012/list/train_aug.txt",
"val_list": "/Downloads/datasets/voc/VOCdevkit/VOC2012/list/val.txt",
"label_file": "/Documents/Pytorch-ImageSegmentation-master/datasets/voc/labels"

[silkylove]

Sorry, I did not train Unet on voc2012 because Unet was usually used on medical dataset which I may train in the future. Instead, I only trained Unet_ae on voc2012.

[yunclouding]

@silkylove thanks for your work
in unet_ae i have the same question, the loss is quiet large, after several training, the model cannot arrive your checkpoint`s performence.
the best val performence is Val: [362/363] | Time: 482m 22s | loss: 0.2995 | oa:0.9015 | ma: 0.6935 | fa: 0.8311 | miou: 0.5661

and the loss is quiet weird, the loss at start is low, after some iters, the loss is larger.
sometimes the loss is always stay around 0.7, it didn`t decline.
INFO:InfoLog:Epoch :[32/34]
2019-01-05 21:55:53,122 - InfoLog - INFO - Train: [0/2646] | Time: 735m 29s | loss: 0.2059 | oa:0.9578 | ma: 0.7589 | fa: 0.9211 | miou: 0.6119
INFO:InfoLog:Train: [0/2646] | Time: 735m 29s | loss: 0.2059 | oa:0.9578 | ma: 0.7589 | fa: 0.9211 | miou: 0.6119
2019-01-05 21:57:33,279 - InfoLog - INFO - Train: [200/2646] | Time: 737m 9s | loss: 0.7595 | oa:0.7677 | ma: 0.3645 | fa: 0.6231 | miou: 0.2810
INFO:InfoLog:Train: [200/2646] | Time: 737m 9s | loss: 0.7595 | oa:0.7677 | ma: 0.3645 | fa: 0.6231 | miou: 0.2810
2019-01-05 21:59:13,380 - InfoLog - INFO - Train: [400/2646] | Time: 738m 49s | loss: 0.7473 | oa:0.7746 | ma: 0.3574 | fa: 0.6336 | miou: 0.2781
INFO:InfoLog:Train: [400/2646] | Time: 738m 49s | loss: 0.7473 | oa:0.7746 | ma: 0.3574 | fa: 0.6336 | miou: 0.2781
2019-01-05 22:00:55,379 - InfoLog - INFO - Train: [600/2646] | Time: 740m 31s | loss: 0.7412 | oa:0.7763 | ma: 0.3616 | fa: 0.6367 | miou: 0.2828

the config with adam optimizer, is there something wrong?
self.optimizer = optim.Adam(self.model.parameters(), lr=self.lr, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False)
{
"model_params": {
"unet": {
"filter_scale": 1
},
"unet_ae": {
"backend": "resnet101",
"pretrained": "imagenet"
},
"dlv3plus": {
"backend": "mobilenet_v2",
"os": 16,
"pretrained": "imagenet"
},
"pspnet": {
"backend": "resnet101",
"pool_scales": [
1,
2,
3,
6
],
"pretrained": "imagenet"
}
},
"loss_params": {
"ce": {
"weight": null,
"ignore_index": 255
},
"dice": {
"smooth": 1
},
"focal": {
"weight": null,
"gamma": 2,
"alpha": 0.5
},
"lovasz": {
"multiclasses": true
}
},
"model": "unet_ae",
"loss": "ce",
"lr": 0.001,
"max_iters": 90000,
"epochs": null,
"batch_size": 4,
"distributed": true,
"gpuid": "0",
"num_workers": 8,
"ckpt_dir": "./checkpoints/voc2012-unet_ae-adam-ce",
"resume": false,
"train_image_size": 256,
"val_image_size": [
256,
256
],
"in_channels": 3,
"data_type": "voc2012",
"image_root": "/home/Downloads/datasets/voc/VOCdevkit/VOC2012",
"train_list": "/home/Downloads/datasets/voc/VOCdevkit/VOC2012/list/train_aug.txt",
"val_list": "/home/Downloads/datasets/voc/VOCdevkit/VOC2012/list/val.txt",
"label_file": "/home/Documents/Pytorch-ImageSegmentation-master/datasets/voc/labels"
}
thanks for your reply

[silkylove]

@yunclouding Try to follow the same config which was used in my experiment. For your changed config for Unet_ae, I think the problem might come from adam and batch_size, as you know, larger batch_size (16 in my experi) usually gives you better results and stable convergence but adam sometimes may not be stable.

So I suggest you to use the same lr 0.01 and SGD with 0.9 momentum optimizer first. If you have more gpus, you can use larger batch_size.

[yunclouding]

@yunclouding Try to follow the same config which was used in my experiment. For your changed config for Unet_ae, I think the problem might come from adam and batch_size, as you know, larger batch_size (16 in my experi) usually gives you better results and stable convergence but adam sometimes may not be stable.

So I suggest you to use the same lr 0.01 and SGD with 0.9 momentum optimizer first. If you have more gpus, you can use larger batch_size.

thanks a lot, i will try

[silkylove]

@yunclouding Thank you for trying this code!

[yunclouding]

@yunclouding Thank you for trying this code!

your code is pretty beautiful. thanks for your share
and i am a beginner for pytorch, your code is easy to read