first initial

Author: XinYu-Andy

README.md

@@ -0,0 +1,86 @@
+# Towards Efficient and Scale-Robust Ultra-High-Definition Image Demoireing
+
+**Towards Efficient and Scale-Robust Ultra-High-Definition Image Demoireing** (ECCV 2022)  
+Xin Yu, Peng Dai, Wenbo Li, Lan Ma, Jiajun Shen, Jia Li, [Xiaojuan Qi](https://scholar.google.com/citations?user=bGn0uacAAAAJ&hl=en).
+<br>[Paper(Coming soon)](---), [Project_page](https://xinyu-andy.github.io/uhdm-page/), [Dataset](https://drive.google.com/drive/folders/1DyA84UqM7zf3CeoEBNmTi_dJ649x2e7e?usp=sharing)
+
+![Example 1](./figures/result.png)
+
+## Introduction
+When photographing the contents displayed on the digital screen, an inevitable frequency aliasing between the camera’s 
+color filter array (CFA) and the screen’s LCD subpixel widely exists. The captured images are thus mixed with colorful 
+stripes, named moire patterns, which severely degrade images’ perceptual qualities. Although a plethora of dedicated 
+demoire methods have been proposed in the research community recently, yet is still far from achieving promising results 
+in the real-world scenes. The key limitation of these methods is that they all only conduct research on low-resolution or 
+synthetic images. However, with the rapid development of mobile devices, modern widely-used mobile phones typically allow 
+users to capture 4K resolution (i.e.,ultra-high-definition) images, and thus the effectiveness of these methods on this
+practical scenes is not promised. In this work, we explore moire pattern removal for ultra-high-definition images. 
+First, we propose the first ultra-high-definition demoireing dataset (UHDM), which contains 5,000 real-world 4K 
+resolution image pair, and conduct a benchmark study on the current state of the art. Then, we analyze limitations 
+of the state of the art and summarize the key issue of them, i.e., not scale-robust. To address their deficiencies, 
+we deliver a plug-and-play semantic-aligned scale-aware module which helps us to build a frustratingly simple baseline 
+model for tackling 4K moire images. Our framework is easy to implement and fast for inference, achieving state-of-the-art 
+results on four demoireing datasets while being much more lightweight. 
+We hope our investigation could inspire more future research in this more practical setting in image demoireing.
+![Example 1](./figures/cost.png)
+
+## Environments
+
+First you have to make sure that you have installed all dependencies. To do so, you can create an anaconda environment called `esdnet` using
+
+```
+conda env create -f environment.yaml
+conda activate esdnet
+```
+
+Our implementation has been tested on one NVIDIA 3090 GPU with cuda 11.2.
+
+## Dataset
+![Example 1](./figures/dataset.png)
+We provide the 4K dataset UHDM for you to evaluate a pretrained model or train a new model.
+To this end, you can download them [here](https://drive.google.com/drive/folders/1DyA84UqM7zf3CeoEBNmTi_dJ649x2e7e?usp=sharing), 
+or you can simply run the following command for automatic data downloading:
+```
+bash scripts/download_data.sh
+```
+Then the dataset will be available in the folder `uhdm_data/`.
+
+## Train
+To train a model from scratch, simply run:
+
+```
+python train.py --config CONFIG.yaml
+```
+where you replace `CONFIG.yaml` with the name of the configuration file you want to use.
+We have included configuration files for each dataset under the folder `config/`.
+
+For example, if you want to train our lightweight model ESDNet on UHDM dataset, run:
+```
+python train.py --config ./config/uhdm_config.yaml
+```
+   
+
+## Test
+To test a model, you can also simply run:
+
+```
+python test.py --config CONFIG.yaml
+```
+
+where you need to specify the value of `TEST_EPOCH` in the `CONFIG.yaml` to evaluate a model trained after specific epochs, 
+or you can also specify the value of `LOAD_PATH` to directly load a pre-trained checkpoint.
+
+We provide pre-trained models [here](https://drive.google.com/drive/folders/12buOOBKDBdQ65gM8U1rRNpSHppQ_u9Lr?usp=sharing). 
+To download the checkpoints, you can also simply run:
+
+```
+bash scripts/download_model.sh
+```
+
+Then the checkpoints will be included in the folder `pretrain_model/`. 
+
+
+## Contact
+If you have any questions, you can email me ([email protected]).
+
+

config/__pycache__/config.cpython-38.pyc

@@ -0,0 +1,44 @@
+GENERAL:
+  GPU_ID: 3
+  SEED: 123
+  WORKER: 8
+  SAVE_PREFIX: './out_dir/aim'
+  EXP_NAME: 'exp_light'
+
+DATA:
+  DATA_TYPE: AIM # Please specify the type of the dataset (select from AIM/UHDM/FHDMi/TIP)
+  TRAIN_DATASET: # The training data path, e.g., ./uhdm_data/Train
+  TEST_DATASET: # The test data path, e.g., ./uhdm_data/Test
+
+MODEL:
+  EN_FEATURE_NUM: 48 # The initial channel number of dense blocks of encoder
+  EN_INTER_NUM: 32 # The growth rate (intermediate channel number) of dense blocks of encoder
+  DE_FEATURE_NUM: 64 # The initial channel number of dense blocks of decoder
+  DE_INTER_NUM: 32 # The growth rate (intermediate channel number) of dense blocks of decoder
+  SAM_NUMBER: 1 # The number of SAM for each encoder or decoder level; set 1 for our ESDNet, and 2 for ESDNet-L
+
+TRAIN:
+  BATCH_SIZE: 2
+  LOADER: crop # The loading way for training data, e.g., crop, resize, default; see ./dataset/load_data.py
+  CROP_SIZE: 512 # Set the crop size if LOADER==crop
+  RESIZE_SIZE: 384 # Set the resizing size if LOADER==crop
+  SAVE_ITER: 500 # Save training images/results at each SAVE_ITER*n iter
+  LOAD_EPOCH: False # If specify it, loading the corresponding model for resuming training
+  LAM: 1 # The loss weight for L1 loss
+  LAM_P: 1 # The loss weight for perceptual loss
+
+TEST:
+  TEST_EPOCH: 200 # Input 'auto' for loading the latest model
+  SAVE_IMG: False # The file type (e.g., jpg, png) for saving the output image; set False to avoid saving
+  LOAD_PATH: False # If specify a load path for a checkpoint, TEST_EPOCH will be deprecated
+  EVALUATION_METRIC: True # If True, calculate metrics
+  EVALUATION_TIME: False # If True, calculate processing time per image; EVALUATION_METRIC will be deprecated for accurate statistics
+  EVALUATION_COST: False #If True, calculate MACs and Parameters number
+
+SOLVER:
+  EPOCHS: 300 # The total training epochs
+  T_0: 50 # The total epochs for the first learning cycle (learning rate warms up then)
+  T_MULT: 1 # The learning cycle would be (T_0, T_0*T_MULT, T_0*T_MULT^2, T_0*T_MULT^3, ...)
+  ETA_MIN: 0.000001 # Initial learning rate in each learning cycle
+  BASE_LR: 0.0002 # Learning rate in the end of each learning cycle
+

config/aim_config.yaml

@@ -0,0 +1,43 @@
+GENERAL:
+  GPU_ID: 3
+  SEED: 123
+  WORKER: 8
+  SAVE_PREFIX: './out_dir/aim'
+  EXP_NAME: 'exp_large'
+
+DATA:
+  DATA_TYPE: AIM # Please specify the type of the dataset (select from AIM/UHDM/FHDMi/TIP)
+  TRAIN_DATASET: # The training data path, e.g., ./uhdm_data/Train
+  TEST_DATASET: # The test data path, e.g., ./uhdm_data/Test
+
+MODEL:
+  EN_FEATURE_NUM: 48 # The initial channel number of dense blocks of encoder
+  EN_INTER_NUM: 32 # The growth rate (intermediate channel number) of dense blocks of encoder
+  DE_FEATURE_NUM: 64 # The initial channel number of dense blocks of decoder
+  DE_INTER_NUM: 32 # The growth rate (intermediate channel number) of dense blocks of decoder
+  SAM_NUMBER: 2 # The number of SAM for each encoder or decoder level; set 1 for our ESDNet, and 2 for ESDNet-L
+
+TRAIN:
+  BATCH_SIZE: 2
+  LOADER: crop # The loading way for training data, e.g., crop, resize, default; see ./dataset/load_data.py
+  CROP_SIZE: 512 # Set the crop size if LOADER==crop
+  RESIZE_SIZE: 384 # Set the resizing size if LOADER==crop
+  SAVE_ITER: 500 # Save training images/results at each SAVE_ITER*n iter
+  LOAD_EPOCH: False # If specify it, loading the corresponding model for resuming training
+  LAM: 1 # The loss weight for L1 loss
+  LAM_P: 1 # The loss weight for perceptual loss
+
+TEST:
+  TEST_EPOCH: 200 # Input 'auto' for loading the latest model
+  SAVE_IMG: False # The file type (e.g., jpg, png) for saving the output image; set False to avoid saving
+  LOAD_PATH: False # If specify a load path for a checkpoint, TEST_EPOCH will be deprecated
+  EVALUATION_METRIC: True # If True, calculate metrics
+  EVALUATION_TIME: False # If True, calculate processing time per image; EVALUATION_METRIC will be deprecated for accurate statistics
+  EVALUATION_COST: False #If True, calculate MACs and Parameters number
+
+SOLVER:
+  EPOCHS: 300 # The total training epochs
+  T_0: 50 # The total epochs for the first learning cycle (learning rate warms up then)
+  T_MULT: 1 # The learning cycle would be (T_0, T_0*T_MULT, T_0*T_MULT^2, T_0*T_MULT^3, ...)
+  ETA_MIN: 0.000001 # Initial learning rate in each learning cycle
+  BASE_LR: 0.0002 # Learning rate in the end of each learning cycle

config/aim_large_config.yaml

@@ -0,0 +1,19 @@
+import argparse
+import yaml
+import os
+
+def get_parser():
+    parser = argparse.ArgumentParser(description='IMAGE_DEMOIREING')
+    parser.add_argument('--config', type=str, default='config/uhdm_config.yaml', help='path to config file')
+    args_cfg = parser.parse_args()
+    assert args_cfg.config is not None
+    with open(args_cfg.config, 'r') as f:
+        config = yaml.load(f, Loader=yaml.FullLoader)
+    for key in config:
+        for k, v in config[key].items():
+            setattr(args_cfg, k, v)
+
+    return args_cfg
+
+
+args = get_parser()

config/config.py

@@ -0,0 +1,44 @@
+GENERAL:
+  GPU_ID: 3
+  SEED: 123
+  WORKER: 8
+  SAVE_PREFIX: './out_dir/fhdmi'
+  EXP_NAME: 'exp_light'
+
+DATA:
+  DATA_TYPE: FHDMi # Please specify the type of the dataset (select from AIM/UHDM/FHDMi/TIP)
+  TRAIN_DATASET: # The training data path, e.g., ./uhdm_data/Train
+  TEST_DATASET: # The test data path, e.g., ./uhdm_data/Test
+
+MODEL:
+  EN_FEATURE_NUM: 48 # The initial channel number of dense blocks of encoder
+  EN_INTER_NUM: 32 # The growth rate (intermediate channel number) of dense blocks of encoder
+  DE_FEATURE_NUM: 64 # The initial channel number of dense blocks of decoder
+  DE_INTER_NUM: 32 # The growth rate (intermediate channel number) of dense blocks of decoder
+  SAM_NUMBER: 1 # The number of SAM for each encoder or decoder level; set 1 for our ESDNet, and 2 for ESDNet-L
+
+TRAIN:
+  BATCH_SIZE: 2
+  LOADER: crop # The loading way for training data, e.g., crop, resize, default; see ./dataset/load_data.py
+  CROP_SIZE: 512 # Set the crop size if LOADER==crop
+  RESIZE_SIZE: 384 # Set the resizing size if LOADER==crop
+  SAVE_ITER: 500 # Save training images/results at each SAVE_ITER*n iter
+  LOAD_EPOCH: False # If specify it, loading the corresponding model for resuming training
+  LAM: 1 # The loss weight for L1 loss
+  LAM_P: 1 # The loss weight for perceptual loss
+
+TEST:
+  TEST_EPOCH: 150 # Input 'auto' for loading the latest model
+  SAVE_IMG: False # The file type (e.g., jpg, png) for saving the output image; set False to avoid saving
+  LOAD_PATH: False # If specify a load path for a checkpoint, TEST_EPOCH will be deprecated
+  EVALUATION_METRIC: True # If True, calculate metrics
+  EVALUATION_TIME: False # If True, calculate processing time per image; EVALUATION_METRIC will be deprecated for accurate statistics
+  EVALUATION_COST: False #If True, calculate MACs and Parameters number
+
+SOLVER:
+  EPOCHS: 150 # The total training epochs
+  T_0: 50 # The total epochs for the first learning cycle (learning rate warms up then)
+  T_MULT: 1 # The learning cycle would be (T_0, T_0*T_MULT, T_0*T_MULT^2, T_0*T_MULT^3, ...)
+  ETA_MIN: 0.000001 # Initial learning rate in each learning cycle
+  BASE_LR: 0.0002 # Learning rate in the end of each learning cycle
+

config/fhdmi_config.yaml

@@ -0,0 +1,44 @@
+GENERAL:
+  GPU_ID: 3
+  SEED: 123
+  WORKER: 8
+  SAVE_PREFIX: './out_dir/fhdmi'
+  EXP_NAME: 'exp_large'
+
+DATA:
+  DATA_TYPE: FHDMi # Please specify the type of the dataset (select from AIM/UHDM/FHDMi/TIP)
+  TRAIN_DATASET: # The training data path, e.g., ./uhdm_data/Train
+  TEST_DATASET: # The test data path, e.g., ./uhdm_data/Test
+
+MODEL:
+  EN_FEATURE_NUM: 48 # The initial channel number of dense blocks of encoder
+  EN_INTER_NUM: 32 # The growth rate (intermediate channel number) of dense blocks of encoder
+  DE_FEATURE_NUM: 64 # The initial channel number of dense blocks of decoder
+  DE_INTER_NUM: 32 # The growth rate (intermediate channel number) of dense blocks of decoder
+  SAM_NUMBER: 2 # The number of SAM for each encoder or decoder level; set 1 for our ESDNet, and 2 for ESDNet-L
+
+TRAIN:
+  BATCH_SIZE: 2
+  LOADER: crop # The loading way for training data, e.g., crop, resize, default; see ./dataset/load_data.py
+  CROP_SIZE: 512 # Set the crop size if LOADER==crop
+  RESIZE_SIZE: 384 # Set the resizing size if LOADER==crop
+  SAVE_ITER: 500 # Save training images/results at each SAVE_ITER*n iter
+  LOAD_EPOCH: False # If specify it, loading the corresponding model for resuming training
+  LAM: 1 # The loss weight for L1 loss
+  LAM_P: 1 # The loss weight for perceptual loss
+
+TEST:
+  TEST_EPOCH: 150 # Input 'auto' for loading the latest model
+  SAVE_IMG: False # The file type (e.g., jpg, png) for saving the output image; set False to avoid saving
+  LOAD_PATH: False # If specify a load path for a checkpoint, TEST_EPOCH will be deprecated
+  EVALUATION_METRIC: True # If True, calculate metrics
+  EVALUATION_TIME: False # If True, calculate processing time per image; EVALUATION_METRIC will be deprecated for accurate statistics
+  EVALUATION_COST: False #If True, calculate MACs and Parameters number
+
+SOLVER:
+  EPOCHS: 150 # The total training epochs
+  T_0: 50 # The total epochs for the first learning cycle (learning rate warms up then)
+  T_MULT: 1 # The learning cycle would be (T_0, T_0*T_MULT, T_0*T_MULT^2, T_0*T_MULT^3, ...)
+  ETA_MIN: 0.000001 # Initial learning rate in each learning cycle
+  BASE_LR: 0.0002 # Learning rate in the end of each learning cycle
+

config/fhdmi_large_config.yaml

@@ -0,0 +1,44 @@
+GENERAL:
+  GPU_ID: 3
+  SEED: 123
+  WORKER: 8
+  SAVE_PREFIX: './out_dir/tip'
+  EXP_NAME: 'exp_light'
+
+DATA:
+  DATA_TYPE: TIP # Please specify the type of the dataset (select from AIM/UHDM/FHDMi/TIP)
+  TRAIN_DATASET: # The training data path, e.g., ./uhdm_data/Train
+  TEST_DATASET: # The test data path, e.g., ./uhdm_data/Test
+
+MODEL:
+  EN_FEATURE_NUM: 48 # The initial channel number of dense blocks of encoder
+  EN_INTER_NUM: 32 # The growth rate (intermediate channel number) of dense blocks of encoder
+  DE_FEATURE_NUM: 64 # The initial channel number of dense blocks of decoder
+  DE_INTER_NUM: 32 # The growth rate (intermediate channel number) of dense blocks of decoder
+  SAM_NUMBER: 1 # The number of SAM for each encoder or decoder level; set 1 for our ESDNet, and 2 for ESDNet-L
+
+TRAIN:
+  BATCH_SIZE: 2
+  #LOADER: crop # The loading way for training data, e.g., crop, resize, default; see ./dataset/load_data.py
+  #CROP_SIZE: 768 # Set the crop size if LOADER==crop
+  #RESIZE_SIZE: 384 # Set the resizing size if LOADER==crop
+  SAVE_ITER: 500 # Save training images/results at each SAVE_ITER*n iter
+  LOAD_EPOCH: False # If specify it, loading the corresponding model for resuming training
+  LAM: 1 # The loss weight for L1 loss
+  LAM_P: 1 # The loss weight for perceptual loss
+
+TEST:
+  TEST_EPOCH: 70 # Input 'auto' for loading the latest model
+  SAVE_IMG: False # The file type (e.g., jpg, png) for saving the output image; set False to avoid saving
+  LOAD_PATH: False # If specify a load path for a checkpoint, TEST_EPOCH will be deprecated
+  EVALUATION_METRIC: True # If True, calculate metrics
+  EVALUATION_TIME: False # If True, calculate processing time per image; EVALUATION_METRIC will be deprecated for accurate statistics
+  EVALUATION_COST: False #If True, calculate MACs and Parameters number
+
+SOLVER:
+  EPOCHS: 70 # The total training epochs
+  T_0: 10 # The total epochs for the first learning cycle (learning rate warms up then)
+  T_MULT: 1 # The learning cycle would be (T_0, T_0*T_MULT, T_0*T_MULT^2, T_0*T_MULT^3, ...)
+  ETA_MIN: 0.000001 # Initial learning rate in each learning cycle
+  BASE_LR: 0.0002 # Learning rate in the end of each learning cycle
+

config/tip_config.yaml

@@ -0,0 +1,44 @@
+GENERAL:
+  GPU_ID: 3
+  SEED: 123
+  WORKER: 8
+  SAVE_PREFIX: './out_dir/tip'
+  EXP_NAME: 'exp_large'
+
+DATA:
+  DATA_TYPE: TIP # Please specify the type of the dataset (select from AIM/UHDM/FHDMi/TIP)
+  TRAIN_DATASET: # The training data path, e.g., ./uhdm_data/Train
+  TEST_DATASET: # The test data path, e.g., ./uhdm_data/Test
+
+MODEL:
+  EN_FEATURE_NUM: 48 # The initial channel number of dense blocks of encoder
+  EN_INTER_NUM: 32 # The growth rate (intermediate channel number) of dense blocks of encoder
+  DE_FEATURE_NUM: 64 # The initial channel number of dense blocks of decoder
+  DE_INTER_NUM: 32 # The growth rate (intermediate channel number) of dense blocks of decoder
+  SAM_NUMBER: 2 # The number of SAM for each encoder or decoder level; set 1 for our ESDNet, and 2 for ESDNet-L
+
+TRAIN:
+  BATCH_SIZE: 2
+  #LOADER: crop # The loading way for training data, e.g., crop, resize, default; see ./dataset/load_data.py
+  #CROP_SIZE: 768 # Set the crop size if LOADER==crop
+  #RESIZE_SIZE: 384 # Set the resizing size if LOADER==crop
+  SAVE_ITER: 500 # Save training images/results at each SAVE_ITER*n iter
+  LOAD_EPOCH: False # If specify it, loading the corresponding model for resuming training
+  LAM: 1 # The loss weight for L1 loss
+  LAM_P: 1 # The loss weight for perceptual loss
+
+TEST:
+  TEST_EPOCH: 70 # Input 'auto' for loading the latest model
+  SAVE_IMG: False # The file type (e.g., jpg, png) for saving the output image; set False to avoid saving
+  LOAD_PATH: False # If specify a load path for a checkpoint, TEST_EPOCH will be deprecated
+  EVALUATION_METRIC: True # If True, calculate metrics
+  EVALUATION_TIME: False # If True, calculate processing time per image; EVALUATION_METRIC will be deprecated for accurate statistics
+  EVALUATION_COST: False #If True, calculate MACs and Parameters number
+
+SOLVER:
+  EPOCHS: 70 # The total training epochs
+  T_0: 10 # The total epochs for the first learning cycle (learning rate warms up then)
+  T_MULT: 1 # The learning cycle would be (T_0, T_0*T_MULT, T_0*T_MULT^2, T_0*T_MULT^3, ...)
+  ETA_MIN: 0.000001 # Initial learning rate in each learning cycle
+  BASE_LR: 0.0002 # Learning rate in the end of each learning cycle
+