Try super-resolution for model training/inference

[plbenveniste]

In this issue, I investigated using super-resolution to improve MS lesion segmentation.
The idea came from this article: https://ieeexplore.ieee.org/abstract/document/10981088
The super-resolution article is here: https://link.springer.com/chapter/10.1007/978-3-031-44689-4_12
The code for super-resolution is here: https://gitlab.com/iacl/smore

[plbenveniste]

Here is the result of the super-resolution on image sub-edm113_ses-M0_PSIR.nii.gz from the canproco dataset.

Code ran:

run-smore --in-fpath image_test/sub-cal056_ses-M0_STIR.nii.gz --out-dir output_test/

COLOR CODE:
blue (0) --> pink (1)

screen.mov

[plbenveniste]

Here is the result on the lesion label.

Code run

smore-test --in-fpath sub-edm113_ses-M0_PSIR_lesion-manual.nii.gz --out-fpath ../inference_already_trained/output_lesion.nii.gz --weight-dir ../output_test/sub-edm113_ses-M0_PSIR/weights/

I then applied thresholding at 0.1 and normalized the lesion segmentation using a min-max method.

Untitled.mov

Code used for thresholding and normalisation:

"""
This script is used to look at the output distribution of the super-resolved lesion segmentation.
"""
lesion_path="/home/plbenveniste/net/ms-lesion-agnostic/super_resolution/inference_already_trained/output_lesion.nii.gz"
import nibabel as nib
import matplotlib.pyplot as plt
import numpy as np

lesion = nib.load(lesion_path)
lesion_data = lesion.get_fdata()

# Remove values below 0.01
lesion_data[lesion_data < 0.1] = 0

# We normalize the data with min-max normalization
lesion_data = (lesion_data - np.min(lesion_data)) / (np.max(lesion_data) - np.min(lesion_data))

# We save the lesion data as a new NIfTI file
output_lesion_path = '/home/plbenveniste/net/ms-lesion-agnostic/super_resolution/inference_already_trained/output_lesion_normalized.nii.gz'
new_lesion = nib.Nifti1Image(lesion_data, lesion.affine, lesion.header)
nib.save(new_lesion, output_lesion_path)

Observations:

• there are issues with the modifications applied to the lesion segmentation!

Possible solution:

• instead of applying the super-resolution model to the lesion segmentation, we can apply sct_register_multimodal -identity 1 to the mask to transform it.

[plbenveniste]

Here is the output with the resampled version:

resampled.mov

Here is what it looks like with a 0.5 threshold after resampling:

resampled_tresh.mov

It seems that the results are better but still not ideal as they are heavily dependent on the initial annotation. It creates this sort of oscilating pattern when looking at the soft segmentation with segmentations are close to 1 around each slice in the initial space. Binarization is hences required. Use of "soft" segmentation is pretty limited.

[plbenveniste]

I tried applying the SCT MS model on the raw image and the super-resolved image:

(Blue: segmentation on raw image; Red: segmentation done on super-resolved image; Green: manual segmentation)

Observations:
→ It seems that the prediction done on the super-resolved image fits better the reality
→ It seems that the prediction done on the raw image fits better the manual segmentation

[plbenveniste]

The super-resolution algorithm should not be run on isotropic images.
In the best case, it does the following: AssertionError: Worst resolution found 1.0000001192092896 matches one of the better resolutions 1.0 or 1.0000001192092896; image is "isotropic" and cannot be run through SMORE.
In the worst case, it does the following:

Traceback (most recent call last):
  File "/home/plbenveniste/miniconda3/envs/venv_super_res/bin/smore-train", line 33, in <module>
    sys.exit(load_entry_point('smore', 'console_scripts', 'smore-train')())
  File "/data/users/pierrelouis/ms-lesion-agnostic/super_resolution/smore/smore/train.py", line 113, in main
    blur_kernel = parse_kernel(args.blur_kernel_fpath, args.blur_kernel, blur_fwhm)
  File "/data/users/pierrelouis/ms-lesion-agnostic/super_resolution/smore/smore/utils/blur_kernel_ops.py", line 16, in parse_kernel
    blur_kernel = blur_kernel.squeeze()[None, None, :, None]
IndexError: too many indices for array: array is 0-dimensional, but 1 were indexed

This is due to images wich are almost isotropic. Example of problematic resolution: x,y,z = 1.0, 1.0000001192092896, 1.0000001192092896

[plbenveniste]

Currently performing super-resolution conversion using (on kronos):

CUDA_VISIBLE_DEVICES=0 python run_super_res_nnunet_data.py --input-dir ~/net/ms-lesion-agnostic/nnunet_experiments/nnUNet_raw/Dataset902_msLesionAgnostic/imagesTs/ --output-dir ~/net/ms-lesion-agnostic/super_resolution/nnUNet_raw/Dataset902_msLesionAgnostic/imagesTs/ --min-index 0 --max-index 100
CUDA_VISIBLE_DEVICES=1 python run_super_res_nnunet_data.py --input-dir ~/net/ms-lesion-agnostic/nnunet_experiments/nnUNet_raw/Dataset902_msLesionAgnostic/imagesTs/ --output-dir ~/net/ms-lesion-agnostic/super_resolution/nnUNet_raw/Dataset902_msLesionAgnostic/imagesTs/ --min-index 101 --max-index 202
CUDA_VISIBLE_DEVICES=2 python run_super_res_nnunet_data.py --input-dir ~/net/ms-lesion-agnostic/nnunet_experiments/nnUNet_raw/Dataset902_msLesionAgnostic/imagesTs/ --output-dir ~/net/ms-lesion-agnostic/super_resolution/nnUNet_raw/Dataset902_msLesionAgnostic/imagesTs/ --min-index 203 --max-index 304
CUDA_VISIBLE_DEVICES=3 python run_super_res_nnunet_data.py --input-dir ~/net/ms-lesion-agnostic/nnunet_experiments/nnUNet_raw/Dataset902_msLesionAgnostic/imagesTs/ --output-dir ~/net/ms-lesion-agnostic/super_resolution/nnUNet_raw/Dataset902_msLesionAgnostic/imagesTs/ --min-index 305 --max-index 433

Code used is stored here: #62

[plbenveniste]

The entirety of the data was transformed and is stored here ~/net/ms-lesion-agnostic/super_resolution/nnUNet_raw/Dataset902_msLesionAgnostic/