About evaluation metric

[zyqz97]

Hello, I have a new question.
The metrics obtained from the released checkpoint differ from those reported in the paper.
I run the following scripts :

get refine image: resolution is 448 x 256

python -m src.main +experiment=dl3dv_mvsplat360 \ wandb.name=dl3dv_480P_ctx5_tgt56 \ mode=test \ dataset/view_sampler=evaluation \ dataset.roots=[datasets/dl3dv] \ checkpointing.load=checkpoints/dl3dv_480p.ckpt

color adjustment

python src/scripts/post_process.py --root_dir=outputs/test_scores/dl3dv_480P_ctx5_tgt56_download_ckpt

compute the metric

python -m src.scripts.compute_dl3dv_metrics --use_pp

and I modified the compute_dl3dv_metrics.py to read folder.

But I get the metric below, which is different from the paper:
and FID: 16.43

So, I’d like to ask if I might have missed some details when evaluating. Thanks!

[donydchen]

Hi @zyqz97, when testing the PSNR and SSIM, we find it better to run all 56 frames through the SVD (modify the eval command by appending model.refiner.svd_num_frames=56 model.refiner.test_time_attn_num_splits=4), after that, apply the color adjustment post-processing before running the evaluation.

Still, even using the README-provided evaluation commands, the performance should be slightly better than the one you obtained since the modifications I mentioned above are better for visual quality but only bring a small gain for the quantitative results. Perhaps there might be some issues with the data processing codes since it is supposed to be a one-off for now. I will find time to clean those parts of the codes in the future.

[zyqz97]

Thanks for your reply!

I implemented the modifications you suggested (model.refiner.svd_num_frames=56 model.refiner.test_time_attn_num_splits=4). As you said, while the metrics have improved slightly, they still don't quite match the results reported in the paper.

As for the data processing, I used the code from DepthSplat. I'm looking forward to the updates of your code. Thank you!