Kindly Inquiry for Evaluation Issue with Self-Converted DL3DV Chunks in MVSplat360

[YaoXingbo]

Dear @donydchen , thank you for your fantastic work!

Recently, I downloaded a few datasets from DL3DV-480p. I succeeded in using the convert_dl3dv.py to get the torch chunks.

However, when I tried to evaluate on my own chunks through MVSplat360, it stops like the image below:

When I evaluate on your example, it works perfectly, but why does it stop when I use my own converted chunks?

I also tried converting with this script, and eventually got torch chunks similar to yours. However, it still stops at the point shown in the image above.

I have debugged for a super long time, but it still can't work. So I sincerely hope to hear back from you, thank you very very very much! : )

[donydchen]

Hey @YaoXingbo, you might want to look into the DL3DV data loader, which relies on self.chunks(chunk path) and self.index(scene index) to fetch the testing item. In particular, you can uncomment the following lines

mvsplat360/src/dataset/dataset_dl3dv.py

Lines 400 to 402 in 5142f31

	# print("chunk", self.chunks)
	# print("updated index", index)
	# print("self chunks", self.chunks)

And tune your code to ensure the above items are correctly loaded. Hope it helps. Cheers.

[YaoXingbo]

Hey, @donydchen ,thank you for your response.

Perhaps there is an issue with my dataset creation process?

• I first downloaded the dataset from https://huggingface.co/datasets/DL3DV/DL3DV-ALL-480P/tree/main/1K and saved it to the datasets/dl3d/ directory, as shown in the image below:
• Then, I used the following command for conversion:
  python src/scripts/convert_dl3dv.py --input_dir datasets/dl3d --output_dir datasets/dl3d_w_b2o --n_test 3
• And followed the instructions for testing here: https://github.com/donydchen/mvsplat360?tab=readme-ov-file#evaluation
• However, while the dataset you provided in the example works, my own convert dataset stops at Testing: 0/? [00:00<?, ?it/s].

I further created a dataset for testing evaluation(can be view here). 000000.torch is my own conversion, and 000002.torch is from your example. I have already modified the index.json file accordingly.

But still, testing only works for your example dataset(000002.torch), and it stops at my dataset(000000.torch), as shown below:

I sincerely hope to hear back from you : ) , this issue has troubled me for a long time :( .Thank you very much!

[donydchen]

Hi @YaoXingbo, during testing, we only apply to scenes from the DL3DV benchmark. In this case, the testing data loader will only fetch those scenes that are recorded in dl3dv_ctx_5v_tgt_56v.json, which are created from the benchmark scenes.

You can either process the DL3DV benchmark or update the evaluation index configuration ('dl3dv_ctx_5v_tgt_56v.json') to ensure that it records the scene you want to test. Sorry but the dataset preprocessing script is supposed to be a one-off and might need some modification to run correctly. For now, you can compare your processed one with the tiny dataset to ensure it works. I will find time to clean this script.