Question about training the model from scratch

[DavidXu-JJ]

Hi everyone,

I recently found this project, and I found it truly fascinating. Thank the authors for sharing such an inspiring piece of work with the community.

I’ve been experimenting with this public implementation and encountered some difficulties reproducing the results when training the model from scratch.

I don't know whether this issue only occurs to me. Many thanks in advance if anyone can share his/her experience on the training.

Specifically, from the following line in this repository:

Dora/pytorch_lightning/configs/shape-autoencoder/Dora-VAE-train.yaml

Line 25 in 814564c

pretrained_model_name_or_path: ./ckpts/Dora-VAE-1.1/dora_vae_1_1.ckpt

I noticed that a pretrained model is loaded during training by default.

However, I am aiming to train the model entirely from scratch using the 3D-FUTURE dataset (a furniture object dataset with about 10k models), preprocessed with the provided script. (And I think the scale of the model should be aligned with the codebase) Despite following the original codebase closely, my model does not converge well, and the reconstruction results (e.g., the green line) are far from satisfactory.

The blue one is starting with the pretrained checkpoint. The green one is training from scratch.

Given that the codebase is labeled as version 1.1 and the original paper describes version 1.0, I wonder if there are any updated training details or important differences I should be aware of to match the performance of the released checkpoint.

[ARuiChen]

Hello, bro. When doing the finetune from scratch, there should be a warmup stage.
During the warmup stage:

1. The KL loss is only applied to the predicted mean (set deterministic=True), which is equivalent to an autoencoder (AE) with a mean regularization term. The variance will impede the early convergence of the model because each sampling will yield a different latent variable, confusing the model.
2. Turn off the random rotation of the points in the warmup stage(set rotate_points= False)
3. Overfit on a very small subset of the data. Once the loss (sharp logits and coarse logits) level reaches the same level as that of the pre-trained model, then open up all the data and apply the complete KL loss (set deterministic=False).
4. If you want to accelerate the convergence when training from scratch, you can use the features of the intermediate layers of the pre-trained model as the ground truth (gt) during the warmup stage to supervise the features of the intermediate layers of the new model. The purpose of doing this is to provide the model with a good initialization, which is of great help to the convergence of the model.

[DavidXu-JJ]

Thank you so much for your quick reply and your time!

I also noticed that the KL loss has some weird curves. Please let me take some time to try your suggestions.

Your advice is highly appreciated!

(I forgot to place the KL loss in the above figure, so I place it here.)

[creiser]

Hey @ARuiChen,

thanks for providing this awesome code and for giving these additional pointers! I'd also like to train a model from scratch. It seems that my model is not really training well though.

Currently, I am training on a small dataset of only 1024 meshes.

It would greatly help me if you could answer the following questions.

• How large is your warm-up dataset? Does it need to consist primarily of really "simple" meshes or are there other requirements to this dataset?
• Do you already use multi-resolution latents (i.e. vary the latent length between 128 and 2048) during warm-up?
• Do you use lambda_kl = 0.001 during warm-up as well?
• Are you using learning rate warmup or decay? If yes, what is your initial learning rate, your learning rate after warm-up and your final learning rate?

[ARuiChen]

Hi @creiser

• According to the way dora-bench divides different complexity levels, we randomly selected 2K meshes from the whole training set, including 500 meshes at L1 level, 1000 meshes at L2 level, and 500 meshes at L3 level. Using extremely complex L4-level meshes during the warm-up phase may affect convergence. There is no particular requirement for this ratio combination. Due to some copyright issues, we could not provide the specific training list.
• Yes, the multi-resolution strategy was used.
• Yes, lambda_kl = 0.001 during warm-up (set deterministic=True).
• It is feasible to consistently use the same learning rate of 5e-5 without any schedule or decay.

Note that using the features of the intermediate layers of the pre-trained model as the ground truth (gt) during the warmup stage to supervise the features of the intermediate layers of the new model could accelerate the convergence.

[DavidXu-JJ]

Hi @creiser and @ARuiChen,

I have tried to overfit a very small dataset with 35 models.

I read your discussion, and I applied deterministic=True and disabled the rotation(disabled the augmentation).

But unfortunately, I find these warmup adaptations still don't help a lot in getting a better result.

Additionally, I notice that the loss weight relative scaling is different from some other projects, like in:

https://github.com/1zb/VecSetX/blob/d029032d4426c94bd11725b77973a5e879db9451/vecset/engines/engine_ae.py#L97

The coarse_near_logits loss and random_points_logits loss seem to share the same loss weight in this project:

Dora/pytorch_lightning/configs/shape-autoencoder/Dora-VAE-train.yaml

Line 44 in 814564c

lambda_coarse_logits: 1.0

Is it possible that the current loss-weight designs may lead to the drawback of training from scratch? (I try to change the relative loss weight scaling, and these changes make a difference in the loss convergence for my case.)

Thanks for your discussion again.

[cshang-rbx]

Hi @ARuiChen

Thanks for the tips!

I tried training from scratch using dora-bench as a warm-up dataset (800 level-1, 800 level-2, 400 level-3) with:

deterministic=True
rotate_points=False
batch size per gpu: 25
GPUs: 8
Data processing: sharp-edge sampling applied directly to the meshes without the first step using v1.1 setting.

Just wondering

1. How many warm-up steps were used in the released v1.1 model?
2. Is this difficulty expected because learning an SDF is inherently harder than learning occupancy?
3. Any other places I missed?

After ~7 hours of training, here are the loss and IoU curves:
I'll let it running longer, but it seems pretty far than the released model.