Training Loss Values.

[mmderakhshani]

Hi, thank you for releasing this GitHub repository. I am trying to reproduce the stage 1 training on ImageNet. Could you please share the W&B log or let me know the initial and final loss values for that stage? I am getting the following loss, and it turns out the model is not converging. Thanks.

[Sierkinhane]

Hi, thanks for your interest. Your loss curve looks a bit weird. When I trained the first stage, the loss would quickly reduce to around 8, and then converge to around 7 (500k steps).

[Sierkinhane]

What's the resolution, are you training on 256x256?

[mmderakhshani]

I am training with 256x256 resolution. Can you tell at which training steps at stage 1, you are getting meaningful sampled images in class conditional manner?

One more thing, can you also tell me what is your total batch size and number of training steps in stage 1?

[Sierkinhane]

Here are the generated samples at 10k iterations. The total batch size of t2i, mmu, and language modeling is 1152, and I guess the batch size of t2i is around 700.

[mmderakhshani]

Thank you for your prompt response! I really appreciate it. Could you clarify what the generated images will look like at the end of Stage 1? I assume that during Stage 1, you are only using ImageNet and RefinedWeb. Is that correct?

…

On Tue, 12 Nov 2024 at 09:19, Jinheng Xie ***@***.***> wrote: Here are the generated samples at 10k iterations. The total batch size of t2i, mmu, and language modeling is 1152, and I guess the batch size of t2i is around 700. image.png (view on web) <https://github.com/user-attachments/assets/95f83469-a300-45ff-a63f-8c0f6c92ed5d> — Reply to this email directly, view it on GitHub <#50 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD7OT66RINXDI2QQIQRYSW32AG2ZDAVCNFSM6AAAAABRSVLPYWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRZHA3DQMRWGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[Sierkinhane]

mmu loss is also used in this stage. Here are samples generated without CFG.

[mmderakhshani]

Could you please clarify whether these samples were generated after the training was completed? Additionally, could you let me know how you adjusted the learning rate? Did you use a method similar to YOLO-style initialization, or did you conduct several experiments to determine the best approach? I want to increase the batch size and train a larger model, so I would appreciate some hints about the initialization of the learning rate.

[Sierkinhane]

"These samples were generated at 499,000 iterations. For the learning rate and adjustments, we just follow Muse[1] and you can refer to the configs, and we didn't put much effort into these configurations.

[1] muse: text-to-image generation via masked generative transformers

[Sierkinhane]

BTW, welcome to star our project :)

[mmderakhshani]

Happy to star! :D

[dhevarghese]

Hi,

Thank you for your work! I'm also working on reproducing the results from the paper. I was able to obtain a similar loss curve to yours for the first stage of training. However, in the second stage, my loss seems to be stuck around 7.

Would you be able to share the loss curve for the second stage of training? I'd like to compare and verify how the training is expected to progress.

Thanks in advance!

[Sierkinhane]

Though the loss oscillated around 7 (actually similar to our loss tendency), the image-text alignment was still improving.

[dhevarghese]

Thanks for your response! That’s good to know. How did you measure image-text alignment during training? Also, in the third stage, did the loss converge further?

[Sierkinhane]

It is easier to learn patterns in the synthetic data in the third stage, so the loss will be smaller but still oscillate around some values.

Btw, welcome to star our repository :)

[dhevarghese]

Ofcourse :) Just did!

Thank you for your response.