Inconsistent time shifting strategy between training and testing

[Luciennnnnnn]

Hi, I notice you leverage FLUX-like time shifting in training, while using fixed time_shifting_factor=6.0 in testing, what's the motivation behind this design?

[ChrisLiu6]

Good question! Your question is related to many very intricate problems. For most of them, we also don't know the true answer, but we are happy to share our understanding.

Should the time-shift factor be consistent between training and inference? Why?

The training timeshift controls how often each span of timestep should be sampled. When the factor goes larger, the earlier timesteps (where stronger noises are applied) would be more likely to be sampled; in other words, training would focus more on the earlier denoising steps. The inference timeshift factor, on the other hand, controls how many steps should be allocated to different timestep spans. When it grows larger, model forwarding will be more dense in the early timestep spans.

While existing works may choose to use the same factor value for training and inference, we are not aware of any theoretical proof showing that this is a must.

Why we use the value 6 for inference?
Because we have manually tried different values and found 6.0 to be a good choice, though it is different from the training value..

Why we use the flux schedule for training?
Because we do not have the resources to ablate training timeshift factors, so we directly followed flux. However, to our knowledge this schedule is also rule of thumb, and the community needs theoretical progress to give better options.