How to resume from previous checkpoints trained under different node topology?

[tjoymeed]

Hi all,

We are doing pretraining Qwen3-30B-A3B LLM using Nemo-Megatron (container Nemo:25.04 but upgraded to latest github repo).

How to resume from previous checkpoints trained under different node topology?

Are there any constraints regarding (tp, pp, ep, cp, sp, vpp, mbs, gbs)?

For example, in resuming from previous checkpoints, what parameter of the above combination can change and which cannot change?

For example, I am guessing ep cannot change?

What are the concrete rules?

I read from somewhere that the versioning system of checkpoint is very "picky" in the sense any slight change will lead the checkpoints to be un-resumable?

How can I bypass the checkpoint versioning system to force max resumability from a previous checkpoint which has slightly different parameter combination?

Thanks!

[suiyoubi]

I think all parallel settings (EP/PP/EP/CP/SP/VPP) can be safely changed during resuming since the checkpoint format is distributed checkpoint.
But I believe you should keep you GBS the same, otherwise the consumed samples will be calculated differently and you might not be able to resume at the desired place

[tjoymeed]

Got you! So GBS cannot change.
And so (tp, pp, ep, cp, sp, vpp) and implicitly the dp can all change, am I right?
What about mbs and ga?

Thanks!

[suiyoubi]

I think as long as you keep the gbs the same, the data resume will be correct.
You can change mbs and ga.

[tjoymeed]

We have tried a few different configurations, but the program just couldn't pick up the previous run checkpoints.

There is no error message so we cannot post here.

But it just could not detect the previous checkpoints.

Sometimes even I don't change anything, it won't detect.

What could be the problem?

Could you please show a working example of the checkpointing and resuming functionality?

Thanks!

[tjoymeed]

Also, I am changing node topology, e.g. from 2 nodes totalling 16 GPU cards to 3 nodes totalling 24 GPU cards, etc.

Is this okay?

Thanks!

[suiyoubi]

The checkpoint resuming simply just try to find the ckpt in the folder with *last suffix.

from 2 nodes totalling 16 GPU cards to 3 nodes totalling 24 GPU cards, etc.
Its fine as long as your GBS is the same

[tjoymeed]

Do you have a working example?

We have tried many different ways of programming the checkpointing functionality.

None of them detected, recognized, and picked up the previous checkpoints - they all start the pretraining from scratch.

Thanks!

[tjoymeed]

Any working example (sample code) that we can follow?

Last time I tried a few checkpointing approaches, none of them detected the previous checkpoints. Strange!

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.