Stepwise Advantages for Multi-Turn Training #536
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
13 tasks
…erifiers into multiturn-discounted-returns
ronaldnetawat
pushed a commit
to ronaldnetawat/verifiers
that referenced
this pull request
Nov 13, 2025
* Extract non-ckpt util function * Implement train ckpt manager * Fix create ckpt path as dir * Do not ckpt at step 0 * Change ckpt name * Fix ckpt step path * Save correct progress state at correct step * Auto-setup max steps * Do not clean by default * Misc improvements to trainer ckpt * Implement orchestrator ckpt * Use same stop condition logic across trainer + orchestrator * Use correct logprob model when resuming * Use resume step instead of path * Cleanup * Resume with async * Init optimizer before load from ckpt * Ensure resume step at least 1 * Load the async logprob model from weights checkpoint * Save pytorch model HF compatible * Load all async logprob models * Fix start ckpt_step * Save all async logprob models correctly * Fixes to async resume logic * Enable resuming a W&B run * Off by one * Implement async save * Disable ckpt by default * Remove ckpt getter * Fix type * Fix keeping async weight ckpts for restart * Remove description * Only clean if not resume * Docs * Increase default seq len to avoid overlong prompt error
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR doesn't change existing behavior, but it adds new options to the
RLConfig:use_stepwise_advantage- If True, treat each assistant turn as its own training sample and use a discounted return per step.stepwise_aggregation- How to compute discounted per-step return R_t from future rewards.stepwise_gamma- Discount factor gamma for previous turns.It also adds a new parameter to
MultiTurnEnvcalledexclude_thinkwhich will remove the CoT from previous turns. I think this only makes sense when using the stepwise advantage implementation.The implementation is based on Kevin: Multi-Turn RL for Generating CUDA Kernels.
Type of Change
Testing
uv run pytestlocally.Checklist
Additional Notes