Skip to content

Implements Offline APO #106

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 116 commits into
base: main
Choose a base branch
from
Open

Implements Offline APO #106

wants to merge 116 commits into from
+866 −47

Conversation

jdchang1
Copy link
Collaborator

@jdchang1 jdchang1 commented Jul 7, 2025 •
edited
Loading

Main Changes:

  • Added non-preference based offline RL track
  • Offline Dataloader
  • Offline Model (HF/MPT)
  • Offline forward / loss

Backwards compatibility breaking change
offline_rl callback now refers to the single prompt/response case
pairwise_offline_rl callback is what was previously offline_rl

Working runs are:
apo-single-stream-openr1-gBSwx1

wensun and others added 30 commits June 18, 2025 10:14
@wensun
apo initial
5a95cab
@wensun
.
d89cd64
@jdchang1
vllm fix
1d925d2
@jdchang1
critic free
e9dbad9
@jdchang1
cleanup
934e864
@jdchang1
local run
b7cb34e
@wensun
added timeout in rlvr utils
28014a9
@wensun
added timeout in rlvr utils
ed4e554
@wensun
.
de6400c
@wensun
fix some comments issues in local yaml
6b4c385
@wensun
.
b2ae469
@jdchang1 @bcui-db
Update compose_rl/algorithms/online/callback.py
4b51437 
Co-authored-by: bcui-db <[email protected]>
@jdchang1
clean comments
d4ffc59
@jdchang1
comment cleanup
1fd7e2d
@jdchang1
cleanup of callback
c769975
@jdchang1
undo rlvr update
0d275ec
@jdchang1
Merge branch 'main' into wensun/apo
d6c24d0
@jdchang1
fix
202b0a2
@jdchang1
Merge branch 'wensun/apo' of https://github.com/databricks/compose-rl …
fc2f835 
…into wensun/apo
@wensun
apo offline initial implementaiton
d6d81ca
@wensun
apo offline initial implementaiton
dd04bff
@wensun
updates
01d28c7
@wensun
add vstar entries
1c62fdb
@wensun
.
a5fa391
@wensun
.
725e3a2
@wensun
.
a0b0209
@wensun
.
39377ae
@wensun
added ope estimator in apo
ce6d18a
@wensun
.
4f0456e
@wensun
.
06ceb47
jdchang1 and others added 28 commits July 9, 2025 10:45
@jdchang1
remove vstar from preference dat
99165b0
@jdchang1
merge multimodal into
7cd408e
@jdchang1
add pixel values to forward pass
b68ca4b
@jdchang1
offline single stream multimodal support
ba9e453
@jdchang1
temperature scaling
06d6a2f
@jdchang1
add processor to Dataset to ensure proper HF checkpointing
c2da2f6
@wensun
quick test for shape
952a976
@wensun
convert it back
4f5fb8b
@wensun
add another metric to track batch advantage
ddd0ad7
@wensun
.
7eb0951
@wensun
.
0a3b8cd
@jdchang1
quick fix
5d0c53f
@jdchang1
Merge branch 'wensun/offline_apo' of https://github.com/databricks/co…
9b4b9b9 
…mpose-rl into wensun/offline_apo
@wensun
added bce
144d80e
@wensun
temporally just set bce to be true
ba2d9e8
@wensun
.
b94b594
@wensun
.
94359e6
@jdchang1
computation
844470d
@jdchang1
model compatibility
8e25f81
@jdchang1
multistep computation
171b0f1
@jdchang1
multimodal fix
26ec9a4
@jdchang1
batch advantage computation fix
331ae42
@jdchang1
qrpo
2997533
@jdchang1
Update model_methods.py
aaef5f8
@jdchang1
fix
204df50
@jdchang1
Merge branch 'wensun/offline_apo' of https://github.com/databricks/co…
bc61d5b 
…mpose-rl into wensun/offline_apo
@jdchang1
fix
2e50aff
@jdchang1
remove the need for preprocessed image inputs
fdf8ec2
@jdchang1 jdchang1 requested a review from bowenyang008 as a code owner July 17, 2025 19:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bcui-db bcui-db bcui-db left review comments

@dakinggg dakinggg Awaiting requested review from dakinggg

@gupta-abhay gupta-abhay Awaiting requested review from gupta-abhay gupta-abhay is a code owner

@abaheti95 abaheti95 Awaiting requested review from abaheti95 abaheti95 is a code owner

@bowenyang008 bowenyang008 Awaiting requested review from bowenyang008 bowenyang008 is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jdchang1 @bcui-db @wensun