Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BugFix] Better handling of batches in vllm wrapper #2853

Open
wants to merge 2 commits into
base: gh/vmoens/116/base
Choose a base branch
from
Open

[BugFix] Better handling of batches in vllm wrapper #2853

wants to merge 2 commits into from

Conversation

vmoens
Copy link
Contributor

@vmoens vmoens commented Mar 14, 2025 •
edited
Loading

@vmoens
Update
3b0d7fc 
[ghstack-poisoned]
@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Mar 14, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2853

Note: Links to docs will display an error until the docs builds have been completed.

❌ 15 New Failures, 5 Unrelated Failures

As of commit 2db3e78 with merge base 619fec6 (image):

NEW FAILURES - The following jobs have failed:

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Mar 14, 2025
This was referenced Mar 14, 2025
Merged
Merged
Closed
@github-actions GitHub Actions
Copy link

github-actions bot commented Mar 14, 2025 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 149. Improved: $\large\color{#35bf28}17$. Worsened: $\large\color{#d91a1a}5$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_simple 0.5886s 0.5100s 1.9607 Ops/s 1.9229 Ops/s $\color{#35bf28}+1.97\%$
test_transformed 1.0847s 0.9979s 1.0021 Ops/s 0.9551 Ops/s $\color{#35bf28}+4.92\%$
test_serial 1.4781s 1.4737s 0.6786 Ops/s 0.6488 Ops/s $\color{#35bf28}+4.59\%$
test_parallel 1.3193s 1.3058s 0.7658 Ops/s 0.7608 Ops/s $\color{#35bf28}+0.66\%$
test_step_mdp_speed[True-True-True-True-True] 0.2078ms 30.1242μs 33.1959 KOps/s 32.9707 KOps/s $\color{#35bf28}+0.68\%$
test_step_mdp_speed[True-True-True-True-False] 54.4130μs 17.8104μs 56.1471 KOps/s 55.3210 KOps/s $\color{#35bf28}+1.49\%$
test_step_mdp_speed[True-True-True-False-True] 53.9810μs 17.1667μs 58.2524 KOps/s 58.8091 KOps/s $\color{#d91a1a}-0.95\%$
test_step_mdp_speed[True-True-True-False-False] 41.0970μs 10.0576μs 99.4269 KOps/s 100.2754 KOps/s $\color{#d91a1a}-0.85\%$
test_step_mdp_speed[True-True-False-True-True] 87.5250μs 32.0827μs 31.1694 KOps/s 31.2712 KOps/s $\color{#d91a1a}-0.33\%$
test_step_mdp_speed[True-True-False-True-False] 51.3370μs 19.7130μs 50.7280 KOps/s 50.7472 KOps/s $\color{#d91a1a}-0.04\%$
test_step_mdp_speed[True-True-False-False-True] 51.6880μs 18.9941μs 52.6479 KOps/s 52.1072 KOps/s $\color{#35bf28}+1.04\%$
test_step_mdp_speed[True-True-False-False-False] 44.2530μs 11.9558μs 83.6414 KOps/s 84.5982 KOps/s $\color{#d91a1a}-1.13\%$
test_step_mdp_speed[True-False-True-True-True] 86.2620μs 34.4512μs 29.0266 KOps/s 29.3096 KOps/s $\color{#d91a1a}-0.97\%$
test_step_mdp_speed[True-False-True-True-False] 63.3900μs 21.7358μs 46.0070 KOps/s 46.0920 KOps/s $\color{#d91a1a}-0.18\%$
test_step_mdp_speed[True-False-True-False-True] 46.3470μs 19.0944μs 52.3713 KOps/s 52.3959 KOps/s $\color{#d91a1a}-0.05\%$
test_step_mdp_speed[True-False-True-False-False] 37.1600μs 11.8113μs 84.6649 KOps/s 83.6584 KOps/s $\color{#35bf28}+1.20\%$
test_step_mdp_speed[True-False-False-True-True] 74.5810μs 35.9466μs 27.8191 KOps/s 27.7774 KOps/s $\color{#35bf28}+0.15\%$
test_step_mdp_speed[True-False-False-True-False] 53.6920μs 23.2501μs 43.0105 KOps/s 42.5809 KOps/s $\color{#35bf28}+1.01\%$
test_step_mdp_speed[True-False-False-False-True] 62.0170μs 20.6654μs 48.3900 KOps/s 48.2477 KOps/s $\color{#35bf28}+0.30\%$
test_step_mdp_speed[True-False-False-False-False] 56.5360μs 13.6249μs 73.3948 KOps/s 72.1612 KOps/s $\color{#35bf28}+1.71\%$
test_step_mdp_speed[False-True-True-True-True] 0.1146ms 34.0495μs 29.3690 KOps/s 29.0225 KOps/s $\color{#35bf28}+1.19\%$
test_step_mdp_speed[False-True-True-True-False] 75.2320μs 21.8640μs 45.7373 KOps/s 45.7984 KOps/s $\color{#d91a1a}-0.13\%$
test_step_mdp_speed[False-True-True-False-True] 2.2505ms 22.0802μs 45.2895 KOps/s 45.5666 KOps/s $\color{#d91a1a}-0.61\%$
test_step_mdp_speed[False-True-True-False-False] 47.3590μs 14.3085μs 69.8884 KOps/s 74.7779 KOps/s $\textbf{\color{#d91a1a}-6.54\%}$
test_step_mdp_speed[False-True-False-True-True] 79.9410μs 35.9254μs 27.8354 KOps/s 27.5327 KOps/s $\color{#35bf28}+1.10\%$
test_step_mdp_speed[False-True-False-True-False] 70.2230μs 23.2812μs 42.9530 KOps/s 42.5235 KOps/s $\color{#35bf28}+1.01\%$
test_step_mdp_speed[False-True-False-False-True] 60.0430μs 23.5750μs 42.4178 KOps/s 42.7395 KOps/s $\color{#d91a1a}-0.75\%$
test_step_mdp_speed[False-True-False-False-False] 75.0120μs 14.9927μs 66.6991 KOps/s 66.1264 KOps/s $\color{#35bf28}+0.87\%$
test_step_mdp_speed[False-False-True-True-True] 76.2140μs 37.8064μs 26.4505 KOps/s 26.6536 KOps/s $\color{#d91a1a}-0.76\%$
test_step_mdp_speed[False-False-True-True-False] 59.3220μs 25.1991μs 39.6840 KOps/s 39.7380 KOps/s $\color{#d91a1a}-0.14\%$
test_step_mdp_speed[False-False-True-False-True] 53.3200μs 23.5146μs 42.5268 KOps/s 42.7886 KOps/s $\color{#d91a1a}-0.61\%$
test_step_mdp_speed[False-False-True-False-False] 43.0410μs 15.1604μs 65.9613 KOps/s 66.4608 KOps/s $\color{#d91a1a}-0.75\%$
test_step_mdp_speed[False-False-False-True-True] 72.9770μs 39.0044μs 25.6381 KOps/s 25.3829 KOps/s $\color{#35bf28}+1.01\%$
test_step_mdp_speed[False-False-False-True-False] 62.9280μs 26.8166μs 37.2903 KOps/s 37.1482 KOps/s $\color{#35bf28}+0.38\%$
test_step_mdp_speed[False-False-False-False-True] 58.6000μs 25.3110μs 39.5085 KOps/s 39.5929 KOps/s $\color{#d91a1a}-0.21\%$
test_step_mdp_speed[False-False-False-False-False] 45.4850μs 16.8164μs 59.4658 KOps/s 59.5853 KOps/s $\color{#d91a1a}-0.20\%$
test_values[generalized_advantage_estimate-True-True] 9.8913ms 9.6731ms 103.3792 Ops/s 101.3358 Ops/s $\color{#35bf28}+2.02\%$
test_values[vec_generalized_advantage_estimate-True-True] 27.0627ms 24.7951ms 40.3306 Ops/s 38.3000 Ops/s $\textbf{\color{#35bf28}+5.30\%}$
test_values[td0_return_estimate-False-False] 0.2595ms 0.1800ms 5.5570 KOps/s 5.6749 KOps/s $\color{#d91a1a}-2.08\%$
test_values[td1_return_estimate-False-False] 27.4454ms 23.5370ms 42.4864 Ops/s 41.0774 Ops/s $\color{#35bf28}+3.43\%$
test_values[vec_td1_return_estimate-False-False] 27.0145ms 24.6160ms 40.6239 Ops/s 38.1891 Ops/s $\textbf{\color{#35bf28}+6.38\%}$
test_values[td_lambda_return_estimate-True-False] 37.3937ms 33.5418ms 29.8135 Ops/s 28.8388 Ops/s $\color{#35bf28}+3.38\%$
test_values[vec_td_lambda_return_estimate-True-False] 27.5068ms 24.7210ms 40.4514 Ops/s 38.3017 Ops/s $\textbf{\color{#35bf28}+5.61\%}$
test_gae_speed[generalized_advantage_estimate-False-1-512] 8.6030ms 8.3485ms 119.7823 Ops/s 117.3629 Ops/s $\color{#35bf28}+2.06\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 2.4837ms 1.8970ms 527.1592 Ops/s 500.2606 Ops/s $\textbf{\color{#35bf28}+5.38\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.4800ms 0.3574ms 2.7983 KOps/s 2.7137 KOps/s $\color{#35bf28}+3.12\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 47.6920ms 45.5725ms 21.9431 Ops/s 21.8146 Ops/s $\color{#35bf28}+0.59\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 4.5191ms 3.4341ms 291.1960 Ops/s 287.7830 Ops/s $\color{#35bf28}+1.19\%$
test_dqn_speed[False-None] 5.5897ms 1.3853ms 721.8483 Ops/s 689.7565 Ops/s $\color{#35bf28}+4.65\%$
test_dqn_speed[False-backward] 1.9209ms 1.8540ms 539.3837 Ops/s 518.7877 Ops/s $\color{#35bf28}+3.97\%$
test_dqn_speed[True-None] 0.6856ms 0.5564ms 1.7971 KOps/s 1.7669 KOps/s $\color{#35bf28}+1.71\%$
test_dqn_speed[True-backward] 1.0236ms 0.9867ms 1.0135 KOps/s 977.0663 Ops/s $\color{#35bf28}+3.73\%$
test_dqn_speed[reduce-overhead-None] 0.7386ms 0.5645ms 1.7715 KOps/s 1.7878 KOps/s $\color{#d91a1a}-0.92\%$
test_dqn_speed[reduce-overhead-backward] 1.0220ms 0.9705ms 1.0304 KOps/s 985.9151 Ops/s $\color{#35bf28}+4.51\%$
test_ddpg_speed[False-None] 3.5919ms 2.8545ms 350.3184 Ops/s 342.8195 Ops/s $\color{#35bf28}+2.19\%$
test_ddpg_speed[False-backward] 4.1196ms 3.9807ms 251.2099 Ops/s 247.8071 Ops/s $\color{#35bf28}+1.37\%$
test_ddpg_speed[True-None] 1.6616ms 1.4301ms 699.2560 Ops/s 693.8978 Ops/s $\color{#35bf28}+0.77\%$
test_ddpg_speed[True-backward] 2.6543ms 2.3493ms 425.6523 Ops/s 399.7873 Ops/s $\textbf{\color{#35bf28}+6.47\%}$
test_ddpg_speed[reduce-overhead-None] 1.9024ms 1.4412ms 693.8853 Ops/s 689.4648 Ops/s $\color{#35bf28}+0.64\%$
test_ddpg_speed[reduce-overhead-backward] 2.3707ms 2.3184ms 431.3317 Ops/s 426.2314 Ops/s $\color{#35bf28}+1.20\%$
test_sac_speed[False-None] 9.8690ms 8.1358ms 122.9142 Ops/s 122.6154 Ops/s $\color{#35bf28}+0.24\%$
test_sac_speed[False-backward] 12.4230ms 10.6322ms 94.0543 Ops/s 92.2415 Ops/s $\color{#35bf28}+1.97\%$
test_sac_speed[True-None] 4.3606ms 2.5936ms 385.5574 Ops/s 368.2610 Ops/s $\color{#35bf28}+4.70\%$
test_sac_speed[True-backward] 4.2893ms 4.2456ms 235.5393 Ops/s 234.1951 Ops/s $\color{#35bf28}+0.57\%$
test_sac_speed[reduce-overhead-None] 2.8451ms 2.5878ms 386.4217 Ops/s 383.9641 Ops/s $\color{#35bf28}+0.64\%$
test_sac_speed[reduce-overhead-backward] 4.5747ms 4.2850ms 233.3700 Ops/s 233.4191 Ops/s $\color{#d91a1a}-0.02\%$
test_redq_speed[False-None] 16.1592ms 13.2146ms 75.6741 Ops/s 73.0622 Ops/s $\color{#35bf28}+3.57\%$
test_redq_speed[False-backward] 27.5383ms 22.6059ms 44.2363 Ops/s 42.8664 Ops/s $\color{#35bf28}+3.20\%$
test_redq_speed[True-None] 9.4976ms 6.7593ms 147.9452 Ops/s 146.2769 Ops/s $\color{#35bf28}+1.14\%$
test_redq_speed[True-backward] 17.5640ms 15.2426ms 65.6057 Ops/s 69.8124 Ops/s $\textbf{\color{#d91a1a}-6.03\%}$
test_redq_speed[reduce-overhead-None] 9.9366ms 8.3418ms 119.8789 Ops/s 141.0266 Ops/s $\textbf{\color{#d91a1a}-15.00\%}$
test_redq_speed[reduce-overhead-backward] 15.6713ms 14.5741ms 68.6149 Ops/s 70.0806 Ops/s $\color{#d91a1a}-2.09\%$
test_redq_deprec_speed[False-None] 15.6122ms 13.4034ms 74.6082 Ops/s 77.6223 Ops/s $\color{#d91a1a}-3.88\%$
test_redq_deprec_speed[False-backward] 20.2913ms 19.0313ms 52.5450 Ops/s 53.3239 Ops/s $\color{#d91a1a}-1.46\%$
test_redq_deprec_speed[True-None] 5.8988ms 5.2714ms 189.7047 Ops/s 185.5714 Ops/s $\color{#35bf28}+2.23\%$
test_redq_deprec_speed[True-backward] 10.8500ms 10.0039ms 99.9610 Ops/s 88.7169 Ops/s $\textbf{\color{#35bf28}+12.67\%}$
test_redq_deprec_speed[reduce-overhead-None] 6.0114ms 5.2104ms 191.9255 Ops/s 177.9232 Ops/s $\textbf{\color{#35bf28}+7.87\%}$
test_redq_deprec_speed[reduce-overhead-backward] 10.7885ms 10.0478ms 99.5240 Ops/s 95.4133 Ops/s $\color{#35bf28}+4.31\%$
test_td3_speed[False-None] 8.7183ms 8.1458ms 122.7625 Ops/s 123.3213 Ops/s $\color{#d91a1a}-0.45\%$
test_td3_speed[False-backward] 12.7724ms 10.7345ms 93.1578 Ops/s 91.4131 Ops/s $\color{#35bf28}+1.91\%$
test_td3_speed[True-None] 2.9486ms 2.3519ms 425.1863 Ops/s 400.3362 Ops/s $\textbf{\color{#35bf28}+6.21\%}$
test_td3_speed[True-backward] 4.8275ms 4.2766ms 233.8323 Ops/s 241.2393 Ops/s $\color{#d91a1a}-3.07\%$
test_td3_speed[reduce-overhead-None] 2.5417ms 2.2828ms 438.0626 Ops/s 423.7712 Ops/s $\color{#35bf28}+3.37\%$
test_td3_speed[reduce-overhead-backward] 4.0304ms 3.9466ms 253.3854 Ops/s 232.0667 Ops/s $\textbf{\color{#35bf28}+9.19\%}$
test_cql_speed[False-None] 39.4878ms 37.0904ms 26.9612 Ops/s 26.4955 Ops/s $\color{#35bf28}+1.76\%$
test_cql_speed[False-backward] 55.2285ms 47.7072ms 20.9612 Ops/s 20.6025 Ops/s $\color{#35bf28}+1.74\%$
test_cql_speed[True-None] 24.0043ms 22.1833ms 45.0790 Ops/s 43.5282 Ops/s $\color{#35bf28}+3.56\%$
test_cql_speed[True-backward] 31.5749ms 29.4014ms 34.0119 Ops/s 33.3686 Ops/s $\color{#35bf28}+1.93\%$
test_cql_speed[reduce-overhead-None] 23.9981ms 22.2373ms 44.9694 Ops/s 43.3446 Ops/s $\color{#35bf28}+3.75\%$
test_cql_speed[reduce-overhead-backward] 30.1401ms 29.1143ms 34.3474 Ops/s 33.2902 Ops/s $\color{#35bf28}+3.18\%$
test_a2c_speed[False-None] 8.7157ms 7.1833ms 139.2115 Ops/s 130.3906 Ops/s $\textbf{\color{#35bf28}+6.76\%}$
test_a2c_speed[False-backward] 16.3522ms 14.0159ms 71.3475 Ops/s 65.4789 Ops/s $\textbf{\color{#35bf28}+8.96\%}$
test_a2c_speed[True-None] 5.5783ms 4.6777ms 213.7822 Ops/s 204.6199 Ops/s $\color{#35bf28}+4.48\%$
test_a2c_speed[True-backward] 12.8751ms 11.1300ms 89.8470 Ops/s 84.8497 Ops/s $\textbf{\color{#35bf28}+5.89\%}$
test_a2c_speed[reduce-overhead-None] 5.1839ms 4.7167ms 212.0139 Ops/s 200.2742 Ops/s $\textbf{\color{#35bf28}+5.86\%}$
test_a2c_speed[reduce-overhead-backward] 15.4532ms 11.9430ms 83.7308 Ops/s 87.6401 Ops/s $\color{#d91a1a}-4.46\%$
test_ppo_speed[False-None] 9.2315ms 7.7528ms 128.9860 Ops/s 128.4236 Ops/s $\color{#35bf28}+0.44\%$
test_ppo_speed[False-backward] 17.3057ms 15.5356ms 64.3682 Ops/s 64.7071 Ops/s $\color{#d91a1a}-0.52\%$
test_ppo_speed[True-None] 5.9182ms 5.1781ms 193.1229 Ops/s 186.9048 Ops/s $\color{#35bf28}+3.33\%$
test_ppo_speed[True-backward] 12.4994ms 11.0558ms 90.4506 Ops/s 89.6387 Ops/s $\color{#35bf28}+0.91\%$
test_ppo_speed[reduce-overhead-None] 6.0711ms 5.1605ms 193.7802 Ops/s 195.0103 Ops/s $\color{#d91a1a}-0.63\%$
test_ppo_speed[reduce-overhead-backward] 13.2398ms 11.6682ms 85.7028 Ops/s 87.6570 Ops/s $\color{#d91a1a}-2.23\%$
test_reinforce_speed[False-None] 7.6256ms 6.7085ms 149.0646 Ops/s 148.3685 Ops/s $\color{#35bf28}+0.47\%$
test_reinforce_speed[False-backward] 10.4765ms 10.0504ms 99.4983 Ops/s 100.8191 Ops/s $\color{#d91a1a}-1.31\%$
test_reinforce_speed[True-None] 4.8290ms 4.0958ms 244.1539 Ops/s 237.0527 Ops/s $\color{#35bf28}+3.00\%$
test_reinforce_speed[True-backward] 17.9978ms 10.6511ms 93.8872 Ops/s 94.8852 Ops/s $\color{#d91a1a}-1.05\%$
test_reinforce_speed[reduce-overhead-None] 5.0698ms 4.1243ms 242.4654 Ops/s 211.2773 Ops/s $\textbf{\color{#35bf28}+14.76\%}$
test_reinforce_speed[reduce-overhead-backward] 11.4590ms 10.1849ms 98.1845 Ops/s 94.6745 Ops/s $\color{#35bf28}+3.71\%$
test_iql_speed[False-None] 37.4278ms 32.7219ms 30.5606 Ops/s 30.0937 Ops/s $\color{#35bf28}+1.55\%$
test_iql_speed[False-backward] 49.2240ms 45.8326ms 21.8185 Ops/s 21.4116 Ops/s $\color{#35bf28}+1.90\%$
test_iql_speed[True-None] 17.9181ms 15.8569ms 63.0638 Ops/s 60.7688 Ops/s $\color{#35bf28}+3.78\%$
test_iql_speed[True-backward] 29.8324ms 27.8783ms 35.8702 Ops/s 35.5297 Ops/s $\color{#35bf28}+0.96\%$
test_iql_speed[reduce-overhead-None] 17.3489ms 16.2320ms 61.6068 Ops/s 61.1790 Ops/s $\color{#35bf28}+0.70\%$
test_iql_speed[reduce-overhead-backward] 29.9689ms 27.6996ms 36.1016 Ops/s 35.8129 Ops/s $\color{#35bf28}+0.81\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 6.5215ms 4.9835ms 200.6619 Ops/s 202.2682 Ops/s $\color{#d91a1a}-0.79\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.8271ms 0.5497ms 1.8190 KOps/s 1.8084 KOps/s $\color{#35bf28}+0.59\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.7740ms 0.5175ms 1.9324 KOps/s 1.9136 KOps/s $\color{#35bf28}+0.98\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 6.5346ms 4.6683ms 214.2114 Ops/s 212.5436 Ops/s $\color{#35bf28}+0.78\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 2.8447ms 0.5375ms 1.8604 KOps/s 1.8661 KOps/s $\color{#d91a1a}-0.31\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.7644ms 0.5114ms 1.9555 KOps/s 1.9297 KOps/s $\color{#35bf28}+1.34\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 2.5506ms 1.6941ms 590.2846 Ops/s 584.3926 Ops/s $\color{#35bf28}+1.01\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 2.1584ms 1.6012ms 624.5209 Ops/s 619.5968 Ops/s $\color{#35bf28}+0.79\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 5.8532ms 4.9545ms 201.8385 Ops/s 201.5869 Ops/s $\color{#35bf28}+0.12\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.1185ms 0.6785ms 1.4739 KOps/s 1.4813 KOps/s $\color{#d91a1a}-0.50\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.9996ms 0.6536ms 1.5300 KOps/s 1.5331 KOps/s $\color{#d91a1a}-0.20\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 4.9954ms 4.7646ms 209.8832 Ops/s 211.2957 Ops/s $\color{#d91a1a}-0.67\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 1.3237ms 0.5491ms 1.8212 KOps/s 1.8316 KOps/s $\color{#d91a1a}-0.57\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.8128ms 0.5192ms 1.9261 KOps/s 1.9341 KOps/s $\color{#d91a1a}-0.41\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 8.4132ms 4.7759ms 209.3837 Ops/s 213.9227 Ops/s $\color{#d91a1a}-2.12\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 2.7413ms 0.5472ms 1.8273 KOps/s 1.8896 KOps/s $\color{#d91a1a}-3.30\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.7782ms 0.5130ms 1.9493 KOps/s 1.9100 KOps/s $\color{#35bf28}+2.05\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 8.1280ms 4.9974ms 200.1048 Ops/s 209.7515 Ops/s $\color{#d91a1a}-4.60\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.2781ms 0.6966ms 1.4356 KOps/s 1.4823 KOps/s $\color{#d91a1a}-3.15\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.9772ms 0.6650ms 1.5037 KOps/s 1.5292 KOps/s $\color{#d91a1a}-1.67\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 5.7298ms 4.3772ms 228.4587 Ops/s 231.0032 Ops/s $\color{#d91a1a}-1.10\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 8.2121ms 2.4000ms 416.6733 Ops/s 414.4242 Ops/s $\color{#35bf28}+0.54\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 4.6434ms 1.3794ms 724.9278 Ops/s 743.5066 Ops/s $\color{#d91a1a}-2.50\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 7.4662ms 4.5439ms 220.0733 Ops/s 234.3966 Ops/s $\textbf{\color{#d91a1a}-6.11\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 8.7334ms 2.4381ms 410.1637 Ops/s 420.5395 Ops/s $\color{#d91a1a}-2.47\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 2.7813ms 1.3831ms 723.0394 Ops/s 677.6897 Ops/s $\textbf{\color{#35bf28}+6.69\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.8163s 20.7507ms 48.1912 Ops/s 224.5389 Ops/s $\textbf{\color{#d91a1a}-78.54\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 3.5728ms 2.3103ms 432.8503 Ops/s 395.6124 Ops/s $\textbf{\color{#35bf28}+9.41\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 2.0710ms 1.4326ms 698.0277 Ops/s 635.6483 Ops/s $\textbf{\color{#35bf28}+9.81\%}$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True] 66.1364ms 50.2807ms 19.8884 Ops/s 19.6751 Ops/s $\color{#35bf28}+1.08\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False] 15.4016ms 14.3197ms 69.8340 Ops/s 68.3863 Ops/s $\color{#35bf28}+2.12\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True] 64.1160ms 51.9998ms 19.2308 Ops/s 19.8671 Ops/s $\color{#d91a1a}-3.20\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False] 16.7501ms 15.0774ms 66.3243 Ops/s 69.0306 Ops/s $\color{#d91a1a}-3.92\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True] 64.9317ms 50.1274ms 19.9492 Ops/s 19.9981 Ops/s $\color{#d91a1a}-0.24\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False] 16.8114ms 15.9521ms 62.6877 Ops/s 64.0760 Ops/s $\color{#d91a1a}-2.17\%$

@vmoens
Update
2db3e78 
[ghstack-poisoned]
tianyu1997 pushed a commit to tianyu1997/RL that referenced this pull request Mar 18, 2025
@vmoens
[BugFix] Better handling of batches in vllm wrapper
1da5d20 
ghstack-source-id: 73d41d803125647fb2902f31a97a443a5c607112
Pull Request resolved: pytorch/rl#2853
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@vmoens @facebook-github-bot