[Feature] Tokenizer for LLMEnv

[vmoens]

Stack from ghstack (oldest at bottom):

• [Feature] Allow using lists of tensors in vllm instead of padded tensors #2861
• [Feature] pass policy-factory in mp data collectors #2859
• [Feature] Make PPO ready for text-based data #2857
• [Feature] Set padded token log-prob to 0.0 #2856
• [BugFix] Right log-prob size in transformer wrapper #2854
• -> [Feature] Tokenizer for LLMEnv #2852

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2852

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

'&' can not be used here

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$

Result of CPU Benchmark Tests

Total Benchmarks: 149. Improved: $11$. Worsened: $13$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_simple	0.6469s	0.5528s	1.8088 Ops/s	1.8487 Ops/s	$-2.16\mathrm{\%}$
test_transformed	1.1955s	1.1035s	0.9062 Ops/s	0.9410 Ops/s	$-3.70\mathrm{\%}$
test_serial	1.7104s	1.6111s	0.6207 Ops/s	0.6340 Ops/s	$-2.09\mathrm{\%}$
test_parallel	1.4389s	1.3553s	0.7378 Ops/s	0.7577 Ops/s	$-2.62\mathrm{\%}$
test_step_mdp_speed[True-True-True-True-True]	0.2744ms	30.5745μs	32.7070 KOps/s	32.5730 KOps/s	$+0.41\mathrm{\%}$
test_step_mdp_speed[True-True-True-True-False]	59.9720μs	18.2339μs	54.8430 KOps/s	55.3711 KOps/s	$-0.95\mathrm{\%}$
test_step_mdp_speed[True-True-True-False-True]	50.8740μs	17.6655μs	56.6074 KOps/s	57.9431 KOps/s	$-2.31\mathrm{\%}$
test_step_mdp_speed[True-True-True-False-False]	39.1830μs	10.4708μs	95.5033 KOps/s	98.9699 KOps/s	$-3.50\mathrm{\%}$
test_step_mdp_speed[True-True-False-True-True]	77.1140μs	32.6091μs	30.6662 KOps/s	30.6618 KOps/s	$+0.01\mathrm{\%}$
test_step_mdp_speed[True-True-False-True-False]	52.9490μs	20.1984μs	49.5089 KOps/s	50.2407 KOps/s	$-1.46\mathrm{\%}$
test_step_mdp_speed[True-True-False-False-True]	55.2830μs	19.5715μs	51.0948 KOps/s	52.2068 KOps/s	$-2.13\mathrm{\%}$
test_step_mdp_speed[True-True-False-False-False]	58.3690μs	12.2863μs	81.3913 KOps/s	82.6900 KOps/s	$-1.57\mathrm{\%}$
test_step_mdp_speed[True-False-True-True-True]	82.0030μs	34.6898μs	28.8269 KOps/s	29.0815 KOps/s	$-0.88\mathrm{\%}$
test_step_mdp_speed[True-False-True-True-False]	66.5540μs	22.0784μs	45.2931 KOps/s	45.1194 KOps/s	$+0.38\mathrm{\%}$
test_step_mdp_speed[True-False-True-False-True]	74.6390μs	19.3378μs	51.7121 KOps/s	51.8426 KOps/s	$-0.25\mathrm{\%}$
test_step_mdp_speed[True-False-True-False-False]	57.7180μs	12.3145μs	81.2052 KOps/s	83.0722 KOps/s	$-2.25\mathrm{\%}$
test_step_mdp_speed[True-False-False-True-True]	86.0100μs	36.4025μs	27.4706 KOps/s	27.6579 KOps/s	$-0.68\mathrm{\%}$
test_step_mdp_speed[True-False-False-True-False]	55.1530μs	23.7059μs	42.1836 KOps/s	42.3837 KOps/s	$-0.47\mathrm{\%}$
test_step_mdp_speed[True-False-False-False-True]	76.9430μs	21.1145μs	47.3608 KOps/s	47.7354 KOps/s	$-0.78\mathrm{\%}$
test_step_mdp_speed[True-False-False-False-False]	46.5970μs	14.0413μs	71.2185 KOps/s	71.5915 KOps/s	$-0.52\mathrm{\%}$
test_step_mdp_speed[False-True-True-True-True]	86.1110μs	34.8615μs	28.6850 KOps/s	28.6631 KOps/s	$+0.08\mathrm{\%}$
test_step_mdp_speed[False-True-True-True-False]	65.1510μs	22.1318μs	45.1838 KOps/s	45.5636 KOps/s	$-0.83\mathrm{\%}$
test_step_mdp_speed[False-True-True-False-True]	68.8180μs	21.9465μs	45.5653 KOps/s	45.7097 KOps/s	$-0.32\mathrm{\%}$
test_step_mdp_speed[False-True-True-False-False]	47.7090μs	13.5622μs	73.7344 KOps/s	74.9777 KOps/s	$-1.66\mathrm{\%}$
test_step_mdp_speed[False-True-False-True-True]	0.1118ms	36.5964μs	27.3251 KOps/s	27.0739 KOps/s	$+0.93\mathrm{\%}$
test_step_mdp_speed[False-True-False-True-False]	2.4079ms	23.9225μs	41.8017 KOps/s	41.5722 KOps/s	$+0.55\mathrm{\%}$
test_step_mdp_speed[False-True-False-False-True]	60.0320μs	23.8185μs	41.9841 KOps/s	42.2243 KOps/s	$-0.57\mathrm{\%}$
test_step_mdp_speed[False-True-False-False-False]	48.4000μs	15.4586μs	64.6890 KOps/s	65.6127 KOps/s	$-1.41\mathrm{\%}$
test_step_mdp_speed[False-False-True-True-True]	80.1990μs	38.2609μs	26.1363 KOps/s	25.9439 KOps/s	$+0.74\mathrm{\%}$
test_step_mdp_speed[False-False-True-True-False]	62.1950μs	25.6639μs	38.9653 KOps/s	39.0822 KOps/s	$-0.30\mathrm{\%}$
test_step_mdp_speed[False-False-True-False-True]	51.6670μs	23.7323μs	42.1367 KOps/s	42.7200 KOps/s	$-1.37\mathrm{\%}$
test_step_mdp_speed[False-False-True-False-False]	47.1680μs	15.5458μs	64.3261 KOps/s	66.0597 KOps/s	$-2.62\mathrm{\%}$
test_step_mdp_speed[False-False-False-True-True]	0.1026ms	39.7963μs	25.1279 KOps/s	25.1593 KOps/s	$-0.12\mathrm{\%}$
test_step_mdp_speed[False-False-False-True-False]	74.3890μs	27.1619μs	36.8163 KOps/s	36.7005 KOps/s	$+0.32\mathrm{\%}$
test_step_mdp_speed[False-False-False-False-True]	72.9460μs	25.3120μs	39.5070 KOps/s	39.6825 KOps/s	$-0.44\mathrm{\%}$
test_step_mdp_speed[False-False-False-False-False]	62.2360μs	17.0398μs	58.6861 KOps/s	59.0965 KOps/s	$-0.69\mathrm{\%}$
test_values[generalized_advantage_estimate-True-True]	12.7021ms	10.3259ms	96.8440 Ops/s	102.5675 Ops/s	$\mathbf{\text{-5.58\%}}$
test_values[vec_generalized_advantage_estimate-True-True]	30.1799ms	26.9932ms	37.0463 Ops/s	41.1072 Ops/s	$\mathbf{\text{-9.88\%}}$
test_values[td0_return_estimate-False-False]	1.0146ms	0.2302ms	4.3445 KOps/s	5.4461 KOps/s	$\mathbf{\text{-20.23\%}}$
test_values[td1_return_estimate-False-False]	29.1547ms	25.2654ms	39.5798 Ops/s	41.4449 Ops/s	$-4.50\mathrm{\%}$
test_values[vec_td1_return_estimate-False-False]	31.2971ms	27.4221ms	36.4670 Ops/s	40.9723 Ops/s	$\mathbf{\text{-11.00\%}}$
test_values[td_lambda_return_estimate-True-False]	36.7722ms	35.9711ms	27.8001 Ops/s	28.3048 Ops/s	$-1.78\mathrm{\%}$
test_values[vec_td_lambda_return_estimate-True-False]	32.7655ms	26.9043ms	37.1688 Ops/s	39.5628 Ops/s	$\mathbf{\text{-6.05\%}}$
test_gae_speed[generalized_advantage_estimate-False-1-512]	8.9335ms	8.6579ms	115.5008 Ops/s	116.3649 Ops/s	$-0.74\mathrm{\%}$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	3.4309ms	2.0315ms	492.2471 Ops/s	513.6427 Ops/s	$-4.17\mathrm{\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.5638ms	0.3720ms	2.6881 KOps/s	2.6733 KOps/s	$+0.55\mathrm{\%}$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	56.3743ms	47.5225ms	21.0427 Ops/s	22.6009 Ops/s	$\mathbf{\text{-6.89\%}}$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	4.6405ms	3.5670ms	280.3465 Ops/s	280.6736 Ops/s	$-0.12\mathrm{\%}$
test_dqn_speed[False-None]	6.6892ms	1.4948ms	669.0035 Ops/s	687.7372 Ops/s	$-2.72\mathrm{\%}$
test_dqn_speed[False-backward]	2.0927ms	1.9883ms	502.9372 Ops/s	516.6431 Ops/s	$-2.65\mathrm{\%}$
test_dqn_speed[True-None]	0.7384ms	0.5745ms	1.7406 KOps/s	1.7402 KOps/s	$+0.02\mathrm{\%}$
test_dqn_speed[True-backward]	1.0490ms	1.0038ms	996.1955 Ops/s	757.5364 Ops/s	$\mathbf{\text{+31.50\%}}$
test_dqn_speed[reduce-overhead-None]	0.9800ms	0.5839ms	1.7125 KOps/s	1.7280 KOps/s	$-0.90\mathrm{\%}$
test_dqn_speed[reduce-overhead-backward]	1.1230ms	1.0166ms	983.6499 Ops/s	956.7195 Ops/s	$+2.81\mathrm{\%}$
test_ddpg_speed[False-None]	4.1347ms	3.0485ms	328.0338 Ops/s	334.5729 Ops/s	$-1.95\mathrm{\%}$
test_ddpg_speed[False-backward]	4.5208ms	4.2684ms	234.2809 Ops/s	240.4402 Ops/s	$-2.56\mathrm{\%}$
test_ddpg_speed[True-None]	1.9350ms	1.4762ms	677.4024 Ops/s	681.4963 Ops/s	$-0.60\mathrm{\%}$
test_ddpg_speed[True-backward]	2.8424ms	2.4915ms	401.3639 Ops/s	422.0464 Ops/s	$-4.90\mathrm{\%}$
test_ddpg_speed[reduce-overhead-None]	2.3467ms	1.4850ms	673.3955 Ops/s	679.0257 Ops/s	$-0.83\mathrm{\%}$
test_ddpg_speed[reduce-overhead-backward]	2.7603ms	2.4454ms	408.9276 Ops/s	413.5263 Ops/s	$-1.11\mathrm{\%}$
test_sac_speed[False-None]	10.0218ms	8.8638ms	112.8179 Ops/s	119.6020 Ops/s	$\mathbf{\text{-5.67\%}}$
test_sac_speed[False-backward]	14.5564ms	11.6293ms	85.9899 Ops/s	90.1738 Ops/s	$-4.64\mathrm{\%}$
test_sac_speed[True-None]	3.3644ms	2.6601ms	375.9243 Ops/s	376.9078 Ops/s	$-0.26\mathrm{\%}$
test_sac_speed[True-backward]	5.3506ms	4.4215ms	226.1671 Ops/s	208.2328 Ops/s	$\mathbf{\text{+8.61\%}}$
test_sac_speed[reduce-overhead-None]	3.3310ms	2.6701ms	374.5199 Ops/s	360.9412 Ops/s	$+3.76\mathrm{\%}$
test_sac_speed[reduce-overhead-backward]	5.8015ms	5.1978ms	192.3890 Ops/s	210.1633 Ops/s	$\mathbf{\text{-8.46\%}}$
test_redq_speed[False-None]	21.3397ms	14.8587ms	67.3006 Ops/s	71.8621 Ops/s	$\mathbf{\text{-6.35\%}}$
test_redq_speed[False-backward]	27.1032ms	24.8920ms	40.1736 Ops/s	42.7717 Ops/s	$\mathbf{\text{-6.07\%}}$
test_redq_speed[True-None]	10.3106ms	8.5278ms	117.2631 Ops/s	124.7626 Ops/s	$\mathbf{\text{-6.01\%}}$
test_redq_speed[True-backward]	18.2141ms	15.7864ms	63.3457 Ops/s	64.0252 Ops/s	$-1.06\mathrm{\%}$
test_redq_speed[reduce-overhead-None]	9.0576ms	7.7844ms	128.4624 Ops/s	125.2104 Ops/s	$+2.60\mathrm{\%}$
test_redq_speed[reduce-overhead-backward]	17.5285ms	15.9575ms	62.6666 Ops/s	64.0201 Ops/s	$-2.11\mathrm{\%}$
test_redq_deprec_speed[False-None]	15.9404ms	14.9682ms	66.8082 Ops/s	68.3060 Ops/s	$-2.19\mathrm{\%}$
test_redq_deprec_speed[False-backward]	22.9145ms	21.3769ms	46.7794 Ops/s	48.3927 Ops/s	$-3.33\mathrm{\%}$
test_redq_deprec_speed[True-None]	7.3408ms	6.0954ms	164.0588 Ops/s	162.7308 Ops/s	$+0.82\mathrm{\%}$
test_redq_deprec_speed[True-backward]	12.5723ms	11.3801ms	87.8726 Ops/s	88.7076 Ops/s	$-0.94\mathrm{\%}$
test_redq_deprec_speed[reduce-overhead-None]	6.5356ms	5.7959ms	172.5351 Ops/s	167.0454 Ops/s	$+3.29\mathrm{\%}$
test_redq_deprec_speed[reduce-overhead-backward]	11.5011ms	10.8930ms	91.8023 Ops/s	85.1107 Ops/s	$\mathbf{\text{+7.86\%}}$
test_td3_speed[False-None]	9.5919ms	8.5749ms	116.6198 Ops/s	116.6807 Ops/s	$-0.05\mathrm{\%}$
test_td3_speed[False-backward]	15.6649ms	11.7272ms	85.2717 Ops/s	87.7094 Ops/s	$-2.78\mathrm{\%}$
test_td3_speed[True-None]	2.7966ms	2.3477ms	425.9486 Ops/s	420.1949 Ops/s	$+1.37\mathrm{\%}$
test_td3_speed[True-backward]	4.8369ms	4.2203ms	236.9480 Ops/s	241.1460 Ops/s	$-1.74\mathrm{\%}$
test_td3_speed[reduce-overhead-None]	2.5520ms	2.3142ms	432.1168 Ops/s	427.0098 Ops/s	$+1.20\mathrm{\%}$
test_td3_speed[reduce-overhead-backward]	4.3215ms	4.0397ms	247.5445 Ops/s	249.5633 Ops/s	$-0.81\mathrm{\%}$
test_cql_speed[False-None]	40.0076ms	37.7755ms	26.4722 Ops/s	26.0791 Ops/s	$+1.51\mathrm{\%}$
test_cql_speed[False-backward]	55.6178ms	50.1232ms	19.9508 Ops/s	20.2839 Ops/s	$-1.64\mathrm{\%}$
test_cql_speed[True-None]	24.2037ms	23.3480ms	42.8302 Ops/s	44.3030 Ops/s	$-3.32\mathrm{\%}$
test_cql_speed[True-backward]	31.5729ms	30.3055ms	32.9973 Ops/s	33.2004 Ops/s	$-0.61\mathrm{\%}$
test_cql_speed[reduce-overhead-None]	23.7780ms	23.0637ms	43.3582 Ops/s	44.4837 Ops/s	$-2.53\mathrm{\%}$
test_cql_speed[reduce-overhead-backward]	32.5694ms	30.5164ms	32.7693 Ops/s	33.8220 Ops/s	$-3.11\mathrm{\%}$
test_a2c_speed[False-None]	8.5615ms	7.5233ms	132.9202 Ops/s	133.7561 Ops/s	$-0.62\mathrm{\%}$
test_a2c_speed[False-backward]	17.9105ms	15.3394ms	65.1915 Ops/s	66.2557 Ops/s	$-1.61\mathrm{\%}$
test_a2c_speed[True-None]	5.1508ms	4.7592ms	210.1181 Ops/s	212.7560 Ops/s	$-1.24\mathrm{\%}$
test_a2c_speed[True-backward]	13.0914ms	11.9995ms	83.3370 Ops/s	83.3428 Ops/s	$-0.01\mathrm{\%}$
test_a2c_speed[reduce-overhead-None]	5.6350ms	4.9280ms	202.9211 Ops/s	207.8093 Ops/s	$-2.35\mathrm{\%}$
test_a2c_speed[reduce-overhead-backward]	13.0068ms	11.7477ms	85.1231 Ops/s	86.4334 Ops/s	$-1.52\mathrm{\%}$
test_ppo_speed[False-None]	9.6899ms	8.0364ms	124.4341 Ops/s	123.7731 Ops/s	$+0.53\mathrm{\%}$
test_ppo_speed[False-backward]	17.7484ms	15.9066ms	62.8670 Ops/s	62.9406 Ops/s	$-0.12\mathrm{\%}$
test_ppo_speed[True-None]	7.1328ms	5.7656ms	173.4429 Ops/s	194.0420 Ops/s	$\mathbf{\text{-10.62\%}}$
test_ppo_speed[True-backward]	11.9916ms	11.2961ms	88.5262 Ops/s	84.9269 Ops/s	$+4.24\mathrm{\%}$
test_ppo_speed[reduce-overhead-None]	5.9620ms	5.1620ms	193.7223 Ops/s	179.2539 Ops/s	$\mathbf{\text{+8.07\%}}$
test_ppo_speed[reduce-overhead-backward]	11.7036ms	11.2490ms	88.8967 Ops/s	87.3002 Ops/s	$+1.83\mathrm{\%}$
test_reinforce_speed[False-None]	8.3953ms	7.0102ms	142.6487 Ops/s	148.8260 Ops/s	$-4.15\mathrm{\%}$
test_reinforce_speed[False-backward]	12.0495ms	10.4013ms	96.1423 Ops/s	99.3098 Ops/s	$-3.19\mathrm{\%}$
test_reinforce_speed[True-None]	4.6884ms	4.1495ms	240.9901 Ops/s	207.0679 Ops/s	$\mathbf{\text{+16.38\%}}$
test_reinforce_speed[True-backward]	11.8380ms	10.5748ms	94.5646 Ops/s	91.2760 Ops/s	$+3.60\mathrm{\%}$
test_reinforce_speed[reduce-overhead-None]	4.7285ms	4.0963ms	244.1226 Ops/s	228.1288 Ops/s	$\mathbf{\text{+7.01\%}}$
test_reinforce_speed[reduce-overhead-backward]	11.5724ms	10.1744ms	98.2863 Ops/s	92.3218 Ops/s	$\mathbf{\text{+6.46\%}}$
test_iql_speed[False-None]	36.7803ms	33.5764ms	29.7828 Ops/s	29.5987 Ops/s	$+0.62\mathrm{\%}$
test_iql_speed[False-backward]	50.2451ms	46.8744ms	21.3336 Ops/s	21.1803 Ops/s	$+0.72\mathrm{\%}$
test_iql_speed[True-None]	16.4094ms	16.0106ms	62.4586 Ops/s	62.0128 Ops/s	$+0.72\mathrm{\%}$
test_iql_speed[True-backward]	31.1124ms	28.4742ms	35.1195 Ops/s	35.3429 Ops/s	$-0.63\mathrm{\%}$
test_iql_speed[reduce-overhead-None]	18.6380ms	16.3372ms	61.2099 Ops/s	61.7919 Ops/s	$-0.94\mathrm{\%}$
test_iql_speed[reduce-overhead-backward]	29.1637ms	27.5480ms	36.3003 Ops/s	36.1138 Ops/s	$+0.52\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	7.3733ms	4.9269ms	202.9686 Ops/s	207.6011 Ops/s	$-2.23\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.9836ms	0.5570ms	1.7953 KOps/s	1.7710 KOps/s	$+1.37\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.9460ms	0.5233ms	1.9109 KOps/s	1.8952 KOps/s	$+0.83\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	5.0155ms	4.6641ms	214.4049 Ops/s	216.9663 Ops/s	$-1.18\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	1.3914ms	0.5378ms	1.8595 KOps/s	482.7004 Ops/s	$\mathbf{\text{+285.23\%}}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.9270ms	0.5294ms	1.8889 KOps/s	1.9644 KOps/s	$-3.85\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000]	2.5045ms	1.7351ms	576.3323 Ops/s	583.7323 Ops/s	$-1.27\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000]	2.3621ms	1.6437ms	608.3741 Ops/s	611.2167 Ops/s	$-0.47\mathrm{\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	5.4855ms	4.7902ms	208.7583 Ops/s	200.8786 Ops/s	$+3.92\mathrm{\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	1.0586ms	0.6908ms	1.4475 KOps/s	1.4306 KOps/s	$+1.18\mathrm{\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.9555ms	0.6570ms	1.5220 KOps/s	1.4953 KOps/s	$+1.79\mathrm{\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	4.9602ms	4.6601ms	214.5872 Ops/s	213.2054 Ops/s	$+0.65\mathrm{\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	1.1656ms	0.5543ms	1.8041 KOps/s	1.8537 KOps/s	$-2.67\mathrm{\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	1.0901ms	0.5225ms	1.9138 KOps/s	1.9381 KOps/s	$-1.25\mathrm{\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	7.4048ms	4.6816ms	213.6030 Ops/s	214.2776 Ops/s	$-0.31\mathrm{\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	1.3742ms	0.5368ms	1.8628 KOps/s	1.8382 KOps/s	$+1.34\mathrm{\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.7946ms	0.5244ms	1.9070 KOps/s	1.8779 KOps/s	$+1.55\mathrm{\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	7.7662ms	4.8384ms	206.6804 Ops/s	189.6199 Ops/s	$\mathbf{\text{+9.00\%}}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	1.2221ms	0.6924ms	1.4442 KOps/s	1.4325 KOps/s	$+0.82\mathrm{\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.9830ms	0.6705ms	1.4914 KOps/s	1.4958 KOps/s	$-0.29\mathrm{\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	5.5986ms	4.3475ms	230.0191 Ops/s	219.3798 Ops/s	$+4.85\mathrm{\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	8.3950ms	2.4334ms	410.9404 Ops/s	420.3792 Ops/s	$-2.25\mathrm{\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	2.8752ms	1.3938ms	717.4566 Ops/s	634.5951 Ops/s	$\mathbf{\text{+13.06\%}}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	7.0230ms	4.4219ms	226.1447 Ops/s	230.7812 Ops/s	$-2.01\mathrm{\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	5.0524ms	2.3798ms	420.2002 Ops/s	419.0002 Ops/s	$+0.29\mathrm{\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	7.3562ms	1.4742ms	678.3183 Ops/s	709.1717 Ops/s	$-4.35\mathrm{\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	0.8304s	21.5500ms	46.4037 Ops/s	233.1147 Ops/s	$\mathbf{\text{-80.09\%}}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	7.4786ms	2.4866ms	402.1598 Ops/s	390.3257 Ops/s	$+3.03\mathrm{\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	6.0284ms	1.5718ms	636.2051 Ops/s	572.4713 Ops/s	$\mathbf{\text{+11.13\%}}$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True]	59.9472ms	49.9522ms	20.0191 Ops/s	19.5051 Ops/s	$+2.64\mathrm{\%}$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False]	15.7843ms	14.4355ms	69.2737 Ops/s	68.9541 Ops/s	$+0.46\mathrm{\%}$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True]	64.1168ms	51.6309ms	19.3683 Ops/s	19.5445 Ops/s	$-0.90\mathrm{\%}$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False]	16.8888ms	14.6741ms	68.1474 Ops/s	66.9540 Ops/s	$+1.78\mathrm{\%}$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True]	84.0999ms	51.2867ms	19.4982 Ops/s	19.4726 Ops/s	$+0.13\mathrm{\%}$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False]	17.8907ms	16.0842ms	62.1728 Ops/s	61.7267 Ops/s	$+0.72\mathrm{\%}$