[Algorithm] GRPO scripts

[vmoens]

Stack from ghstack (oldest at bottom):

• -> [Algorithm] GRPO scripts #2970

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2970

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 11 New Failures, 1 Pending, 4 Unrelated Failures

As of commit fa89fc0 with merge base 023c965 ():

NEW FAILURES - The following jobs have failed:

• Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
  Process completed with exit code 1.
• Habitat Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 29dca799d00a8fd9ff8d95f8f917f56fdfe85985221ee2fcede5330bdfe43e90 /exec failed with exit code 1
• Lint / python-source-and-configs / linux-job (gh)
  RuntimeError: Command docker exec -t 7d2ce184e4f09df9bcdd011a06b6019c5b2777f8516229bf330217b008bae150 /exec failed with exit code 1
• SOTA Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t f6827b1df1ba57e14b8b05c92fd8f85b8a300386f5d5a1e57a31411accac7af7 /exec failed with exit code 1
• Unit-tests on Linux / tests-cpu (3.10) / linux-job (gh)
  test/test_env.py::TestAsyncEnvPool::test_async_transformed[threading]
• Unit-tests on Linux / tests-cpu (3.11) / linux-job (gh)
  test/test_env.py::TestAsyncEnvPool::test_async_transformed[threading]
• Unit-tests on Linux / tests-cpu (3.12) / linux-job (gh)
  test/test_env.py::TestAsyncEnvPool::test_async_transformed[threading]
• Unit-tests on Linux / tests-cpu (3.9) / linux-job (gh)
  test/test_env.py::TestAsyncEnvPool::test_async_transformed[threading]
• Unit-tests on Linux / tests-gpu (3.11, 12.8) / linux-job (gh)
  test/test_env.py::TestAsyncEnvPool::test_async_transformed[threading]
• Unit-tests on Linux / tests-optdeps (3.11, 12.8) / linux-job (gh)
  test/test_env.py::TestAsyncEnvPool::test_async_transformed[threading]
• Unit-tests on Linux / tests-stable-gpu (3.10, 11.8) / linux-job (gh)
  test/test_env.py::TestAsyncEnvPool::test_async_transformed[threading]

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Libs Tests on Linux / unittests-gym (3.9, 12.8) / linux-job (gh) (trunk failure)
  test/test_libs.py::TestGym::test_gym_fake_td[True-False-3-HalfCheetah-v2]
• LLM Tests on Linux / unittests (3.9, 12.8) / linux-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• Unit-tests on Linux / tests-olddeps (3.9, 11.6) / linux-job (gh) (trunk failure)
  test/test_transforms.py::TestActionDiscretizer::test_transform_env[cheetah-SamplingStrategy.RANDOM-False-True]
• Unit-tests on Windows / unittests-cpu (3.10, windows.4xlarge, cpu) / windows-job (gh) (trunk failure)
  Process completed with exit code 2.

This comment was automatically generated by Dr. CI and updates every 15 minutes.