v0 param server (using collectives not object store)

[mikaylagawarecki]

Stack from ghstack (oldest at bottom):

• -> v0 param server (using collectives not object store) #2865

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2865

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 13 New Failures, 10 Unrelated Failures

As of commit 0a265a9 with merge base 7df8317 ():

NEW FAILURES - The following jobs have failed:

• Build Windows Wheels / pytorch/rl / build-wheel-py3_9-cuda11_8 (gh)
• Build Windows Wheels / pytorch/rl / build-wheel-py3_9-cuda12_6 (gh)
• Continuous Benchmark (PR) / CPU Pytest benchmark (gh)
  Process completed with exit code 1.
• Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
  Process completed with exit code 1.
• Habitat Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t fbf3397525bcd40f315b721d6895a606c00f91a60fad216d40d92743ab178b08 /exec failed with exit code 1
• Lint / python-source-and-configs / linux-job (gh)
  torchrl/collectors/vllm_weight_update.py:22:1: E302 expected 2 blank lines, found 1
• LLM Tests on Linux / unittests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 8ea5b88b9d9197f1fc790d50e4cc6e308763b7647f6ee99ca1437ed16ff2b5a4 /exec failed with exit code 1
• SOTA Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 1e9ae3098c71eebe5d840e3c0eb51cb42171b00edb81ecd89bddbc8fcd10eb0c /exec failed with exit code 1
• Unit-tests on Linux / tests-cpu (3.10) / linux-job (gh)
  test/test_tensordictmodules.py::TestTDSequence::test_stateful_probabilistic[True-bounded-False]
• Unit-tests on Linux / tests-cpu (3.12) / linux-job (gh)
  test/test_tensordictmodules.py::TestTDSequence::test_stateful[True-bounded-False]
• Unit-tests on Linux / tests-cpu (3.9) / linux-job (gh)
  test/test_modules.py::TestMultiAgent::test_multiagent_mlp_lazy
• Unit-tests on Linux / tests-gpu (3.11, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 726112f0955e97bb0369fb611ef3928a66c210b6a16a8a9c38373a2e829d39a3 /exec failed with exit code 250
• Unit-tests on Linux / tests-olddeps (3.8, 11.6) / linux-job (gh)
  RuntimeError: Command docker exec -t ee0c319d113631d0a5b697065ef9b4aad8960f5c96e3af0c92f2d90280dbd76b /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Build Windows Wheels / pytorch/rl / build-wheel-py3_9-cpu (gh) (trunk failure)
• Build Windows Wheels / pytorch/rl / build-wheel-py3_9-cuda12_8 (gh) (trunk failure)
  Process completed with exit code 1.
• Build Windows Wheels / pytorch/rl / upload / upload-wheel-py3_9-cpu (gh) (trunk failure)
• Build Windows Wheels / pytorch/rl / upload / upload-wheel-py3_9-cuda11_8 (gh) (trunk failure)
• Build Windows Wheels / pytorch/rl / upload / upload-wheel-py3_9-cuda12_6 (gh) (trunk failure)
  Unable to download artifact(s): Artifact not found for name: pytorch_rl__3.9_cu126_
• Build Windows Wheels / pytorch/rl / upload / upload-wheel-py3_9-cuda12_8 (gh) (trunk failure)
  Unable to download artifact(s): Artifact not found for name: pytorch_rl__3.9_cu128_
• Unit-tests on Linux / tests-cpu (3.11) / linux-job (gh) (trunk failure)
  test/test_tensordictmodules.py::TestTDSequence::test_stateful_probabilistic[True-bounded-False]
• Unit-tests on Linux / tests-optdeps (3.11, 12.8) / linux-job (gh) (trunk failure)
  test/test_tensordictmodules.py::TestTDSequence::test_stateful[True-bounded-False]
• Unit-tests on Linux / tests-stable-gpu (3.10, 11.8) / linux-job (gh) (trunk failure)
  test/test_transforms.py::TestTimer::test_transform_model
• Unit-tests on Windows / unittests-cpu (3.10, windows.4xlarge, cpu) / windows-job (gh) (trunk failure)
  test/test_transforms.py::TestTimer::test_transform_env

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[mikaylagawarecki]

@vmoens This is one part where I'm trying to call a method on the LLM object to init a process group with the vllm workers, the second part is below on L293

In this case
SyncDataCollector is remote and has an attribute .policy

.policy is the ModuleDict object returned by from_vllm, and the actual llm instance is in the generate key (the LLM instance is local to the SyncDataCollector)

How can I have a handle to the LLM instance within the SyncDataCollector to call remote methods on it without the hacky call_policy_method implementation below?

[mikaylagawarecki]

@vmoens I keep finding that I want to get info off vllm directly :/, what would you do here?

Should this vLLMRemoteWeightUpdaterBase be aware of all the vllm engines and tp size of each owned by its parent RayCollector in its __init__, I already needed to pass separate master_address and master_port I guess