vk::PhysicalDevice::createDeviceUnique: ErrorInitializationFailed when running multiple tasks #28
Description
hello! I'm working on the basis of mani-skill. When I run one task, everying is ok, but when I'm running 2 tasks(e.g. I run
python -m tools.run_rl configs/sac/sac_mani_skill_state_1M_train.py --seed=0 --cfg-options "env_cfg.env_name=OpenCabinetDrawer_1000_link_0-v0" --num-gpus 1 --clean-up
python -m tools.run_rl configs/sac/sac_mani_skill_state_1M_train.py --seed=0 --cfg-options "env_cfg.env_name=OpenCabinetDrawer_1000-v0" --num-gpus 1 --clean-up
python -m tools.run_rl configs/sac/sac_mani_skill_state_1M_train.py --seed=0 --cfg-options "env_cfg.env_name=OpenCabinetDrawer-v0" --num-gpus 1 --clean-upat the same time on different gpus). The last instruction will result in an error
OpenCabinetDrawer_1000-v0 - INFO - 2022-04-22 20:46:49 - Set random seed to 0
Traceback (most recent call last):
File "/home/ruichentie/miniconda3/envs/mani_skill/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/ruichentie/miniconda3/envs/mani_skill/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/ruichentie/mani/ManiSkill-Learn/tools/run_rl.py", line 290, in <module>
main()
File "/home/ruichentie/mani/ManiSkill-Learn/tools/run_rl.py", line 271, in main
obs_shape, action_shape, action_space = get_env_info(cfg.env_cfg)
File "/home/ruichentie/mani/ManiSkill-Learn/mani_skill_learn/env/env_utils.py", line 46, in get_env_info
env = build_env(env_cfg)
File "/home/ruichentie/mani/ManiSkill-Learn/mani_skill_learn/env/env_utils.py", line 95, in build_env
return build_from_cfg(cfg, ENVS, default_args)
File "/home/ruichentie/mani/ManiSkill-Learn/mani_skill_learn/utils/meta/registry.py", line 132, in build_from_cfg
return obj_cls(**args)
File "/home/ruichentie/mani/ManiSkill-Learn/mani_skill_learn/env/env_utils.py", line 60, in make_gym_env
env = gym.make(env_name, **tmp_kwargs)
File "/home/ruichentie/miniconda3/envs/mani_skill/lib/python3.8/site-packages/gym/envs/registration.py", line 145, in make
return registry.make(id, **kwargs)
File "/home/ruichentie/miniconda3/envs/mani_skill/lib/python3.8/site-packages/gym/envs/registration.py", line 90, in make
env = spec.make(**kwargs)
File "/home/ruichentie/miniconda3/envs/mani_skill/lib/python3.8/site-packages/gym/envs/registration.py", line 59, in make
cls = load(self.entry_point)
File "/home/ruichentie/miniconda3/envs/mani_skill/lib/python3.8/site-packages/gym/envs/registration.py", line 18, in load
mod = importlib.import_module(mod_name)
File "/home/ruichentie/miniconda3/envs/mani_skill/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 848, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/ruichentie/ManiSkill/mani_skill/env/open_cabinet_door_drawer.py", line 5, in <module>
from mani_skill.env.base_env import BaseEnv
File "/home/ruichentie/ManiSkill/mani_skill/env/base_env.py", line 59, in <module>
_renderer = sapien.VulkanRenderer(default_mipmap_levels=1)
RuntimeError: vk::PhysicalDevice::createDeviceUnique: ErrorInitializationFailedI'm sure the gpu driver is installed properly ,nvidia_icd.json exists and environment variable set.
Do you have any idea about this? Thank you!