[Feature] Added RGBD diffusion policy implementation as well as Draw Triangle and Draw SVG task #643
Conversation
| MP_SOLUTIONS = {"DrawTriangle-v1": solveDrawTriangle} | ||
|
|
||
|
|
||
| def parse_args(args=None): |
There was a problem hiding this comment.
thanks for adding this file.
we might want to consider having a more unified motion planning interface for all robots with some base classes and run scripts etc., this should be fine for now though!
mani_skill/utils/wrappers/flatten.py
Outdated
| @@ -50,7 +51,12 @@ def observation(self, observation: Dict): | |||
| if self.include_rgb and not self.include_depth: | |||
| ret["rgb"] = images | |||
| elif self.include_rgb and self.include_depth: | |||
| ret["rgbd"] = images | |||
| # todo: might not be the best way or the right place to perform this | |||
There was a problem hiding this comment.
remove the todo. I think this is fine. I normally myself wouldn't merge rgb and depth but this is just for compatibility with a lot of old code from MS2 that assumed "rgbd" as a key.
| @@ -0,0 +1,330 @@ | |||
| import math | |||
There was a problem hiding this comment.
Is it possible to make the outline 1. not transparent and 2. just be below the dots/ink? instead of over it. Looks a bit strange here
| to create their own drawing type tasks. | ||
| """ | ||
|
|
||
| MAX_DOTS = 1010 |
There was a problem hiding this comment.
reduce this down significantly, maybe to 300. and max episode steps should be down to 300
| vis=vis, | ||
| base_pose=env.unwrapped.agent.robot.pose, | ||
| visualize_target_grasp_pose=vis, | ||
| print_env_info=False, |
There was a problem hiding this comment.
can you lower the max allowed joint velocities? Right now the result drawings are a bit too fast (dots are too spread out).
maybe lets tune it so that the typical solve time is about the 200-250 steps mark
|
|
||
|
|
||
| def make_eval_envs(env_id, num_envs: int, sim_backend: str, env_kwargs: dict, other_kwargs: dict, video_dir: Optional[str] = None, wrappers: list[gym.Wrapper] = []): | ||
| class DictFrameStack(GymFrameStack): |
There was a problem hiding this comment.
can this not just inherit the maniskill frame stack wrapper?
I purposely didn't use the original framestack wrapper from gymnasium since it was not properly GPU parallelized.
Draw svg
arnavg115
changed the title
There was a problem hiding this comment.
Add a new baselines.sh file and update README to point to that file. Write a script to train for each of the tasks you tested, using the datasets generated by the script here: https://github.com/haosulab/ManiSkill/blob/main/scripts/data_generation/replay_for_il_baselines.sh
Also make sure to add a draw triangle replay trajectory script to the file above.
|
all the changes nearly look ready. I will make a few small edits on baselines.sh file to be closer to what we did to act baselines. The diffusion policy one i did before for state based results is kind of hard to read, it was meant to experiment with DP training on motion planning or RL generated datasets (but doing that needs tuning other hyparameters anyway). Will also merge in #870 which is blocking me from easily testing all the baselines and uploading results to wandb since things take forever for the physx_cpu evals. |
StoneT2000
changed the title
|
Merging to a separate branch for now so i make the final minor changes and merge. nice work! |
* [Feature] Added RGBD diffusion policy implementation as well as Draw Triangle and Draw SVG task (#643) * Added draw triangle with success condition * parallelized progress * fixed triangle rotation issues * clean up and format * rgbd diffusion policy progress * diff policy rgbd cpu fixes * minor diff policy fixes and finished draw triangle parallelization * Added depth arg to diff pol rgbd + formatting * Removed unused code * Made requested fixes, made bugfix to frame stack wrapper * Edited make_env and frame_stack * Added state obs to draw triangle * Update draw_triangle max steps * Added DrawTriangle Docs * Fixed naming * draw svg progress * fixed most issues and parallelized draw svg * Update draw_svg.py * added success condition and state based obs * Added discontinuous paths for draw svg * formatting, discontinuous state * Updated run.py * fixed state obs error for drawing envs * Changed drawsvg imports * Update draw_svg.py * Update draw_svg.py * Small bugfix * Update draw_svg.py * Update draw_svg.py * Update draw_triangle.py * Bugfixes, speed progress * success condition speed up * small fix * Updated draw_triangle * drawing env gpu bugfixes * diff poll rgbd fixes * minor changes * fix for wandb logging * autobuild docs for DrawTriangle and SVG * Update utils.py to add PushCube * adjustments for rgb * added initial pose for drawing envs * Update draw_triangle.py * adjusted training arguments and shell scripts * StackCube fix * work * w * w * w * w * w * w * w * update docs * update docs * work * w * Update utils.py * remove dead code * Update flatten.py * Update flatten.py * fix flatten wrapper * possibly better docs * simplify cli args and remove assumption of rgb existing * Update README.md --------- Co-authored-by: Arnav G. <[email protected]>
DrawTriangle, DrawSVG: #638
Diff policy RGBD: #616