Skip to content

liuff19/DreamReward

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

32 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

DreamReward: Text-to-3D Generation with Human Preference

Junliang Ye1,2*, Fangfu Liu1*, Qixiu Li1, Zhengyi Wang1,2,
Yikai Wang1, Xinzhou Wang1,2, Yueqi Duan1†, Jun Zhu1,2†,
*Equal Contribution.
Corresponding authors.
1Tsinghua University, 2ShengShu,

              

pipeline method

Release

  • [2025/3/19] 🔥🔥We released the DreamFL code of DreamReward.
  • [2025/3/19] 🔥🔥We open-source two versions of Reward3D weights.

Contents

Ablation

We open-source two versions of Reward3D: Reward3D_Scorer, corresponding to the implementation presented in our paper, and Reward3D_CrossViewFusion, an extended version leveraging self-attention to enhance multi-view feature integration. Furthermore, we release two hyperparameter configurations of the DreamFL algorithm: DreamReward_guidance1, offering more conservative guidance, and DreamReward_guidance2, providing more aggressive guidance. Additionally, we provide ablation for further analysis and reference.

Installation

Our environment has been tested on Ubuntu 22, CUDA 11.8 with A100, A800.

  1. Clone our repo and create conda environment
git clone https://github.com/liuff19/DreamReward.git && cd DreamReward
conda create -n dreamreward python=3.9
conda activate dreamreward
pip install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 --index-url https://download.pytorch.org/whl/cu118
pip install xformers==0.0.23.post1
pip install ninja
git clone https://github.com/bytedance/MVDream extern/MVDream
pip install -e extern/MVDream 
pip install -r requirements.txt
pip install fairscale
  1. Install the pretrained Reward3D weight
pip install -U "huggingface_hub[cli]"
huggingface-cli login
huggingface-cli download yejunliang23/Reward3D --local-dir checkpoints/

Usage

Command line inference

# inference for CrossViewFusion1
python launch.py --config "configs/dreamreward1.yaml" --train --gpu 0 system.guidance.alg_type="Reward3D_CrossViewFusion" system.guidance_type="DreamReward-guidance1" name="CrossViewFusion1" system.prompt_processor.prompt="A delicate porcelain teacup, painted with intricate flowers, rests on a saucer" system.guidance.reward_model_path="checkpoints/Reward3D_CrossViewFusion.pt"
# or you can
bash scripts/cvf1.sh

# inference for CrossViewFusion2
python launch.py --config "configs/dreamreward2.yaml" --train --gpu 0 system.guidance.alg_type="Reward3D_CrossViewFusion" system.guidance_type="DreamReward-guidance1" name="CrossViewFusion2" system.prompt_processor.prompt="A delicate porcelain teacup, painted with intricate flowers, rests on a saucer" system.guidance.reward_model_path="checkpoints/Reward3D_CrossViewFusion.pt"
# or you can
bash scripts/cvf2.sh

# inference for Scorer1
python launch.py --config "configs/dreamreward1.yaml" --train --gpu 0 system.guidance.alg_type="Reward3D_Scorer" system.guidance_type="DreamReward-guidance1" name="Scorer1" system.prompt_processor.prompt="A delicate porcelain teacup, painted with intricate flowers, rests on a saucer" system.guidance.reward_model_path="checkpoints/Reward3D_Scorer.pt"
# or you can
bash scripts/scorer1.sh

# inference for Scorer2
python launch.py --config "configs/dreamreward2.yaml" --train --gpu 0 system.guidance.alg_type="Reward3D_Scorer" system.guidance_type="DreamReward-guidance1" name="Scorer2" system.prompt_processor.prompt="A delicate porcelain teacup, painted with intricate flowers, rests on a saucer" system.guidance.reward_model_path="checkpoints/Reward3D_Scorer.pt"
# or you can
bash scripts/scorer2.sh

Todo

  • Release the 3DReward-training code.

Acknowledgement

Our code is based on these wonderful repos:

Also we invite you to explore our latest work DeepMesh about applying direct preference optimization to 3D-mesh generation.

BibTeX

@inproceedings{ye2024dreamreward,
  title={Dreamreward: Text-to-3d generation with human preference},
  author={Ye, Junliang and Liu, Fangfu and Li, Qixiu and Wang, Zhengyi and Wang, Yikai and Wang, Xinzhou and Duan, Yueqi and Zhu, Jun},
  booktitle={European Conference on Computer Vision},
  pages={259--276},
  year={2024},
  organization={Springer}
}

About

[ECCV 2024] DreamReward: Text-to-3D Generation with Human Preference

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published