We are releasing ZO2 v0.1.0, the initial public version of our high-throughput, memory-efficient zeroth-order fine-tuning framework for large language models.
Although still in early development (pre-1.0), this release contains all core features to reproduce the main results of our ZO2 paper.
🚀 Highlights
- Full-parameter fine-tuning with only 18GB GPU for models up to 175B
- CPU offloading for gradients and optimizer states
- Hugging Face Transformers compatibility
- Minimal codebase for educational and experimental use
🧪 Tested Models
- OPT-125M to OPT-175B
📄 Getting Started
git clone https://github.com/liangyuwang/zo2.git
cd zo2/
conda env create -f env.yml
conda activate zo2