Orca is a state-of-the-art language model that pushes the boundaries of natural language understanding and generation. Designed for high performance and efficiency, Orca is built upon advanced techniques that make it a strong contender against the likes of OpenAI's GPT-4 and PALM.
Get started:
- Clone the repository and install the required packages.
git clone https://github.com/kyegomez/Orca
cd Orca
pip3 install -r requirements.txt
cd Orca
python3 training_distributed.py
First:
Accelerate Config
Enable Deepspeed 3:
Accelerate launch train_distributed_accelerate.py