-
Notifications
You must be signed in to change notification settings - Fork 4k
Open
Description
When I use 4 A100 40G to train the alpaca, I encountered an oom error during training.
this is my training arguments:
#!/bin/bash
module load compilers/cuda/11.8 compilers/gcc/9.3.0 cudnn/8.4.0.27_cuda11.x anaconda
source activate torch_new
export PYTHONUNBUFFERED=1
torchrun --nproc_per_node=4 --master_port=11223 train.py \
--model_name_or_path llama2 \
--data_path alpaca_data.json \
--output_dir alpaca_llama2 \
--num_train_epochs 3 \
--per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 \
--gradient_accumulation_steps 1 \
--evaluation_strategy "no" \
--save_strategy "steps" \
--save_steps 1000 \
--save_total_limit 1 \
--learning_rate 2e-5 \
--weight_decay 0. \
--warmup_ratio 0.03 \
--lr_scheduler_type "cosine" \
--logging_steps 1 \
--bf16 true\
--fsdp "full_shard auto_wrap" \
--fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer'
and this is the error:
22%|██▏ | 8402/39003 [10:12:57<36:10:57, 4.26s/it]Traceback (most recent call last):
File "/home/bingxing2/home/scx6203/luckychao/stanford_alpaca/train.py", line 220, in <module>
train()
'''
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 208.00 MiB. GPU 1 has a total capacty of 39.41 GiB of which 183.50 MiB is free. Including non-PyTorch memory, this process has 39.23 GiB memory in use. Of the allocated memory 37.37 GiB is allocated by PyTorch, and 539.20 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
anyone know the reason? :( thanks again!!
Metadata
Metadata
Assignees
Labels
No labels