Reproducing the result of cifar100

[Ldoun]

Hello, first and foremost, I would like to express my gratitude for the work you have done. Thank you.
I was trying to reproduce the result of cifar100 in the appendix using retrain and FT. But for some reason, I couldn't reproduce the result of unlearning by FT.

I tested cifar100 under 6 different independent trials. As you can see FT with class-wise forgetting has quite a low score of forget efficacy on Sparse(pruned model)/Dense(unpruned) class-wise forgetting and Sparse random data forgetting.

The hyperparameters of these experiments are the same as ciafar10.

for seed in 1 2 3 4 5 6
do
    python -u main_forget.py --save_dir ${save_dir}/random/${seed}_retrain --mask ${base_dir}/${seed}/base/0model_SA_best.pth.tar --unlearn retrain --unlearn_epochs 160 --unlearn_lr 0.1 --dataset $data --class_to_replace -1 --num_indexes_to_replace 4500 --seed $seed
    python -u main_forget.py --save_dir ${save_dir}/random/${seed}_FT_only --mask ${base_dir}/${seed}/base/0model_SA_best.pth.tar --unlearn FT --unlearn_lr 0.04 --unlearn_epochs 10 --dataset $data --class_to_replace -1 --num_indexes_to_replace 4500 --seed $seed
    
    python -u main_forget.py --save_dir ${save_dir}/class/${seed}_retrain --mask ${base_dir}/${seed}/base/0model_SA_best.pth.tar --unlearn retrain --unlearn_epochs 160 --unlearn_lr 0.1 --dataset $data --seed $seed
    python -u main_forget.py --save_dir ${save_dir}/class/${seed}_FT_only --mask ${base_dir}/${seed}/base/0model_SA_best.pth.tar --unlearn FT --unlearn_lr 0.01 --unlearn_epochs 10 --dataset $data --seed $seed

    python -u main_forget.py --save_dir ${save_dir2}/random/${seed}_retrain --mask ${base_dir}/${seed}/base/1model_SA_best.pth.tar --unlearn retrain --unlearn_epochs 160 --unlearn_lr 0.1 --dataset $data --class_to_replace -1 --num_indexes_to_replace 4500 --seed $seed
    python -u main_forget.py --save_dir ${save_dir2}/random/${seed}_FT_only --mask ${base_dir}/${seed}/base/1model_SA_best.pth.tar --unlearn FT --unlearn_lr 0.04 --unlearn_epochs 10 --dataset $data --class_to_replace -1 --num_indexes_to_replace 4500 --seed $seed
    
    python -u main_forget.py --save_dir ${save_dir2}/class/${seed}_retrain --mask ${base_dir}/${seed}/base/1model_SA_best.pth.tar --unlearn retrain --unlearn_epochs 160 --unlearn_lr 0.1 --dataset $data --seed $seed
    python -u main_forget.py --save_dir ${save_dir2}/class/${seed}_FT_only --mask ${base_dir}/${seed}/base/1model_SA_best.pth.tar --unlearn FT --unlearn_lr 0.01 --unlearn_epochs 10 --dataset $data --seed $seed
done

#4
I'm aware that you've used different learning rates of cifar10 for class-wise forgetting(0.01) and random data forgetting(0.04). Is learning rates for cifar100 different?

Thank you.

[jinghanjia]

Thank you for reaching out. I'd like to clarify that the hyperparameters for the cifar100 dataset differ slightly from Cifar10. We've observed that a duration of only 10 epochs is insufficient for effective unlearning on cifar100 by FT. Consequently, we've adjusted the training to span 20 epochs. Additionally, we used the learning rate to 0.01 for both class and random forgetting methods on cifar100.

If you encounter any further issues or have more questions, please don't hesitate to get in touch.

[Ldoun]

Thank you for the new info.
We're planning additional experiments on svhn too. Does the hyperparameter for the svhn differs from cifar10?

Thank you.

[Ldoun]

I've tested using the suggested hyperparameters above. still, the result differs by a margin from Table A3.
As you can see in the image below the score of MIA efficacy for Dense network random data forgetting and Sparse network class-wise data forgetting are too low compared to the Table A3 result
(1.13% vs 13.61111
,98.88% vs 83.703).

I've used the below script to create the base model for unlearning. Are there any hyperparameters specific for cifar100 when creating these models?

data=cifar100
base_dir=exp/${data}/new_seed_test
arch=resnet18
epochs=182
warmup=1

mkdir -p $base_dir

for seed in 1 2 3 4 5 6
do
    python -u main_imp.py --data ../data --dataset ${data} --arch $arch --prune_type rewind_lt --rewind_epoch 8 --save_dir exp --rate 0.95 --pruning_times 2 --warmup $warmup --epochs $epochs --save_dir ${base_dir}/${seed}/base --seed $seed
done

[jinghanjia]

Thank you for your diligence in reviewing our table. Upon re-examination, we've discovered an typo in the CIFAR-100 random forgetting results; the correct MIA is 11.13, not 1.13. We've also verified that for class-wise forgetting on CIFAR-100, 10 epochs with a learning rate of 0.01 suffices, and we used this setting in our latest version. These corrections have been made in the latest version of our paper on arXiv. The differences of your results with ours in the class-wise forgetting may because of its large variance.

Regarding SVHN, we consistently use 10 epochs for both class-wise and random forgetting, with learning rates set to 0.01 for class-wise and 0.04 for random forgetting.

If you encounter any issues with the updated results, please do not hesitate to reach out.

[Ldoun]

Thank you for your kind reply!
In your comment "We've also verified that for class-wise forgetting on CIFAR-100, 10 epochs with a learning rate of 0.01 suffices" is this the same for the random-data forgetting in CIFAR-100? (unlearn_epochs=10 and unlera_lr=0.01 for random data forgetting)

Thank you.