implement activation offloading and opt_in_bwd in knowledge_distillation recipes #1959
Labels
community help wanted
We would love the community's help completing this issue
good first issue
Good for newcomers
Current knowledge distillation recipes don't have support for activation offloading and opt_in_bwd.
The implementation should be similar to the one in other recipes, like full_finetuning_distributed.
After enabling it in the recipe, it should also be enabled in the configs related to KD.
PRs with reference implementation:
activation offloading: #1847
opt_in_bwd implementation: #1833
KD recipes:
https://github.com/pytorch/torchtune/blob/main/recipes/knowledge_distillation_single_device.py
https://github.com/pytorch/torchtune/blob/main/recipes/knowledge_distillation_distributed.py
The text was updated successfully, but these errors were encountered: