[feature] Load saved model to fine tune

[supreme-gg-gg]

User Story

Instead of always picking from a base model from hf or unsloth, use an already-tuned model. This sounds weird but in some use case you might want to first do instruction tuning with SFT for domain adaptation, then do DPO for preference alignment, both requires different dataset and different setup. This is not possible now since you cannot load a saved model to fine tune.

Alternative

We can also implement trainers like ORPO that does instruction + preference in one step + doesn't need reference model
https://huggingface.co/docs/trl/main/en/orpo_trainer

Acceptance Criteria

Find a use case for this and test it and submit a performance improvement validation

[supreme-gg-gg]

This might be referring to using an existing LoRA with DPO's reference model: https://huggingface.co/docs/trl/main/en/dpo_trainer#reference-model-considerations-with-peft

[supreme-gg-gg]

This from the recipe for alignment on Gemma 7B is exactly the use case that we expect here: https://github.com/huggingface/alignment-handbook/tree/main/recipes/zephyr-7b-gemma

I believe this is an extremely common use case after looking over a lot of industry application examples for reference.

[supreme-gg-gg]

Will be grinding this, the goal is:

1. the user can pick an existing job ID and "start training from this job ID"
2. The backend loads the model either adapter OR full fine-tune save
3. The only caveat is that we don't support multi LoRA right now (too much work), so you only get to train existing LoRA again Or keep training a full fine-tune (i.e. you cannot change from lora to full fine tune and vice versa, also can't add lora)
4. everything else stays the same, no code change needed, we create a new job so no conflict

EDIT:

I think we're just being too optimistic lmao:

ValueError: You cannot perform fine-tuning on purely quantized models. Please attach trainable adapters on top of the quantized model to correctly perform fine-tuning. Please see: https://huggingface.co/docs/transformers/peft for more details

Apparently trying to do QLoRA on a loaded adapter model won't work. You need to add another adapter? I will search more and post a question on PEFT / TRL tomorrow, this seems not to be that simple 👀

After this is done I will:

1. Make it possible to do multi-lora -- the user can select if they want to "add" adapter or "reuse" existing adapter
2. Make it possible to do lora after full fine-tune (not the other way around it doesn't make sense logically) -- easy to do
3. Update inference to support tehse two cases (the other cases are already support OOTB)

EDIT: ok at some poitn we might as well generate a graph showing what is tuned what is adapter lol since if you can just chain training, will be helpful to show it in a systematic generated schematics of the model "composition" after many rounds of training

[supreme-gg-gg]

For full fine tune note that we only use 16bit precision usually, can't do quantised, but here are related sources to read on:

• https://docs.pytorch.org/torchtune/0.3/tutorials/qat_finetune.html
• https://arxiv.org/abs/2310.07147

[supreme-gg-gg]

Hmm I'll put this in backlog first and go ahead with other features, noticeably shipping a ORPO trainer support instead of doing multi-stage because it solves the SFT + DPO use case for beginner complexity.

If the community wants this feature (I cannot find much example of how this will be useful), please upvote the issue / comment and it will be implemented.

I worry it will cause confusion while still not being useful at all, since it might get mixed up with regular multi lora and resuming from checkpoint, which isn't exactly what this does I believe.

I hope to gather more thoughts before implementing this, which shouldn't take long anyways