Hotswap allow different alpha scalings and ranks (#2177)

Hotswapping of LoRA adapters is already implemented, but when alpha
scalings or ranks differ, this triggers recompilation of the model is
compiled, which is inefficient. Users can now call
prepare_model_for_compiled_hotswap to prevent recompilation in many
cases (see the doc update for caveats).

Author: BenjaminBossan

docs/source/package_reference/hotswap.md

@@ -8,6 +8,8 @@ The idea of hotswapping an adapter is the following: We can already load multipl
 
 In general, this should be faster than deleting one adapter and loading the adapter in its place, which would be the how to achieve the same final outcome without hotswapping. Another advantage of hotswapping is that it prevents re-compilation in case the PEFT model is already compiled using `torch.compile`. This can save quite a lot of time.
 
+## Example without `torch.compile`
+
 ```python
 import torch
 from transformers import AutoModelForCausalLM
@@ -21,7 +23,6 @@ model = AutoModelForCausalLM.from_pretrained(model_id).to(device)
 
 # load lora 0
 model = PeftModel.from_pretrained(model, <path-adapter-0>)
-model = torch.compile(model)  # optionally compile the model
 with torch.inference_mode():
     output_adapter_0 = model(inputs)
 
@@ -31,12 +32,42 @@ with torch.inference_mode():
     output_adapter_1 = model(inputs).logits
 ```
 
+## Example with `torch.compile`
+
+```python
+import torch
+from transformers import AutoModelForCausalLM
+from peft import PeftModel
+from peft.utils.hotswap import hotswap_adapter, prepare_model_for_compiled_hotswap
+
+model_id = ...
+inputs = ...
+device = ...
+max_rank = ...  # maximum rank among all LoRA adapters that will be used
+model = AutoModelForCausalLM.from_pretrained(model_id).to(device)
+
+# load lora 0
+model = PeftModel.from_pretrained(model, <path-adapter-0>)
+# Prepare the model to allow hotswapping even if ranks/scalings of 2nd adapter differ.
+# You can skip this step if all ranks and scalings are identical.
+prepare_model_for_compiled_hotswap(model, target_rank=max_rank)
+model = torch.compile(model)
+with torch.inference_mode():
+    output_adapter_0 = model(inputs)
+
+# replace the "default" lora adapter with the new one
+hotswap_adapter(model, <path-adapter-1>, adapter_name="default", torch_device=device)
+with torch.inference_mode():
+    output_adapter_1 = model(inputs).logits
+```
+
+## Caveats
+
 Hotswapping works with transformers models and diffusers models. However, there are some caveats:
 
-- It only works for the same PEFT method, so no swapping LoRA and LoHa, for example.
 - Right now, only LoRA is properly supported.
-- The adapters must be compatible (e.g. same LoRA alpha, same target modules).
-- If you use `torch.compile` and want to avoid recompilation, the LoRA rank must be the same.
+- It only works for the same PEFT method, so no swapping LoRA and LoHa, for example.
+- The adapter that is being swapped in must target the same layers as the previous adapter or a subset of those layers. It cannot target new layers. Therefore, if possible, start with the adapter that targets most layers.
 
 [[autodoc]] utils.hotswap.hotswap_adapter
     - all