XLoRA embed_scale Support #2830

[sambhavnoobcoder]

Summary

This PR adds embed_scale support to XLoraEmbeddingLayer, ensuring X-LoRA correctly handles models with scaled embeddings (e.g., Gemma3TextScaledWordEmbedding). This is a companion PR to the LoRA/TrainableTokens embed_scale fix, following the same approach but adapted for X-LoRA's architecture.

Changes

Code

• src/peft/tuners/xlora/layer.py: Modified XLoraEmbeddingLayer.forward() to apply embed_scale to X-LoRA adapter contributions
  • Retrieves embed_scale via self.target._get_embed_scale() (inherits from BaseTunerLayer)
  • Applies scaling to adapter output before adding to result
  • Follows same pattern as LoRA implementation with comments explaining the approach

Tests

• tests/test_xlora.py: Added test_xlora_embed_scale_is_applied
  • Uses real Gemma3 model with multiple LoRA adapters (X-LoRA's primary use case)
  • Varies embed_scale to extreme values (10000.0, 0) to verify scaling is applied correctly
  • Follows same test pattern as LoRA/TrainableTokens tests

Key Differences from LoRA Implementation

Implementation:

• Uses self.target._get_embed_scale() instead of self._get_embed_scale() because X-LoRA wraps a LoRA Embedding layer (via self.target)
• No mixed batch forward variant needed (X-LoRA handles multiple adapters through its scalings parameter, not adapter_names)

Testing:

• Creates Gemma3-compatible LoRA adapters on-the-fly (X-LoRA requires saved LoRA adapters as input)
• Sets use_cache=False (X-LoRA requirement)
• Converts adapter list to dict format (X-LoRA config requirement)

Test Results

• ✅ New test passes (2.69s)
• ✅ All 16 existing X-LoRA tests pass (no regression)
• ✅ make style passed

Fixes : #2830
cc: @BenjaminBossan