Multi-modal projections for Llama-Adapter

[efraimdahl]

Just want to make sure I'm not missing anything. Currently the implementation of Llama-Adapter/AdaptionPrompt does not include any way we can add additional input that is added onto the prefix? In the Llama-Adapters paper (implementation here) they include a section on unlocking multimodal reasoning by projecting CLIP embeddings onto the prefix. Are there other PEFT-methods or a configuration of AdaptionPrompt that already implement this functionality?

[BenjaminBossan]

AFAICT, this is not possible. @yeoedward added the method to PEFT, maybe they can clarify.