Does T5 actually utilize Encoder prefixes in PrefixTuning? past_key_values seems ignored by the Encoder.

[david900125]

System Info

• PEFT version: 0.18.0
• Transformers version: 4.57.3
• Model: T5 / Flan-T5

Description

I am exploring Prefix-Tuning on a Seq2Seq model (T5/Flan-T5) and I noticed a potential discrepancy between the paper's implementation and the current codebase regarding the Encoder's prefix.

According to the [Prefix-Tuning paper](https://arxiv.org/abs/2101.00190), prefixes should be prepended to both the Encoder and the Decoder for Encoder-Decoder architectures.

I configured my PEFT model with num_transformer_submodules=2 to generate prompts for both components.

However, tracing the code:

1. In peft/src/peft/peft_model.py (PeftModelForSeq2SeqLM):
   The code generates the prompt and passes it into kwargs["past_key_values"]:

if peft_config.peft_type == PeftType.PREFIX_TUNING:
    # overwrite past_kv in kwargs
    kwargs["past_key_values"] = self.get_prompt(batch_size)
    return self.base_model(
        input_ids=input_ids,
        ...
        **kwargs, # past_key_values is inside here
    )

2. In transformers/models/t5/modeling_t5.py (T5Model):
   The forward method separates the execution flow. It calls self.encoder without passing past_key_values.
   Reference Line: [modeling_t5.py#L991 (v5.0.0rc1)](https://github.com/huggingface/transformers/blob/v5.0.0rc1/src/transformers/models/t5/modeling_t5.py#L991)

encoder_outputs = self.encoder(
    input_ids=input_ids,
    attention_mask=attention_mask,
    inputs_embeds=inputs_embeds,
    output_attentions=output_attentions,
    output_hidden_states=output_hidden_states,
    return_dict=return_dict,

)

Question

It seems that even if num_transformer_submodules=2 is set, the past_key_values intended for the Encoder are effectively discarded before reaching the encoder's attention layers.

1. Is my understanding correct that currently, PrefixTuning on T5 effectively only works on the Decoder?
2. Is there an internal mechanism in transformers that automatically handles this injection that I might have missed?

Any clarification would be appreciated. Thanks!

[BenjaminBossan]

Thanks for opening this discussion. Your observation is correct, in encoder-decoder models, prefix tuning does not affect the encoder. The kv cache (aka past_key_values) is not applied to the encoder since it is not "causal", but prefix tuning relies on the kv cache to inject its embeddings.

I have asked internally if there is a different way we could adapt the encoder, but possibly there is no solution. I'll update once I have an answer.

[BenjaminBossan]

@david900125 After discussion with my colleague, who checked whether something can be done on transformers to enable past_key_values in the encoder part, she said there is no easy solution. Basically, we would have to go through each and every encoder-decoder model architecture and adjust it. This is too much work for a rather niche application, so we decided not to support it.

This means that encoder-decoder models don't work with prefix tuning as described in the paper. We should document this in PEFT, LMK if you want to create a PR for that, otherwise I'll take it. Note, though, that we can still use prefix tuning with encoder-decoder models and it can still be fine-tuned, it just doesn't conform to the paper description.

[david900125]

Hi @BenjaminBossan,

Thank you so much for the thorough investigation and confirmation. It is very helpful to have a definitive answer on this behavior.

I completely understand that refactoring the encoder-decoder architectures in transformers is a massive undertaking and out of scope. Clarifying this in the documentation is definitely the right move to prevent confusion for future users.

I will leave the documentation PR to you, as you likely have a standard format for such warnings.
Thanks again for your help!

[BenjaminBossan]

PR is in the works: #2989. Edit: merged.