prepare_inputs_for_generation missing when using newer transformers version

[Hrovatin]

System Info

The err depends on transformers version: it errors out using 4.56.0 but not 4.49.0 (did not test the versions in between)
Using pfts 0.17.1, linux, python 3.12.11

Who can help?

No response

Reproduction

from transformers import AutoModelForMaskedLM
from peft import LoraConfig, get_peft_model

model = AutoModelForMaskedLM.from_pretrained("Synthyra/ESMplusplus_small",trust_remote_code=True ) # I actually use local version of weights, but I would expect the same with the remote one
lora_target_modules = ["layernorm_qkv.1", "out_proj", "query", "key",
                  "value", "dense"]
lora_config = LoraConfig(
    r=32,
    lora_alpha=64,
    lora_dropout=0.01,
    bias="none", 
    target_modules=lora_target_modules,
    use_rslora=True, 
    task_type='CAUSAL_LM',
)
model = get_peft_model(model, lora_config)

Expected behavior

When doing peft on ESMplusplusForMaskedLM (from https://github.com/Synthyra/FastPLMs/blob/4b38cb271bdbb06eecb310218f4c3b33b5ad8c12/modeling_esm_plusplus.py#L846) I sometimes get error object has no attribute 'prepare_inputs_for_generation', depending on transformers version, as explained above.

[BenjaminBossan]

Thanks for reporting this error. Could you please paste the whole error stack trace you get?

[Hrovatin]

Sadly I did not save it before downgrading the env. But as far as I remember at least the bottom few lines related to this: #1827
And if I tried to access the problematic parameter from the model (before peft) it was only present in the earlier transformers version:

model.prepare_inputs_for_generation existed in transformers 4.49.0 but not 4.56.0

[Hrovatin]

Maybe related to this change (although it refers to a different version, but maybe there were some other changes in what is automatically inherited)? https://github.com/huggingface/transformers/blob/25b4a0d8aef515ed309c935607473da319d9291c/src/transformers/models/auto/auto_factory.py#L764

[BenjaminBossan]

Ah okay. As this model is not an official transformers implementation but a custom model instead, we can't really fix this issue easily. In the thread you mentioned, I proposed some solutions:

#1827 (comment)

Solution 1 and 2 would require PRs on the Synthyra repos to add these changes, I'm not sure if these approaches are viable. Did you try solution 3, i.e. removing the task_type argument?

Another solution that would not require a PR would be for you to wrap the whole model and expose the method yourself:

class MyEsmModel(nn.Module):
    def __init__(self, model_id="Synthyra/ESMplusplus_small"):
        super().__init__()
        self.base_model = AutoModelForMaskedLM.from_pretrained(model_id, trust_remote_code=True)

    def prepare_inputs_for_generation(self, *args, **kwargs):
        # not sure what to put here, maybe it can be left empty?
        pass

    def __getattr__(self, name: str):
        try:
            return super().__getattr__(name)  # defer to nn.Module's logic
        except AttributeError:
            if name == "base_model":
                raise
            return getattr(self.base_model, name)

I haven't tested it, but maybe you can give it a try.

[Hrovatin]

I now removed the task_type from Lora config. However, the same issue then occurs in reloading the model.
Do you think the below would be an ok workaround for loading assuming no special preparation of inputs is needed?

from peft import PeftConfig, PeftModelForCausalLM, AutoPeftModelForCausalLM

# Workaround over using AutoPeftModelForCausalLM.from_pretrained
# Which raises errors due to custom classes
base_model=AutoModelForMaskedLM.from_pretrained(
            PeftConfig.from_pretrained(pt_path).base_model_name_or_path,
            local_files_only=True, trust_remote_code=True,
    )
def prepare_inputs_for_generation(input_ids, **kwargs) :
        # Adapted from https://huggingface.co/transformers/v3.5.1/_modules/transformers/generation_utils.html

        return {"input_ids": input_ids}
base_model.prepare_inputs_for_generation = prepare_inputs_for_generation
model2=PeftModelForCausalLM.from_pretrained(
    base_model,
    pt_path,
    adapter_name='default',
    is_trainable = False,
    local_files_only=True, trust_remote_code=True
)

[BenjaminBossan]

However, the same issue then occurs in reloading the model.

Does the adapter_config.json of that checkpoint still contain the task type? If yes, you can just manually delete the entry or set it to null.

Do you think the below would be an ok workaround for loading assuming no special preparation of inputs is needed?

If we can make that assumption, this kind of monkey patching should be valid. Depending on how you serve the model (if at all), it might or might not be possible for you to apply this patch, I can't answer that for you.

[Hrovatin]

The adapter_config.json does not contain the task_type

The model is only being used locally.

[BenjaminBossan]

The adapter_config.json does not contain the task_type

Strange, not sure what else is going on then.

The model is only being used locally.

Then it should work as you suggested.

[Hrovatin]

Loading peft model without task type:

from peft import PeftConfig, PeftModelForCausalLM


# Workaround over using AutoPeftModelForCausalLM.from_pretrained
# Which raises errors due to custom classes
model2=PeftModelForCausalLM.from_pretrained(
    AutoModelForMaskedLM.from_pretrained(
            PeftConfig.from_pretrained(pt_path).base_model_name_or_path,
            local_files_only=True, trust_remote_code=True,
    ),
    pt_path,
    adapter_name='default',
    is_trainable = False,
    local_files_only=True, trust_remote_code=True
)

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
File ~/miniforge3/envs/protein/lib/python3.10/site-packages/peft/tuners/lora/model.py:382, in LoraModel.__getattr__(self, name)
    381 try:
--> 382     return super().__getattr__(name)  # defer to nn.Module's logic
    383 except AttributeError:

File ~/miniforge3/envs/protein/lib/python3.10/site-packages/torch/nn/modules/module.py:1931, in Module.__getattr__(self, name)
   1930         return modules[name]
-> 1931 raise AttributeError(
   1932     f"'{type(self).__name__}' object has no attribute '{name}'"
   1933 )

AttributeError: 'LoraModel' object has no attribute 'prepare_inputs_for_generation'

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
Cell In[6], line 7
      3
      5 # Workaround over using AutoPeftModelForCausalLM.from_pretrained
      6 # Which raises errors due to custom classes
----> 7 model2=PeftModelForCausalLM.from_pretrained(
      8     AutoModelForMaskedLM.from_pretrained(
      9             PeftConfig.from_pretrained(pt_path).base_model_name_or_path,
     10             local_files_only=True, trust_remote_code=True,
     11     ),
     12     pt_path,
     13     adapter_name='default',
     14     is_trainable = False,
     15     local_files_only=True, trust_remote_code=True
     16 )

File ~/miniforge3/envs/protein/lib/python3.10/site-packages/peft/peft_model.py:539, in PeftModel.from_pretrained(cls, model, model_id, adapter_name, is_trainable, config, autocast_adapter_dtype, ephemeral_gpu_offload, low_cpu_mem_usage, key_mapping, **kwargs)
    536                 raise ValueError("If model_id is a local path, then `adapters` must be passed in kwargs.")
    538 if config.task_type not in MODEL_TYPE_TO_PEFT_MODEL_MAPPING.keys():
--> 539     model = cls(
    540         model,
    541         config,
    542         adapter_name,
    543         autocast_adapter_dtype=autocast_adapter_dtype,
    544         low_cpu_mem_usage=low_cpu_mem_usage,
    545     )
    546 else:
    547     model = MODEL_TYPE_TO_PEFT_MODEL_MAPPING[config.task_type](
    548         model,
    549         config,
   (...)
    552         low_cpu_mem_usage=low_cpu_mem_usage,
    553     )

File ~/miniforge3/envs/protein/lib/python3.10/site-packages/peft/peft_model.py:1816, in PeftModelForCausalLM.__init__(self, model, peft_config, adapter_name, **kwargs)
   1812 def __init__(
   1813     self, model: torch.nn.Module, peft_config: PeftConfig, adapter_name: str = "default", **kwargs
   1814 ) -> None:
   1815     super().__init__(model, peft_config, adapter_name, **kwargs)
-> 1816     self.base_model_prepare_inputs_for_generation = self.base_model.prepare_inputs_for_generation

File ~/miniforge3/envs/protein/lib/python3.10/site-packages/peft/tuners/lora/model.py:386, in LoraModel.__getattr__(self, name)
    384 if name == "model":  # see #1892: prevent infinite recursion if class is not initialized
    385     raise
--> 386 return getattr(self.model, name)

File ~/miniforge3/envs/protein/lib/python3.10/site-packages/torch/nn/modules/module.py:1931, in Module.__getattr__(self, name)
   1929     if name in modules:
   1930         return modules[name]
-> 1931 raise AttributeError(
   1932     f"'{type(self).__name__}' object has no attribute '{name}'"
   1933 )

AttributeError: 'ESMplusplusForMaskedLM' object has no attribute 'prepare_inputs_for_generation'

adapter_config.json

{
  "alpha_pattern": {},
  "auto_mapping": {
    "base_model_class": "ESMplusplusForMaskedLM",
    "parent_library": "transformers_modules.modeling_esm_plusplus"
  },
  "base_model_name_or_path": "", # Removed
  "bias": "none",
  "corda_config": null,
  "eva_config": null,
  "exclude_modules": null,
  "fan_in_fan_out": false,
  "inference_mode": true,
  "init_lora_weights": true,
  "layer_replication": null,
  "layers_pattern": null,
  "layers_to_transform": null,
  "loftq_config": {},
  "lora_alpha": 64,
  "lora_bias": false,
  "lora_dropout": 0.01,
  "megatron_config": null,
  "megatron_core": "megatron.core",
  "modules_to_save": null,
  "peft_type": "LORA",
  "qalora_group_size": 16,
  "r": 32,
  "rank_pattern": {},
  "revision": null,
  "target_modules": [
    "out_proj",
    "layernorm_qkv.1",
    "key",
    "query",
    "value",
    "dense"
  ],
  "target_parameters": null,
  "task_type": null,
  "trainable_token_indices": null,
  "use_dora": false,
  "use_qalora": false,
  "use_rslora": true
}

[BenjaminBossan]

Loading peft model without task type:

Okay, this is because you use PeftModelForCausalLM, which is effectively the same as passing CAUSAL_LM as task_type. If you use PeftModel instead, it should work.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.