Refactor: PEFT method registration function

[BenjaminBossan]

This is a refactor of how new PEFT methods are registered. It got bigger than I initially expected.

Goal

The goal of this refactor is the following: Right now, when a new PEFT method is added, a new directory is created in src/peft/tuners/<name> with a config, model, etc. This is fine and self-contained.

However, in addition to that, a couple of other places in the PEFT code base need to be touched for this new PEFT method to become usable.

As an example, take the recently added Bone method (#2172). Ignoring tests, docs, and examples, we have the additions to src/peft/tuners/bone, but also need to:

1. Add an entry to PEFT_TYPE_TO_CONFIG_MAPPING in mapping.py.
2. Add an entry to PEFT_TYPE_TO_TUNER_MAPPING in mapping.py.
3. Add an entry to PEFT_TYPE_TO_MODEL_MAPPING in peft_model.py
4. Add an entry to PEFT_TYPE_TO_PREFIX_MAPPING in utils/constants.py
5. Add some code to get_peft_model_state_dict in utils.save_and_load.py

With the changes in this PR, all these steps can be omitted.

On top of that, we also have the re-imports to peft/__init__.py and peft/tuners/__init__.py but those are still required (I'm hesitant to mess with the import system). Furthermore, it's still required to add an entry to PeftType in utils.peft_types.py. Since this is an enum, it can't be easily generated automatically. Therefore, adding a new PEFT method is still not 100% self-contained.

Changes in this PR

With this PR, less book-keeping is required. Instead of the 5 steps described above, contributors now only need to call

# example for the Bone method
register_peft_method(name="bone", config_cls=BoneConfig, model_cls=BoneModel)

in the __init__.py of their PEFT method. In addition to registering the method, this also performs a couple of sanity checks (e.g. no duplicate names, method name and method prefix being identical).

Moreover, since so much book keeping is removed, this PR reduces the number of lines of code overall (at the moment +317, - 343).

Implementation

The real difficulty of this task is that the module structure in PEFT is really messy, easily resulting in circular imports. This has been an issue in the past but has been especially painful here. For this reason, some stuff had to be moved around:

• MODEL_TYPE_TO_PEFT_MODEL_MAPPING is now in auto.py instead of mapping.py
• PEFT_TYPE_TO_PREFIX_MAPPING has been moved to mapping.py from constants.py
• get_peft_model had to be moved out of mapping.py and is now in its own module, func.py (better name suggestions welcome). This should be safe, as the function is re-imported to the main PEFT namespace, which all examples use.

The PEFT_TYPE_TO_MODEL_MAPPING dict could be completely removed, as it was basically redundant with PEFT_TYPE_TO_TUNER_MAPPING. The get_peft_model_state_dict could be simplified, as a lot of code was almost duplicated.

There were a few instances in peft_model.py like:

        elif config.peft_type == PeftType.P_TUNING:
            prompt_encoder = PromptEncoder(config)

Now, instead of hard-coding the model, I just do model_cls = PEFT_TYPE_TO_TUNER_MAPPING[config.peft_type].

Overall, I think this is a cleaner module structure, but still not very clean overall.

Open questions

I'm not 100% sure if this should be merged. AFAICT it should be a safe refactor that does not affect user code. There could be other packages out there that use some PEFT internals that could break with this refactor. If we decide to merge this, we should consider alerting potentially affected packages to test it.

I'm also open to the argument that the benefits are not outweighing the cost of the refactor.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[githubnemo]

As for the name of func.py: Maybe mapping_func.py is a more appropriate name?

[BenjaminBossan]

As for the name of func.py: Maybe mapping_func.py is a more appropriate name?

Good idea, I renamed the module.

[githubnemo]

In general I do like static mapping more than dynamic, makes the code more readable without interpreting too much. However, in this case I can see that it is easy to forget certain mappings and keeping them at one place is not always easy. It also makes for a good public interface.

If you could add the doc string for register_peft_method we can merge this, I think.

[BenjaminBossan]

In general I do like static mapping more than dynamic, makes the code more readable without interpreting too much. However, in this case I can see that it is easy to forget certain mappings and keeping them at one place is not always easy. It also makes for a good public interface.

I agree. The change is not pure upside, but overall I think it is cleaner than it was before. Another possible advantage: If we ever require something more to happen when a new PEFT method is added, we could possibly change it in a single place instead of having to touch each method individually.

If you could add the doc string for register_peft_method we can merge this, I think.

Thanks for catching this, I added the docstring.