If you have any questions about the library, please feel free to contact us. Email: [email protected]

A comprehensive list of papers about '[Knowledge Fusion: The Integration of Model Capabilities.]'.

As the comprehensive capabilities of foundational large models rapidly improve, similar general abilities have emerged across different models, making capability transfer and fusion between them more feasible. Knowledge fusion aims to integrate existing LLMs of diverse architectures and capabilities into a more powerful model through efficient methods such as knowledge distillation, model merging, mixture of experts, and PEFT, thereby reducing the need for costly LLM development and adaptation. We provide a comprehensive overview of model merging methods and theories, covering their applications across various fields and scenarios, including LLMs, MLLMs, image generation, model compression, continual learning, and more. Finally, we highlight the challenges of knowledge fusion and explore future research directions.

• Awesome-Knowledge-Fuse
  • 1. Connectivity and Alignment
    • 1.1 Model Connectivity
    • 1.2 Weight Alignment
  • 2. Parameter Merging
    • 2.1 Merging Methods
      • Gradient based
      • Task Vector based
    • 2.2 During or After Training
      • During Training
      • After Training
    • 2.3 For LLMs and MLLMs
      • For LLMs
      • For MLLMs
  • 3. Model Ensemble
    • 3.1 Ensemble Methods
      • Weighted Averaging
      • Routing
      • Voting
    • 3.2 Ensemble Object
      • Entire Model
      • Adapter
  • 4. Decouple and Reuse
    • 4.1 Reprogramming
    • 4.2 Mask
  • 5. Distillation
    • 5.1 Transformer
    • 5.2 CNN
    • 5.3 GNN
  • 6. Model Reassemble
    • 6.1 Model Stitch
    • 6.2 Model Evolution
  • 7. Others
    • 7.1 External Data Retrieval
    • 7.2 Other Surveys

Star History

We invite all researchers to contribute to this repository, 'Knowledge Fusion: The Integration of Model Capabilities'. If you have any questions about the library, please feel free to contact us.

Email: [email protected]