Feature Request: Add Ernie4.5MoE support

[EliEron]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Currently llama.cpp only supports the 0.3B dense ERNIE 4.5 model, but all other models in the ERNIE family are MoE models which has not been implemented yet.

Motivation

ERNIE 4.5 is an exciting new family of open multimodal Apache 2.0 licensed models from Baidu, boasting some pretty impressive benchmarks.

There is a lot of hype around these models, and I feel a lot of people would love to see this supported fully in llama.cpp. Especially since these models should run quite well on CPU centered setups due to it's MoE architecture.

Possible Implementation

There are open PRs for vLLM and SGLang that can be used as reference.

[Downtown-Case]

One particularly interesting bit is a 2-bit QAT model:

... convolutional code quantization algorithm to achieve 4-bit/2-bit lossless quantization...

https://huggingface.co/baidu/ERNIE-4.5-300B-A47B-2Bits-Paddle

From the paper:

https://yiyan.baidu.com/blog/publication/ERNIE_Technical_Report.pdf

To address the aforementioned issues, we propose Convolutional Code Quantization (CCQ), a scalar
quantization algorithm based on the convolutional code. The approach not only retains the high-precision
data quantization capability of vector quantization but also preserves the low computational complexity
of scalar quantization. Combined with scale quantization and optimization, we achieve the highest
possible compression ratio while simultaneously minimizing inference overhead.
Convolutional Codebook. Inspired by QTIP (Tseng et al., 2024b), we innovatively integrate convolutional code with scalar quantization through a series of meticulously designed coding structures. Based
on convolutional codes, we construct a lookup-free codebook that achieves a linear mapping between the
codebook and weight vectors, thereby optimizing inference performance. Meanwhile, by drawing on the
concept of data mapping from vector quantization, we minimize the performance degradation of the
model under extremely low-bit conditions.
Hybrid Encoding. We employ convolutional codes with varying coding configurations to accommodate the storage of encoded values in INT8 and INT16 formats. As a result, we successfully compress
4-bit scalar quantization to an equivalent of 2.75 bits and 3-bit scalar quantization to 2.5 bits.
Code Clustering. Furthermore, by analyzing the distribution of encoded values across each channel,
we observe that they conform to a normal distribution, enabling deeper compression along the coding
dimension. Through clustering of the convolutional codes, we can compress any coding configuration to
an equivalent of 2 bits, thereby further enhancing the model compression rate.

(sorry for formatting)

There's also details on KV cache quantization.

[foldl]

FYI: chatllm.cpp supports this model.