llama: Attempt to add ModernBert

[huydt84]

I don't know whether my implementation is correct or not

[huydt84]

hparams.set_swa_pattern can't work properly with ModernBert

[huydt84]

The embedding result seems random and very low. There is something wrong with this

[CISC]

Delete the files you added in models, we don't need them, just make sure test-tokenizer-0 succeeds with the GGUF.

[ggerganov]

This should probably become:

Suggested change

	auto * inp_attn = build_attn_inp_kv_unified_iswa();
	auto * inp_attn = build_attn_inp_no_cache_iswa();

And add the corresponding mask logic in llama-graph. Special attention should be taken about how the SWA works for this model - i.e. is it symmetric or not:

# non-symmetric
token i attends to [i - n_swa, i]

# symmetric:
token i attends to [i - n_swa/2, i + n_swa/2]

[ggerganov]

You have to add the new arch here:

llama.cpp/src/llama-model.cpp

Lines 13195 to 13203 in 5a8ae30

	switch (arch) {
	case LLM_ARCH_BERT:
	case LLM_ARCH_JINA_BERT_V2:
	case LLM_ARCH_NOMIC_BERT:
	case LLM_ARCH_NOMIC_BERT_MOE:
	case LLM_ARCH_WAVTOKENIZER_DEC:
	{
	res = nullptr;
	} break;

To avoid creating a memory module (a.k.a. KV cache) for these models.

[CISC]

So, since vocab is BPE you need to add modern-bert vocab handling a few places:

llama.cpp/src/llama-vocab.cpp

Line 1557 in 9f47fa5

tokenizer_pre == "roberta-bpe") {

Set correct attribute on [MASK] token, similarly to this:

llama.cpp/src/llama-vocab.cpp

Lines 2097 to 2105 in 9f47fa5

	if (false
	|| _contains_any(tokenizer_pre, {"jina-v2-de", "jina-v2-es", "jina-v2-code"})
	|| _contains_any(general_arch, {"nomic-bert-moe"})
	) {
	if (token_to_id.count("<mask>") == 0) {
	LLAMA_LOG_WARN("%s: Mask token is missing in vocab, please reconvert model!\n", __func__);
	} else {
	_set_token_attr("<mask>", LLAMA_TOKEN_ATTR_LSTRIP, true);
	}

[CISC]

The embedding result seems random and very low. There is something wrong with this

Yep, I also noticed the same with jina-reranker-v2, most likely the same issue, will investigate.

[CISC]

So, since vocab is BPE you need to add modern-bert vocab handling a few places:

@huydt84 Don't forget this-^ it's important.

[CISC]

The embedding result seems random and very low. There is something wrong with this

Yep, I also noticed the same with jina-reranker-v2, most likely the same issue, will investigate.

Will dig into this tonight/this weekend...

[huydt84]

So, since vocab is BPE you need to add modern-bert vocab handling a few places:

@huydt84 Don't forget this-^ it's important.

Thank you! I have just added it

[CISC]

@huydt84 Don't forget this-^ it's important.

Thank you! I have just added it

The tokenizer_pre check is the most important one, please add that too. :)

[ggerganov]

Need to add new enum llama_swa_type:

LLAMA_SWA_TYPE_SYMMETRIC  = 3,

[ggerganov]

Since this is not an actual iSWA (interleaved SWA) model, we should use simply build_attn_inp_no_cache().

[ggerganov]

This is already available from the hparams - no need to duplicate it here.

[ggerganov]

This branch is actually non-causal attention + sliding window. So merge it with the existing implementation below.