support Chinese model ,please

[louis-gg]

Feature request

Here is the Chinese model git:https://github.com/ymcui/Chinese-LLaMA-Alpaca.
Support it please!

Motivation

support Chinese model

Your contribution

no

[cosmic-snow]

Is there a ggml version of that somewhere?

If not, you need to find someone to create that or try yourself. Then it should automatically be supported, as it's based on LLaMA.

P.S. With "automatically supported" I mean that the model type would be, not that it would automatically be in the download list. But you could download that version from somewhere and put it next to your other models.

[louis-gg]

Is there a ggml version of that somewhere?

If not, you need to find someone to create that or try yourself. Then it should automatically be supported, as it's based on LLaMA.

P.S. With "automatically supported" I mean that the model type would be, not that it would automatically be in the download list. But you could download that version from somewhere and put it next to your other models.

Thank you !
I download many Model on https://huggingface.co/models ,format is ggml,but warning not support.
link is:https://huggingface.co/models?language=zh&sort=trending&search=ggml

[cosmic-snow]

Thank you ! I download many Model on https://huggingface.co/models ,format is ggml,but warning not support. link is:https://huggingface.co/models?language=zh&sort=trending&search=ggml

I'm not sure if I understand you right.

Have you tried for example: https://huggingface.co/TheBloke/baichuan-llama-7B-GGML/tree/main? I recommend baichuan-llama-7b.ggmlv3.q5_K_M.bin or if you have a Mac M1/M2 baichuan-llama-7b.ggmlv3.q4_0.bin.

This is the right format. If you download it and put it next to the other models (the download directory), it should just work.

For https://github.com/ymcui/Chinese-LLaMA-Alpaca, it would need to be converted into a format like that.

[louis-gg]

Thank you ! I download many Model on https://huggingface.co/models ,format is ggml,but warning not support. link is:https://huggingface.co/models?language=zh&sort=trending&search=ggml

I'm not sure if I understand you right.

Have you tried for example: https://huggingface.co/TheBloke/baichuan-llama-7B-GGML/tree/main? I recommend baichuan-llama-7b.ggmlv3.q5_K_M.bin or if you have a Mac M1/M2 baichuan-llama-7b.ggmlv3.q4_0.bin.

This is the right format. If you download it and put it next to the other models (the download directory), it should just work.

For https://github.com/ymcui/Chinese-LLaMA-Alpaca, it would need to be converted into a format like that.
thank you reply!
I already down the llama-7b.ggmlv3.q5_K_M.bin. My os is windows 11.
But I still get error:

I guess some bugs in chat .

[cosmic-snow]

Not necessarily. What's your CPU and how much RAM do you have?

There's also a possibility the download didn't complete right, and maybe there is something wrong with that model.

I might have to download it myself to compare.

Edit: I've downloaded it, verified its checksum and tried to load it. I'm running into the same error as you. So I think there's something wrong with that model. I might have a closer look later.

[louis-gg]

I might have a closer look later.

Thank you for your reply!
My System is :

Thank you for looking !
I test almost Chinese models in https://huggingface.co/models?language=zh&sort=trending&search=ggml , but same error.
I debug in the qt,found that:

When choose the impl,the impl is null!
So, I think it might choose model impl will cause some error.

[cosmic-snow]

I've looked into it some more. The good news is that it is possible to get it to run by disabling a check. The bad news is: that check is there for a reason, it is used to tell LLaMA apart from Falcon.

If you are not going to use a Falcon model and since you are able to compile yourself, you can disable the check on your own system if you want. In gpt4all-backend/llamamodel.cpp comment out return false in the vocabulary size check:

gpt4all/gpt4all-backend/llamamodel.cpp

Lines 285 to 289 in 2d02c65

	llama_file_hparams hparams;
	f.read(reinterpret_cast<char*>(&hparams), sizeof(hparams));
	if (!(hparams.n_vocab >= 32000 && hparams.n_vocab <= 32100)) {
	return false; // not a llama.
	}

That is, you can change it to:

    llama_file_hparams hparams;
    f.read(reinterpret_cast<char*>(&hparams), sizeof(hparams));
    if (!(hparams.n_vocab >= 32000 && hparams.n_vocab <= 32100)) {
        //return false; // not a llama.
    }

It's not generally recommended to do that, however.

[louis-gg]

I've looked into it some more. The good news is that it is possible to get it to run by disabling a check. The bad news is: that check is there for a reason, it is used to tell LLaMA apart from Falcon.

If you are not going to use a Falcon model and since you are able to compile yourself, you can disable the check on your own system if you want. In gpt4all-backend/llamamodel.cpp comment out return false in the vocabulary size check:

gpt4all/gpt4all-backend/llamamodel.cpp

Lines 285 to 289 in 2d02c65

	llama_file_hparams hparams;
	f.read(reinterpret_cast<char*>(&hparams), sizeof(hparams));
	if (!(hparams.n_vocab >= 32000 && hparams.n_vocab <= 32100)) {
	return false; // not a llama.
	}

That is, you can change it to:

    llama_file_hparams hparams;
    f.read(reinterpret_cast<char*>(&hparams), sizeof(hparams));
    if (!(hparams.n_vocab >= 32000 && hparams.n_vocab <= 32100)) {
        //return false; // not a llama.
    }

It's not generally recommended to do that, however.

Thank you ! I had to change it.
Work well!
Maybe new version will fix it?

[cosmic-snow]

Thank you ! I had to change it.
Work well!
Maybe new version will fix it?

Not sure.

As I said, the check is necessary to distinguish LLaMA from Falcon models, so it's kind of important. And this model is a bit special. Only remove the check if you are not going to use Falcon models.

[louis-gg]

Thank you!

[cvcuiwei]

文章有帮助

[niansa]

Duplicate of #176