text generation webui_en

Inference with text-generation-webui

A Gradio web UI for running large models (such as huggingface format, gguf format, etc.)

Step 1: install text-generation-webui

Text-generation-webui currently supports Windows/Linux/macOS/WSL systems. Refer to webui installation for more details.

Step 2: Preparing model weights

text-generation-webui currently supports huggingface and gguf models.

huggingface format

Copy the full weights of Chinese-Mixtral-Instruct to the text-generation-webui models folder, as shown below:

text-generation-webui
└── models
    └── Chinese-Mixtral-Instruct
        ├── config.json
        ├── generation_config.json
        ├── model-00001-of-00019.safetensors
        ├── model-00002-of-00019.safetensors
        ├── ……
        ├── model-00019-of-00019.safetensors
        ├── model.safetensors.index.json
        ├── special_tokens_map.json
        ├── tokenizer_config.json
        └── tokenizer.model

GGUF format

GGUF models are a single file and should be placed directly into models. Example:

text-generation-webui
└── models
    └── Chinese-Mixtral-Instruct-q4_k.gguf

Step 3: Load the model and start the webui

Run the following command to talk to Chinese-Mixtral-Instruct.

python server.py

On the Web page that opens, select the model you want to chat with. The webui will select the loading mode according to the model format. Then select Parameters -> Instruction template in order, then in Instruction template, choose Mistral from the drop-down menu. Finally, go back to the Text generation interface, type your instruction in the chat box, and you can start a conversation with chinese-mixtral-instruct.

Please refer to webui using docs for more detailed official instructions. If you encounter installation or running problems, please go to the original repo to ask questions, the process has been run under commit-id 837bd88.