Chat template for Pixtral 12B model

[ajithkumarkaliyappan]

I need help on hosting Pixtral 12b model using Vllm with the chat template enabled. If any anyone has chat template for Pixtral 12B model, drop your template and thoughts.

[nintwentydo]

I've been doing some experimenting with Pixtral 12B on VLLM. This is the chat template I use if it helps.

It's an expanded version of the default Mistral template. It allows for images to be provided in all response types, allows non-alternating roles, and inline system messages. Works pretty well if you want to use it.

{{- bos_token }}
{%- for message in messages %}
    {%- if message['role'] == 'user' %}
        {{- '[INST]' }}
        {%- if message['content'] is not string %}
            {%- for chunk in message['content'] %}
                {%- if chunk['type'] == 'text' %}
                    {{- chunk['content'] }}
                {%- elif chunk['type'] == 'image' %}
                    {{- '[IMG]' }}
                {%- else %}
                    {{- raise_exception('Unrecognized content type!') }}
                {%- endif %}
            {%- endfor %}
        {%- else %}
            {{- message['content'] }}
        {%- endif %}
        {%- if tools is defined and tools is not none and loop.last %}
            {{- '[AVAILABLE_TOOLS][' }}
            {%- for tool in tools %}
                {%- set tool = tool.function %}
                {{- '{"type": "function", "function": {' }}
                {%- for key, val in tool.items() if key != "return" %}
                    {%- if val is string %}
                        {{- '"' + key + '": "' + val + '"' }}
                    {%- else %}
                        {{- '"' + key + '": ' + val|tojson }}
                    {%- endif %}
                    {%- if not loop.last %}
                        {{- ', ' }}
                    {%- endif %}
                {%- endfor %}
                {{- '}}' }}
                {%- if not loop.last %}
                    {{- ', ' }}
                {%- endif %}
            {%- endfor %}
            {{- ']' }}
            {{- '[/AVAILABLE_TOOLS]' }}
        {%- endif %}
        {{- '[/INST]' }}
    {%- elif message['role'] == 'system' %}
        {{- '[SYSTEM_PROMPT]' + message['content'] + '[/SYSTEM_PROMPT]' }}
    {%- elif message['role'] == 'assistant' %}
        {%- if message['content'] is not string %}
            {%- for chunk in message['content'] %}
                {%- if chunk['type'] == 'text' %}
                    {{- chunk['content']|trim }}
                {%- elif chunk['type'] == 'image' %}
                    {{- '[IMG]' }}
                {%- else %}
                    {{- raise_exception('Unrecognized content type!') }}
                {%- endif %}
            {%- endfor %}
        {%- else %}
            {{- message['content']|trim }}
        {%- endif %}
        {{- eos_token }}
    {%- elif message['role'] == 'tool_results' or message['role'] == 'tool' %}
        {%- if message.content is defined and message.content.content is defined %}
            {%- set content = message.content.content %}
        {%- else %}
            {%- set content = message.content %}
        {%- endif %}
        {{- '[TOOL_RESULTS]{"content": ' + content|string + ', ' }}
        {%- if message.content is mapping and 'image' in message.content %}
            {{- '"image": "[IMG]", ' }}
        {%- endif %}
        {%- if not message.tool_call_id is defined or message.tool_call_id|length != 9 %}
            {{- raise_exception("Tool call IDs should be alphanumeric strings with length 9!") }}
        {%- endif %}
        {{- '"call_id": "' + message.tool_call_id + '"}[/TOOL_RESULTS]' }}
    {%- elif message.tool_calls is defined and message.tool_calls is not none %}
        {{- '[TOOL_CALLS][' }}
        {%- for tool_call in message.tool_calls %}
            {%- set out = tool_call.function|tojson %}
            {{- out[:-1] }}
            {%- if not tool_call.id is defined or tool_call.id|length != 9 %}
                {{- raise_exception("Tool call IDs should be alphanumeric strings with length 9!") }}
            {%- endif %}
            {{- ', "id": "' + tool_call.id + '"}' }}
            {%- if not loop.last %}
                {{- ', ' }}
            {%- endif %}
        {%- endfor %}
        {{- ']' + eos_token }}
    {%- else %}
        {{- raise_exception('Only user, system, assistant, tool, and tool_results roles are supported!') }}
    {%- endif %}
{%- endfor %}

If you use it make sure you also pass the content format arg as it might default to OpenAI format and break. e.g. use --chat-template /path/to/pixtral12b.jinja --chat-template-content-format string

Also if you're using a chat template with pixtral you need to use a HF version of the model, not the mistral version.
FP16 version - https://huggingface.co/unsloth/Pixtral-12B-2409

I've got some quants published. W4A16 brings model size down to ~9GB https://huggingface.co/nintwentydo/pixtral-12b-2409-W4A16-G128

And these are some others I've tried
https://huggingface.co/DewEfresh/pixtral-12b-8bit
https://huggingface.co/neuralmagic/pixtral-12b-FP8-dynamic
https://huggingface.co/unsloth/Pixtral-12B-2409-bnb-4bit

I'd recommend avoiding bitsandbytes though - prompt processing is already slow on pixtral compared to other models and bnb has been by far the slowest for me.

[ajithkumarkaliyappan]

Can we use this chat template to work with Autogen agentic framework? I tried to use it. but getting following error,
raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'object': 'error', 'message': "2 validation errors for ChatCompletionRequest\nmessages.1.assistant.name\n Extra inputs are not permitted [type=extra_forbidden, input_value='user', input_type=str]\n For further information visit https://errors.pydantic.dev/2.9/v/extra_forbidden\nmessages.2.system.name\n Extra inputs are not permitted [type=extra_forbidden, input_value='checking_agent', input_type=str]\n For further information visit https://errors.pydantic.dev/2.9/v/extra_forbidden", 'type': 'BadRequestError', 'param': None, 'code': 400}

[nintwentydo]

Haven’t tried on Autogen so don’t have much to offer I’m afraid - looks like it’ll only allow text input on assistant messages. If you want a baseline setup remove image inputs from assistant and tool response messages. That might work for you on Autogen.

I’ve tested my template on VLLM, TGI, TabbyAPI, and directly with Transformers and it works on them all.