Probably incorrect usage of USE_REASONING_MODEL and USE_INSTRUCT_MODEL flags for `Qwen3Tokenizer`

[labdmitriy]

Hi @rasbt,

Maybe this is also related to #828, but there is the current definition of Qwen3Tokenizer for Qwen3 notebooks:

if USE_REASONING_MODEL:
    tokenizer_file_path = f"Qwen3-{CHOOSE_MODEL}/tokenizer.json"
else:
    tokenizer_file_path = f"Qwen3-{CHOOSE_MODEL}-Base/tokenizer.json"

hf_hub_download(
    repo_id=repo_id,
    filename="tokenizer.json",
    local_dir=local_dir,
)

tokenizer = Qwen3Tokenizer(
    tokenizer_file_path=tokenizer_file_path,
    repo_id=repo_id,
    apply_chat_template=USE_REASONING_MODEL,
    add_generation_prompt=USE_REASONING_MODEL,
    add_thinking=not USE_INSTRUCT_MODEL
)

Maybe I didn't understand correctly the meaning of USE_REASONING_MODEL and USE_INSTRUCT_MODEL, but don't we need to swap them during initialization?

tokenizer = Qwen3Tokenizer(
    tokenizer_file_path=tokenizer_file_path,
    repo_id=repo_id,
    apply_chat_template=USE_INSTRUCT_MODEL,
    add_generation_prompt=USE_INSTRUCT_MODEL,
    add_thinking=not USE_REASONING_MODEL
)

[rasbt]

Thanks for double-checking. Getting this right is a bit tricky and requires some mental gymnastics (at least for me, haha).

I think the current way it's implemented is correct. Let me write my chain of thought:

1. Qwen3 0.6B comes in both a base (let's call it model 1) and non-base version (let's call it model 2). I call the non-base version (model 2) "reasoning" version as it reasons by default (but it can also be used as instruct model). If USE_REASONING_MODEL=True it uses model 2, which requires the chat template at all times. Hence apply_chat_template=USE_REASONING_MODEL

2. Assuming we use model 2, i.e., USE_REASONING_MODEL=True, we now have a choice: do we want to use it as a reasoning model or as an instruct model. If we want to use it as an instruct model, we have to set add_thinking=False, hence add_thinking=not USE_INSTRUCT_MODEL.

Pls let me know in case you think I made a mistake here. It could well be that I overlooked something.

[labdmitriy]

To be honest, it's not entirely intuitive to me. :)

Probably it is just for me who has not enough experience, but based on the names of the variables and corresponding arguments in this function explain me the following:

• If model is instruct (reasoning or not), then we apply chat template and add generation prompt
• If mode is not reasoning, so we need to add empty think tags to force the model to skip thinking, otherwise we just add new line to let the model think

[casinca]

I think it also looks correct, when we want to use reasoning, all the flags should be true (not USE_INSTRUCT_MODEL will be True by default because USE_INSTRUCT_MODEL is False)

If you want to use instruct, you also toggle to USE_INSTRUCT_MODEL=True which will turns to add_thinking=False without changing anything else.

But now I'm wondering if there's not a typo in cell 3 @rasbt :

USE_INSTRUCT_MODEL = False
# Uses the instruct mode (without reasoning) if 
# USE_REASONING_MODEL = True
# USE_INSTRUCT_MODEL = False
# This setting does have no effect if USE_REASONING_MODEL = False

Shouldn't this be both true, I mean with # USE_INSTRUCT_MODEL = True?

[labdmitriy]

I mean maybe the values are correct (I haven't checked yet), but probably the names of these 2 flags are confusing (at least for me) based on the explanation above and maybe the name should be swapped?

Because as I understand it seems that reasoning itself should not decide if we use generation prompt and use chat template (it is about instruct property), and instruct itself should not decide if we need to add empty think tags or not (it is about reasoning property).

Maybe I am wrong and I would be very grateful for an explanation of what I'm wrong about.

[rasbt]

No worries! And I think I see the loophole now: What if you want to use the base model with a chat template (i.e., if you want to further finetune the model).

In this case, your suggestion:

tokenizer = Qwen3Tokenizer(
    tokenizer_file_path=tokenizer_file_path,
    repo_id=repo_id,
    apply_chat_template=USE_INSTRUCT_MODEL,
    add_generation_prompt=USE_INSTRUCT_MODEL,
    add_thinking=not USE_REASONING_MODEL
)

is better but it also breaks this then:

if USE_REASONING_MODEL:
    tokenizer_file_path = f"Qwen3-{CHOOSE_MODEL}/tokenizer.json"
else:
    tokenizer_file_path = f"Qwen3-{CHOOSE_MODEL}-Base/tokenizer.json"

if you are using the official instruct model.

The different scenarios are:

1 Plain base model

repo_id = f"Qwen/Qwen3-Base"
tokenizer_file_path = f"Qwen3-Base/tokenizer.json"

tokenizer = Qwen3Tokenizer(
    tokenizer_file_path=tokenizer_file_path,
    repo_id=repo_id,
    apply_chat_template=False,
    add_generation_prompt=# can be True/False,
    add_thinking=# can be True/False
)

2 Base model someone wants to instruction finetune

repo_id = f"Qwen/Qwen3-Base"
tokenizer_file_path = f"Qwen3-Base/tokenizer.json"

tokenizer = Qwen3Tokenizer(
    tokenizer_file_path=tokenizer_file_path,
    repo_id=repo_id,
    apply_chat_template=True,
    add_generation_prompt=True,
    add_thinking=True
)

3 Reasoning model

repo_id = f"Qwen/Qwen3"
tokenizer_file_path = f"Qwen3/tokenizer.json"

tokenizer = Qwen3Tokenizer(
    tokenizer_file_path=tokenizer_file_path,
    repo_id=repo_id,
    apply_chat_template=True,
    add_generation_prompt=True,
    add_thinking=True
)

4 Instruct model

repo_id = f"Qwen/Qwen3"
tokenizer_file_path = f"Qwen3/tokenizer.json"

tokenizer = Qwen3Tokenizer(
    tokenizer_file_path=tokenizer_file_path,
    repo_id=repo_id,
    apply_chat_template=True,
    add_generation_prompt=True,
    add_thinking=False
)

[labdmitriy]

Now I think that you swapped add_thinking values in sections 3 and 4 in the last comment, because as I understand from the notebook that this name means to add empty think tags if you don't have reasoning model, but section 3 is about reasoning model, and vice versa for section 4.

So probably renaming the variables and arguments will help a lot to have a more clear picture of what options are available and how they are processed.

[rasbt]

@labdmitriy

Now I think that you swapped add_thinking values in sections 3 and 4 in the last comment,

No, I think that's correct. I.e.,

here thinking is disabled, which means think tags are added:

tokenizer = Qwen3Tokenizer(
    tokenizer_file_path=tokenizer_file_path,
    repo_id=repo_id,
    apply_chat_template=True,
    add_generation_prompt=True,
    add_thinking=False
)

# '<|im_start|>user\nGive me a short introduction to large language models.<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n'

And here thinking is enabled (no think tags), meaning the reasoning model will generate the think tags:

tokenizer = Qwen3Tokenizer(
    tokenizer_file_path=tokenizer_file_path,
    repo_id=repo_id,
    apply_chat_template=True,
    add_generation_prompt=True,
    add_thinking=True
)

# '<|im_start|>user\nGive me a short introduction to large language models.<|im_end|>\n<|im_start|>assistant\n'

I know the add_thinking logic seems counterintuitive, but I am just mimicking the official tokenizer.

[labdmitriy]

Ah, sorry, yes you are correct, my mistake!

[rasbt]

@casinca

Shouldn't this be both true, I mean with # USE_INSTRUCT_MODEL = True?

Yes, looking at this again, the comment should say # USE_INSTRUCT_MODEL = True. I probably typed this before I deciced to use the not USE_INSTRUCT_MODEL negation.

[rasbt]

@labdmitriy

Maybe it would help to add comment with these 4 scenarios so that users can just decide to set them explicitly?

[labdmitriy]

@rasbt I suppose you and @casinca consider that these names of the flags are not swapped, so you have more experience and probably this is a problem to understand it just for me :) But I am still thinking that these flags are not clearly defined.
Even the part of the comment above also confuses me:

...
# Uses the instruct mode (without reasoning) if 
# USE_REASONING_MODEL = True
...

[labdmitriy]

About 4 scenarios you provided - probably the section 2 is also can have add_thinking values true or false, not just true?

If you ask about these scenarios without considering flag names (USE_REASONING_MODEL and USE_INSTRUCT_MODEL), but the description itself with specific values - to be honest for me it is much clear description 👍

[rasbt]

About 4 scenarios you provided - probably the section 2 is also can have add_thinking values true or false, not just true?

That's a good question and add_thinking=False would be the intuitive thing to do here. However, not that it then add the <think></think> tags, which you don't need. If you use add_thinking=True, it doesn't add anything. I know it's a bit weird this way, but that's because we are using the tokenizer of the reasoning model for the base model. (And I am also following the official Qwen3 tokenizer behavior; I don't want to add extra complication by making my own conventions or rules 😅)