Hidden states are different for model() and model.generate()

[okihnjo]

System Info

Model: Qwen/Qwen2.5-0.5B-Instruct, BitsAndBytesConfig(load_in_8bit=True)

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

When doing model.generate() and accessing hidden_states, I access the first element, which consists again of the number of layers.
Accessing an element again gives something like (batch_size, tokens, embedding_size) -> similar to model() when accessing the hidden_states.

I always thought that

generated_hidden_states[0][layer][:,-1,:],
hidden_states[layer][:,-1,:],

should be equal, both have the same shape and I thought both represent the same thing, but when looking at the cosine similarity between these vectors, I get different results

Does model.generate() do stuff that is not present in model() or vice versa ?

For reproducing:

model_name = "Qwen/Qwen2.5-0.5B-Instruct"

def load_model(model_name):
    bnb_config = BitsAndBytesConfig(load_in_8bit=True)

    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        quantization_config=bnb_config,
        
    )
    model.config.use_cache = False

    tokenizer = AutoTokenizer.from_pretrained(model_name, padding_side='left')

  
    tokenizer.pad_token = tokenizer.eos_token
    tokenizer.pad_token_id = tokenizer.eos_token_id
    return model, tokenizer

model, tokenizer = load_model(model_name)
model_inputs = tokenizer(
    ["A list of colors: red, blue", "Portugal is"], return_tensors="pt", padding=True
).to("cuda")
output = model(**model_inputs, output_hidden_states=True)
my_hidden_states = output.hidden_states
generated_ids = model.generate(**model_inputs, output_hidden_states=True,return_dict_in_generate=True, max_new_tokens=13, output_scores=True)
F.cosine_similarity(my_hidden_states[-1][0,-1,:], generated_ids["hidden_states"][0][-1][0,-1,:], dim=0)
# results in different values

### Expected behavior


Should have equal values.

[akshathmangudi]

Hi @okihnjo , I hope you're doing good.

I took some time to reproduce your script and verify few things. On a architectural note, calling model() and model.generate() do mean different things. Calling model passes over the entire input sequence in one forward pass while model.generate() performs incremental decoding, so it computes a forward pass, per generated token.

I also took some time to run your reproduction script, and the discrepancy could be quantization-induced. This was the output I got when I ran your reproduction script:

tensor(0.9980, device='cuda:0', dtype=torch.float16, grad_fn=<SumBackward1>)

0.9980 is very very close to 1. To make sure, I ran it with two more tests, one without quantization and one with Mistral/7B-Instruct-v0.1, these were the results I got respectively:

Output for test 1:

Cosine similarity (no quant): 1.000000

Output for test 2:

tensor(0.9985, device='cuda:0', dtype=torch.float16, grad_fn=<SumBackward1>)

This discrepancy doesn't seem to be limited to just the Qwen family, but other models too.

Codes I used for non-quantized version:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
import torch.nn.functional as F
from huggingface_hub import login
import os

model_name = "Qwen/Qwen2.5-0.5B-Instruct"

def load_model_no_q(model_name):
    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        torch_dtype=torch.float32,
        device_map="auto"
    )
    model.config.use_cache = False

    tokenizer = AutoTokenizer.from_pretrained(model_name, padding_side='left')
    tokenizer.pad_token = tokenizer.eos_token
    tokenizer.pad_token_id = tokenizer.eos_token_id

    return model, tokenizer

model, tokenizer = load_model_no_q(model_name)

model_inputs = tokenizer(
    ["A list of colors: red, blue", "Portugal is"], return_tensors="pt", padding=True
).to(model.device)

with torch.no_grad():
    output = model(**model_inputs, output_hidden_states=True)
    my_hidden_states = output.hidden_states

    generated_ids = model.generate(
        **model_inputs,
        output_hidden_states=True,
        return_dict_in_generate=True,
        max_new_tokens=13,
        output_scores=True
    )

cos_sim = F.cosine_similarity(
    my_hidden_states[-1][0, -1, :],
    generated_ids["hidden_states"][0][-1][0, -1, :],
    dim=0
)

print(f"Cosine similarity (no quant): {cos_sim.item():.6f}")

If you think I have misunderstood your problem, kindly let me know :)

[okihnjo]

Hey, I am good, hope you are doing well too!

Thanks for your response! When running this for a lot of queries, the cosine similarity tends to become worse (still in 0.9x region though) - I think I generally struggle with for activation/embedding analysis, I am not quite sure which method to call.
I know that model.generate() is for token generation, however I thought the first element represents the same as when doing simply model(). I want my model to do open text generation, but I want to analyze the query embedding (not the generated tokens but basically the last token of my input to the model), would you recommend doing one model()call for hidden_states analysis and one additional model.generate()call in order to get the text ? So two calls in one batch loop so to say?

[akshathmangudi]

Thanks for your response!

That seems to make the most sense. For your use-case where you want to analyze the last token of your input to the model, it would be wise to:

1. first call model() with output_hidden_states=True to extract the final hidden state of the prompt.
2. Then we pass input_ids, attention_maskto model.generate() for actual text output.

So, step 1 for analysis and step 2 for generation.

I have provided some code which may help you:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch.nn.functional as F

model_name = "Qwen/Qwen2.5-0.5B-Instruct"
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")
model.config.use_cache = False
tokenizer = AutoTokenizer.from_pretrained(model_name, padding_side='left')
tokenizer.pad_token = tokenizer.eos_token
tokenizer.pad_token_id = tokenizer.eos_token_id

inputs = tokenizer(["A list of colors: red, blue", "Portugal is"], return_tensors="pt", padding=True).to(model.device)


with torch.no_grad():
    output = model(**inputs, output_hidden_states=True)
    hidden_states = output.hidden_states
    last_hidden = hidden_states[-1]


    input_lengths = (inputs['attention_mask'] != 0).sum(dim=1)
    last_token_hiddens = torch.stack([
        last_hidden[i, input_lengths[i] - 1, :] for i in range(len(input_lengths))
    ])

with torch.no_grad():
    gen_output = model.generate(
        **inputs,
        return_dict_in_generate=True,
        output_scores=True,
        max_new_tokens=20
    )
    generated_texts = tokenizer.batch_decode(gen_output.sequences, skip_special_tokens=True)

cos_sim = F.cosine_similarity(last_token_hiddens[0], last_token_hiddens[1], dim=0)
print(f"Cosine similarity between inputs: {cos_sim.item():.6f}")

Where the output for this will be:

Cosine similarity between inputs: -0.142544

This actually makes sense because they both represent very different contexts semantically. So yes, running two calls where we do model() for analysis and model.generate() for generation will decouple the two tasks cleanly.

[gspeter-max]

Hi maintainers and @okihnjo (the original poster),

I've attempted to reproduce this issue (#38538) on the latest main branch of the transformers library (after pulling upstream main).

I used the following environment and code:

Environment: Google Colab with GPU runtime.

Libraries: transformers, accelerate, bitsandbytes (all freshly installed with !pip install -e . and !pip install accelerate bitsandbytes).

Model: Qwen/Qwen1.5-0.5B-Chat (corrected model ID as Qwen/Qwen1.5-0.5B-Instruct was not found).

Here is the exact code I used for reproduction:

import torch
import torch.nn.functional as F
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig 
from huggingface_hub import login
import os
from google.colab import userdata

# Authenticate to Hugging Face Hub
try:
    hf_token = userdata.get('HF_TOKEN') 
    login(token=hf_token)
except Exception:
    login() # Fallback to interactive login

# Reproduction Code from Issue #38538
model_name = "Qwen/Qwen1.5-0.5B-Chat" 

def load_model_and_tokenizer(model_name): 
    bnb_config = BitsAndBytesConfig(
        load_in_8bit=True, 
        bnb_4bit_quant_type="nf4", 
        bnb_4bit_compute_dtype=torch.float16, 
        bnb_4bit_use_double_quant=True 
    )
    
    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        quantization_config=bnb_config,
        device_map="auto" 
    )
    model.config.use_cache = False 

    tokenizer = AutoTokenizer.from_pretrained(model_name, padding_side='left')
    tokenizer.pad_token = tokenizer.eos_token 

    return model, tokenizer

model, tokenizer = load_model_and_tokenizer(model_name)

if torch.cuda.is_available():
    print("Model loaded on GPU by bitsandbytes and device_map='auto'.")
else:
    print("CUDA not available. Model running on CPU (bitsandbytes might not fully accelerate).")

model_inputs = tokenizer(
    "A list of colors: red, blue, 'Portugal is'",
    return_tensors="pt",
    padding=True
).to(model.device) 

with torch.no_grad():
    output_model = model(**model_inputs, output_hidden_states=True)
my_hidden_states = output_model.hidden_states 
print(f"Shape of my_hidden_states (from model()): Last layer tensor shape: {my_hidden_states[-1].shape}") 

generated_ids = model.generate(
    **model_inputs,
    output_hidden_states=True,
    return_dict_in_generate=True,
    max_new_tokens=1, 
    output_scores=True
)
generated_hidden_states = generated_ids.hidden_states 

# Debugging prints for generated_hidden_states
print(f"\n--- Debugging generated_hidden_states (from model.generate()) ---")
print(f"Type of generated_hidden_states: {type(generated_hidden_states)}")
print(f"Number of decoding steps (elements in outer tuple): {len(generated_hidden_states)}")
print(f"Type of generated_hidden_states[0] (first decoding step): {type(generated_hidden_states[0])}")
print(f"Number of layers (elements in inner tuple for step 0): {len(generated_hidden_states[0])}")
print(f"Shape of generated_hidden_states[0][0] (first layer's tensor for step 0): {generated_hidden_states[0][0].shape}")
print(f"Shape of generated_hidden_states[0][-1] (last layer's tensor for step 0): {generated_hidden_states[0][-1].shape}")

# Compare hidden states (cosine similarity)
hidden_state_model = my_hidden_states[-1][0, -1, :] 
hidden_state_generate = generated_hidden_states[0][-1][0, -1, :] # Corrected indexing

cos_sim = F.cosine_similarity(hidden_state_model, hidden_state_generate, dim=0)

print(f"\nCosine similarity (model() vs model.generate()): {cos_sim.item():.6f}")

My Results:

Model loaded on GPU by bitsandbytes and device_map='auto'.
Shape of my_hidden_states (from model()): Last layer tensor shape: torch.Size([1, 14, 1024])

--- Debugging generated_hidden_states (from model.generate()) ---
Type of generated_hidden_states: <class 'tuple'>
Number of decoding steps (elements in outer tuple): 2
Type of generated_hidden_states[0] (first decoding step): <class 'tuple'>
Number of layers (elements in inner tuple for step 0): 25
Shape of generated_hidden_states[0][0] (first layer's tensor for step 0): torch.Size([1, 14, 1024])
Shape of generated_hidden_states[0][-1] (last layer's tensor for step 0): torch.Size([1, 14, 1024])

Cosine similarity (model() vs model.generate()): 1.000000

As you can see, the cosine similarity I obtained is 1.000000. This suggests that the discrepancy reported in the original issue (where it was 0.9980) might have been resolved in a more recent update to the transformers library.

Given these results, I believe this issue can potentially be closed as resolved.

Thank you for your time!

[okihnjo]

Thank you very much for this detailed analysis! Does this generally mean, I can use the embeddings from my model.generate()method in order to analyze the embeddings ? Because that way I could generate text and also collect the important embeddings without making another forward pass.

Or should I stick to the model()method, as a high cosine similarity doesn't necessarily mean numerical equality ?

[akshathmangudi]

Hi,

I would like to answer your question since no one has yet. model.generate() is convenient, but you're not guaranteed full control due to the differences in decoding, while model() uses full-sequence decoding, which leads to numerical deviations especially under quantization or long prompts. If you quantize, the deltas could accumulate across layers.

Regardless, I'd still recommend model() when your analysis relies on the final embedding, as that's what you wanted to do. If you're hesitating because you're considering the cost of a second forward pass, the cost will be marginal in comparison to the benefit of getting stable embeddings.

If this answers your question, I believe this issue can be closed.

Have a nice day.

[gspeter-max]

Exactly, I think too

[okihnjo]

Thank you all!