[Usage]: how to get last hidden states？vllm dont has api like transformers.

[JustinZhao0622]

Your current environment

/root/anaconda3/envs/llm/lib/python3.11/site-packages/vllm/model_executor/layers/sampler.py

logits = _apply_min_tokens_penalty(logits, sampling_metadata)

    # Apply presence and frequency penalties.
    if do_penalties:
        logits = _apply_penalties(logits, sampling_tensors.prompt_tokens,
                                  sampling_tensors.output_tokens,
                                  sampling_tensors.presence_penalties,
                                  sampling_tensors.frequency_penalties,
                                  sampling_tensors.repetition_penalties)

    # Use float32 to apply temperature scaling.
    # Use in-place division to avoid creating a new tensor.
    logits = logits.to(torch.float)
    logits.div_(sampling_tensors.temperatures.unsqueeze(dim=1))

How would you like to use vllm

I want to run inference of a [specific model](put link here). I don't know how to integrate it with vllm.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

This is not supported in general because it will harm the performance (so you might as well just use transformers library). See #18176

[JustinZhao0622]

Thanks for the reply!
I'm a newbie in the direction of researching large models. a lot of experiments are done based on vllm, so I wanted to try it out.
Code in the execute_model method under ModelRunner Class in /vllm/worker/model_runner.py:

            # we only need to pass hidden states of most recent token
            assert model_input.sampling_metadata is not None
            indices = model_input.sampling_metadata.selected_token_indices
            if model_input.is_prompt:
                hidden_states = hidden_or_intermediate_states.index_select(
                    0, indices)
                output.prefill_hidden_states = hidden_or_intermediate_states
            elif decode_meta.use_cuda_graph:
                hidden_states = hidden_or_intermediate_states[:len(indices)]
            else:
                hidden_states = hidden_or_intermediate_states

            output.hidden_states = hidden_states
            print(output.hidden_states)
        return [output]

Is the return the last hidden states please?
Looking forward to your reply! @DarkLight1337

[DarkLight1337]

You should use triple (not just one) backticks to format code blocks spanning more than one line.

Your code snippet looks like V0 code which will be removed shortly:

        if self.return_hidden_states:
            # we only need to pass hidden states of most recent token
            assert model_input.sampling_metadata is not None
            indices = model_input.sampling_metadata.selected_token_indices
            if model_input.is_prompt:
                hidden_states = hidden_or_intermediate_states.index_select(
                    0, indices)
                output.prefill_hidden_states = hidden_or_intermediate_states
            elif decode_meta.use_cuda_graph:
                hidden_states = hidden_or_intermediate_states[:len(indices)]
            else:
                hidden_states = hidden_or_intermediate_states

            output.hidden_states = hidden_states

        return [output]

I suggest looking at the V1 model runners instead.

[JustinZhao0622]

Thank you for your reply.
I've used transformers and clearly felt that vllm is much faster than this while using it, thanks for your contribution to the community!@DarkLight1337