✅ add tokenizer option for testing

[prashantgupta24]

Description

Add VLLM_SPYRE_TEST_TOKENIZER_LIST as an optional parameter while testing. This is used by tiny-granite models which need to supply tokenizer as an additional input to vllm.

export VLLM_SPYRE_TEST_MODEL_LIST=tiny-granite

export VLLM_SPYRE_TEST_TOKENIZER_LIST=ibm-granite/granite-3.2-8b-Instruct 

then

pytest -v -m "spyre and cb" tests

will start with

test_cb_handling[max_num_seqs(4)-num_prompts(4)-eager-model(models/tiny-granite), tokenizer(ibm-granite/granite-3.2-8b-Instruct)]
.
.
.

Also changes max_model_len to 256 in CB tests.

[github-actions]

👋 Hi! Thank you for contributing to vLLM support on Spyre.
Just a reminder: Make sure that your code passes all the linting checks, otherwise your PR won't be able to be merged. To do so, first install the linting requirements, then run format.sh and commit the changes. This can be done with uv directly:

uv sync --frozen --group lint --active --inexact

Or this can be done with pip:

uv pip compile --group lint > requirements-lint.txt
pip install -r requirements-lint.txt
bash format.sh

Now you are good to go 🚀

[prashantgupta24]

We want this to be changed less frequently while parameterizing tests

[prashantgupta24]

added prompt with len=1 for testing