Use hugging face as baseline to test CB output

[sducouedic]

Closes issue #168

@joerunde About the llama issue, maybe we can wait until and e2e image gets published with CB enabled before removing it from the list of model for cb? it won't be used in the meantime in the CI/CD anyway, and this allows to test it on cpu for now, what do you think?

[github-actions]

👋 Hi! Thank you for contributing to vLLM support on Spyre.
Just a reminder: Make sure that your code passes all the linting checks, otherwise your PR won't be able to be merged. To do so, first install the linting requirements, then run format.sh and commit the changes. This can be done with uv directly:

uv sync --frozen --group lint --active --inexact

Or this can be done with pip:

uv pip compile --group lint > requirements-lint.txt
pip install -r requirements-lint.txt
bash format.sh

Now you are good to go 🚀

[joerunde]

About the llama issue, maybe we can wait until and e2e image gets published with CB enabled before removing it from the list of model for cb? it won't be used in the meantime in the CI/CD anyway, and this allows to test it on cpu for now, what do you think?

I think we can leave it in the list, it sounds like we should be fine to test llama models on cpu going forward. We can update the test configs over on the spyre hardware to swap to granite models

[prashantgupta24]

I'm going to request changes just so that we can pause more changes for CB. Right now there's a ton of uncertainty surrounding CB and we don't want to make any changes until we have more clarity

[sducouedic]

I'm going to request changes just so that we can pause more changes for CB. Right now there's a ton of uncertainty surrounding CB and we don't want to make any changes until we have more clarity

@prashantgupta24 mmh since the code change has already be done, and don't fundamentally change the logic of the tests (just a bit of refactoring and better output comparison), I would prefer still to merge that branch, instead of keeping it aside for a long time.

[prashantgupta24]

The changes that you have here are amazing! But there's still some uncertainty around the output that continuous batching is generating on the actual spyre hardware. Ideally, we want to get to the point where this PR works as is, But clearly both llama and granite models are not generating output which is gonna match what HF is producing at the moment on spyre.

Probably worth spending a bit of time understanding why and see if we want to push a "working" implementation which doesn't produce good results in the meantime (Which could be that we make sure that there is some output). In that case, we will not be able to merge this PR as is because the outputs will never match HF. That's the only reason I recommended a pause for a little while until the investigation gets us more insight?

[sducouedic]

@prashantgupta24 ah I think there is some confusion, the tests were not passing because of some breaking changes in vllm upstream (a new parameter introduced again), fixed in this PR #245. I just synced with main and all the tests are passing now (at least on the CPU)

[prashantgupta24]

@prashantgupta24 ah I think there is some confusion, the tests were not passing because of some breaking changes in vllm upstream (a new parameter introduced again), fixed in this PR #245. I just synced with main and all the tests are passing now (at least on the CPU)

I am talking about the tests failing on the actual spyre hardware for CB with the latest changes.

[sducouedic]

Ah ok I see. Fundamentally it doesn't really change to compare with HF output or a hardcoded ground truth. But yes let's freeze the code if that can simplify debugging for spyre 👍

[yannicks1]

LGTM

[prashantgupta24]

LGTM! We have a plan on how to handle testing with spyre