Blog: Inside vLLM’s New Offloading Connector

[orozery]

No description provided.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[esmeetu]

Maybe we can drop this benchmark script with small change into https://github.com/vllm-project/vllm/tree/main/benchmarks.

[orozery]

Good idea! I think this script is useful and will get more exposure there.
However, it will take some time to get it in there, and I think getting the blog post now is a higher priority.

[esmeetu]

Thanks for the great contribution! Along with the comments above, it might be helpful to add a few more offloading connector use cases at the end of the blog.
Adding a Slack channel could also help the community collaborate and iterate on this over time.

[orozery]

Thanks for the great contribution! Along with the comments above, it might be helpful to add a few more offloading connector use cases at the end of the blog. Adding a Slack channel could also help the community collaborate and iterate on this over time.

Thanks for reviewing!
I don't have any more use-cases to elaborate on at this point.
I've added a section at the end to engage community discussion (including a Slack channel).

[esmeetu]

LGTM! Thanks @orozery

[mgoin]

I feel like the title should be "KV Offloading" since it isn't clear if we are talking about model offloading

[esmeetu]

Agreed. This applies to the blog name as well.
@orozery Could we update this in another PR before we promote on social medias?