Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Outlines's cache not reusable across vllm startup #1130

Open
Lap1n opened this issue Sep 4, 2024 · 1 comment
Open

Outlines's cache not reusable across vllm startup #1130

Lap1n opened this issue Sep 4, 2024 · 1 comment
Labels
vLLM Things involving vLLM support

Comments

@Lap1n
Copy link

Lap1n commented Sep 4, 2024

Describe the issue as clearly as possible:

When using vllm and outlines, when running it from a VM, it seems that the diskcache functionality is not working correctly. Every time the server is startup, it doesn't seem to be able to reuse the previously computed FSM cache.

One way that can fix this issue is to serialize the cache key object as a string.
The changes can be found in this PR that I submitted.

Steps/code to reproduce the bug:

- Start vllm server
- send a request
- FSM computation happens
- Stops and relaunch the server
- send a request
- FSM computation still happens

Expected result:

- Start vllm server
- send a request
- FSM computation happens
- Stops and relaunch the server
- send a request
- FSM computation does not happens as it is already in the cache

Error message:

No response

Outlines/Python version information:

Version information

``` (command output here) ```
Latest from main.

Context for the issue:

No response

@brandonwillard
Copy link
Member

brandonwillard commented Sep 12, 2024

The source of this issue appears to be vllm's use of outlines.cache on functions that are ultimately used as class methods. Those functions include the class type instances in their signatures and that affects caching (e.g. equality doesn't necessarily hold after deserialization of types between Python sessions).

See #1145.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
vLLM Things involving vLLM support
Projects
None yet
Development

No branches or pull requests

3 participants