You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When using vllm and outlines, when running it from a VM, it seems that the diskcache functionality is not working correctly. Every time the server is startup, it doesn't seem to be able to reuse the previously computed FSM cache.
One way that can fix this issue is to serialize the cache key object as a string.
The changes can be found in this PR that I submitted.
- Start vllm server
- send a request
- FSM computation happens
- Stops and relaunch the server
- send a request
- FSM computation does not happens as it is already in the cache
Error message:
No response
Outlines/Python version information:
Version information
```
(command output here)
```
Latest from main.
Context for the issue:
No response
The text was updated successfully, but these errors were encountered:
The source of this issue appears to be vllm's use of outlines.cache on functions that are ultimately used as class methods. Those functions include the class type instances in their signatures and that affects caching (e.g. equality doesn't necessarily hold after deserialization of types between Python sessions).
Describe the issue as clearly as possible:
When using vllm and outlines, when running it from a VM, it seems that the diskcache functionality is not working correctly. Every time the server is startup, it doesn't seem to be able to reuse the previously computed FSM cache.
One way that can fix this issue is to serialize the cache key object as a string.
The changes can be found in this PR that I submitted.
Steps/code to reproduce the bug:
Expected result:
- Start vllm server - send a request - FSM computation happens - Stops and relaunch the server - send a request - FSM computation does not happens as it is already in the cache
Error message:
No response
Outlines/Python version information:
Version information
Context for the issue:
No response
The text was updated successfully, but these errors were encountered: