[Feature] LRU Eviction Strategy for Lora Adapters: Evicting Adapters with Priority

[whybeyoung]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

In scenarios where Lora adapters are shared, since a single base model is used collectively and there is an upper limit on the number of Lora adapters that a single node can host, we may need to introduce an LRU strategy (similar to VLLM's approach) to evict some adapters. However, the reactivation and deactivation of adapters can affect service quality (SLOs), which may be unacceptable in certain production environments. Therefore, a feature is required to ensure that specific adapters are not automatically evicted by the LRU mechanism.

Related resources

CC @Fridge003 @lifuhuang @lw9527

[lifuhuang]

@whybeyoung , thank you for creating the feature request!

Is my understanding correct that there are two features:

1. LRU eviction: today our "eviction algorithm" is trivial - we essentially evict based on buffer index order. We need to introduce an LRU-based eviction strategy.
2. adapter pinning: we need a way to tell server that certain adapters should stay in GPU and never be evicted.

Is that the right understanding of your problem? If so, how would you prioritize between this two?

Also, just for my curiosity, how do you decide (from application PoV) what adapters should be "pinned"?

Thanks!

[whybeyoung]

@whybeyoung , thank you for creating the feature request!

Is my understanding correct that there are two features:

1. LRU eviction: today our "eviction algorithm" is trivial - we essentially evict based on buffer index order. We need to introduce an LRU-based eviction strategy.
2. adapter pinning: we need a way to tell server that certain adapters should stay in GPU and never be evicted.

Is that the right understanding of your problem? If so, how would you prioritize between this two?

Also, just for my curiosity, how do you decide (from application PoV) what adapters should be "pinned"?

Thanks!

correct！