Code

Hybrid recurrent cache#13904

Code

Closed

gabe-l-hart wants to merge 24 commits intoggml-org:gg/kv-cache-simplify-part3ggml-org/llama.cpp:gg/kv-cache-simplify-part3from

gabe-l-hart:HybridRecurrentCachegabe-l-hart/llama.cpp:HybridRecurrentCacheCopy head branch name to clipboard

Commits

Commits on May 30, 2025

• kv-cache : simplify the "struct llama_kv_cache" interface

  Show description for 582792b
  ggerganov

  committed
  582792b

  Copy full SHA for 582792b

• kv-cache : revert the (n_swa + n_ubatch) change (for next PR)

  Show description for 99653c3
  ggerganov

  committed
  99653c3

  Copy full SHA for 99653c3

• kv-cache : some comments

  Show description for 052f3f3
  ggerganov

  committed
  052f3f3

  Copy full SHA for 052f3f3

• context : fix graph reserve for multiple sequences

  Show description for 5693eb6
  ggerganov

  committed
  5693eb6

  Copy full SHA for 5693eb6

• kv-cache : fix typo [no ci]

  ggerganov

  committed
  cb2175f

  Copy full SHA for cb2175f

• kv-cache : fix find_slot() logic for free slots

  Show description for 3c6b330
  ggerganov

  committed
  3c6b330

  Copy full SHA for 3c6b330

• llama : add TODO for deprecating the defrag API in the future

  ggerganov

  committed
  f98b8d0

  Copy full SHA for f98b8d0

• kv-cache : improve find_slot() using min/max seq pos info

  Show description for 7e6d403
  ggerganov

  committed
  7e6d403

  Copy full SHA for 7e6d403

• llama : handle aborts and compute errors

  Show description for 47e570c
  ggerganov

  committed
  47e570c

  Copy full SHA for 47e570c

• memory : extract state into llama_memory_state

  Show description for 2b984f4
  ggerganov

  committed
  2b984f4

  Copy full SHA for 2b984f4

• kv-cache : add comments

  Show description for f23e4cc
  ggerganov

  committed
  f23e4cc

  Copy full SHA for f23e4cc

• feat: Add llama_model_is_hybrid API call

  Show description for 3fd6dd5
  gabe-l-hart

  committed
  3fd6dd5

  Copy full SHA for 3fd6dd5

• feat: Add c++ side constants for attention layer indices hparam

  Show description for dbad513
  gabe-l-hart

  committed
  dbad513

  Copy full SHA for dbad513

• feat: Add support for distinguishing recurrent vs non-recurrent layers in hparams

  Show description for 453d253
  gabe-l-hart

  committed
  453d253

  Copy full SHA for 453d253

• feat: Auto-fill hparams.recurrent_layer_arr based on whether the model is recurrent

  Show description for 26e51f4
  gabe-l-hart

  committed
  26e51f4

  Copy full SHA for 26e51f4

• refactor: rename *_is_hybrid -> *_is_hybrid_recurrent

  Show description for 33a41f5
  gabe-l-hart

  committed
  33a41f5

  Copy full SHA for 33a41f5

• feat: Move layer_filter_cb up to llama_kv_cache

  Show description for 162639c
  gabe-l-hart

  committed
  162639c

  Copy full SHA for 162639c

• feat: Add layer filter to recurrent cache

  Show description for a886cc1
  gabe-l-hart

  committed
  a886cc1

  Copy full SHA for a886cc1

• fix: Fix indexing into k_l for recurrent cache with filter

  Show description for 5c149d2
  gabe-l-hart

  committed
  5c149d2

  Copy full SHA for 5c149d2

• fix: Use per-layer sizing everywhere in kv caches

  Show description for 4470221
  gabe-l-hart

  committed
  4470221

  Copy full SHA for 4470221

• feat: First pass at llama_kv_cache_hybrid_recurrent

  Show description for ec7695f
  gabe-l-hart

  committed
  ec7695f

  Copy full SHA for ec7695f

• feat: Construct hybrid recurrent cache for hybrid recurrent models

  Show description for 728f514
  gabe-l-hart

  committed
  728f514

  Copy full SHA for 728f514

• fix: Fix wrong bool condition for split equal in hybrid cache

  Show description for b58351e
  gabe-l-hart

  committed
  b58351e

  Copy full SHA for b58351e

• feat: Support hybrid recurrent cache in llm_graph_context

  Show description for 4a2709f
  gabe-l-hart

  committed
  4a2709f

  Copy full SHA for 4a2709f