Code

Async continuous batching loop#401

Code

Merged

aniketmaurya merged 19 commits intomainLightning-AI/LitServe:mainfrom aniket/optimize-loopLightning-AI/LitServe:aniket/optimize-loopCopy head branch name to clipboardJan 8, 2025

+1,321-1,003

Commits

Commits on Dec 18, 2024

• prefill after n steps

  aniketmaurya

  committed
  da91ef5

  Copy full SHA for da91ef5

Commits on Dec 19, 2024

• make queue pop no-wait

  aniketmaurya

  committed
  8290f3e

  Copy full SHA for 8290f3e

• run async loop

  aniketmaurya

  committed
  d7a0d5a

  Copy full SHA for d7a0d5a

• update

  aniketmaurya

  committed
  142d840

  Copy full SHA for 142d840

• async step

  aniketmaurya

  committed
  8f9995d

  Copy full SHA for 8f9995d

• update

  aniketmaurya

  committed
  e59ad9c

  Copy full SHA for e59ad9c

• update

  aniketmaurya

  committed
  15f7c53

  Copy full SHA for 15f7c53

• simplify

  aniketmaurya

  committed
  579ea4c

  Copy full SHA for 579ea4c

• update

  aniketmaurya

  committed
  cfe547d

  Copy full SHA for cfe547d

• update

  aniketmaurya

  committed
  8eb7f19

  Copy full SHA for 8eb7f19

• update

  aniketmaurya

  committed
  b5ff91a

  Copy full SHA for b5ff91a

Commits on Jan 7, 2025

• Merge branch 'main' into aniket/optimize-loop

  aniketmaurya

  authored
  9921ff8

  Copy full SHA for 9921ff8

• reorg

  aniketmaurya

  committed
  5747809

  Copy full SHA for 5747809

• fix tests

  aniketmaurya

  committed
  b373afc

  Copy full SHA for b373afc

• fix tests

  aniketmaurya

  committed
  e06aa37

  Copy full SHA for e06aa37

• refactor

  aniketmaurya

  committed
  d67337a

  Copy full SHA for d67337a

• fix test

  aniketmaurya

  committed
  9245059

  Copy full SHA for 9245059

• fix hanging tests

  aniketmaurya

  committed
  359e83b

  Copy full SHA for 359e83b

• simple test

  aniketmaurya

  committed
  6cec895

  Copy full SHA for 6cec895