Dynamo v0.5.1

Dynamo Release v0.5.1

Dynamo is a high-performance, low-latency inference framework designed to serve generative AI models at data-center scale. We're an open-source first project under the Apache 2.0 license; built in Rust for performance and Python for extensibility. Dynamo is available for installation via pip wheels and containers from NVIDIA NGC.

Dynamo supports multiple large language model (LLM) inference engines (see Support Matrix for details):

• NVIDIA TensorRT-LLM
• vLLM
• SGLang

Release Highlights

This release delivers major advances in KV routing capabilities with the new vLLM prefill router and commit router, comprehensive canary health checks across all backends, and significant tool calling enhancements. We strengthened production reliability with request cancellation support, improved Kubernetes deployment workflows, and expanded multinode capabilities. Lastly, we enhanced KVBM performance with vectorized memory transfers and tighter integration with TensorRT-LLM v1.1.0rc5.

Major Features and Improvements

1. Advanced KV Routing & Cache Management

KV Router

• Introduced vLLM prefill router for optimized prefill phase handling (#3155)
• Implemented KV commit router for improved cache consistency (#3024)
• Added router benchmarking capabilities with mooncake-style testing (#3068, #2828)
• Enabled router to optionally skip tracking active blocks during prefill and cached blocks during decode (#3135)
• Router replicas with state-sharing for improved scalability (continued from v0.4.1)

KVBM (KV Block Manager)

• Implemented vectorized copy between pinned memory and device memory for improved transfer performance (#2989)
• Enhanced KVBM transfer context v2 (#2873)
• Added KV indexer metrics for better observability (#2905)
• Updated integration with TensorRT-LLM v1.1.0rc5 connector API (#2979, #3119)
• Improved error handling with early stop for missing CPU/disk configuration (#2997)

2. Enhanced Health Checks & Reliability

Canary Health Checks

• Implemented canary health check framework (#2903)
• Added TensorRT-LLM canary health check with BOS token support (#3082, #3145)
• Deployed SGLang canary health check (#3103, #3123)
• Enabled vLLM prefill-specific health check payload (#3126)

Request Management

• Added request cancellation support for unary requests (#3004)
• Enabled vLLM abort while engine generates next token (#3102)
• Implemented router-level request rejection for better resource management

3. Tool Calling & Reasoning Enhancements

• Enabled tool calling with stream=True support (#2932)
• Added Deepseek V3.1 tool parser with library refactoring (#2832)
• Implemented Granite class reasoning parser (#2936)
• Enhanced GPT-OSS frontend with Harmony tool calling and reasoning parsers (#2999)
• Added finish reason tool_calls for non-streaming responses (#3087)
• Fixed null tools processing via minijinja (#3340)

4. Kubernetes & Deployment Improvements

Grove Integration

• Updated to official Grove 0.1.0-alpha release (#3030)
• Added planner manifest support for Grove (#3203)

Deployment Enhancements

• Installed Dynamo operator cluster-wide by default (#3199)
• Added multinode K8s examples for TensorRT-LLM and vLLM (#3100)
• Enabled in-cluster performance benchmarks with kubectl one-liner (#3144)
• Implemented namespace isolation for improved multi-tenancy (#2394, #2970)
• Added virtual connector for 3rd party deployments (#2913)
• Improved SGLang multinode handling in operator (#3151)

5. Observability & Metrics

• Added HTTP queue metrics for NIM frontend request tracking (#2914)
• Implemented NIM FE runtime config metrics with periodic polling (#3107)
• Added metrics labels for multimodal workloads (#2835)
• Implemented frontend disconnect metrics (#2953)
• Unified component metric names to prevent Kubernetes label collisions (continued from v0.4.1)

6. Frontend & Model Support

• Added support for serving multiple models from single endpoint (continued from v0.4.1)
• Implemented --custom-jinja-template argument for custom chat templates (#2829)
• Added chat_template_kwargs parameter to v1/chat/completion (#3016)
• Enabled framework tokenization/detokenization (#3134)
• Implemented ModelExpress Dynamo integration (#3191)
• Added SLA Planner support for TensorRT-LLM (#2980) and SGLang MoE models (#3185)

7. Performance & Optimization

• Refactored discovery ModelManager to use parking_lot::RwLock (#2902)
• Ported vLLM port allocator to Rust bindings for improved performance (#3125)
• Implemented JailedStream for better resource management (#3034)
• Added generic tensor type for inference (#2746)
• Updated benchmarking and deployment utilities (#2933, #2973, #3098)

8. Bug Fixes

• Fixed OpenAI-compliant usage stats for streaming responses (#3022)
• Resolved token loss bug in final packet (#2985)
• Fixed aggregate logprobs calculation (#2928)
• Corrected Harmony parser streaming behavior (#3074)
• Fixed router slot manager force expire requests (#2840)
• Resolved metrics collection and namespace sanitization issues (#2868)
• Fixed polling from exhausted stream in preprocessor (#3349)
• Addressed KVBM fully contiguous memory region size bug (#3175)

Documentation

• Revamped Kubernetes documentation (#3173)
• Created deployment and benchmarking recipes for Llama3-70B and GPT-OSS-120B (#2792)
• Added AWS ECS deployment example for Dynamo vLLM (#2415, #3381)
• Published Python runtime request cancellation examples (#2893)
• Added health check and structured logs documentation (#2805)
• Created mermaid diagrams showcasing KV router features (#3184)
• Updated consistent hashing documentation for KV events (#2981)
• Published profiling-related documentation updates (#2816)
• Fixed broken links and Sphinx structural errors (#3186, #3342)

Build, CI, and Test

• Restructured TensorRT-LLM and SGLang to follow container strategy structure (#3009, #2803)
• Moved to ARC runners for CI (#2904)
• Added SGLang functional tests (#2943)
• Implemented fault injection tests for Kubernetes (#3194)
• Added concurrency checks to auto-cancel running actions (#2438)
• Created broken links checker (#2927)
• Converted vLLM multimodal examples to pytest framework (continued from v0.4.1)
• Updated TensorRT-LLM to v1.1.0rc5 (#3119)

Migration Notes

• Component metric names continue to use the dynamo_component_* pattern. Ensure dashboards and alerting rules are updated accordingly.
• The Dynamo operator now installs cluster-wide by default. If namespace-scoped installation is required, use the appropriate Helm values.
• TensorRT-LLM has been updated to v1.1.0rc5, which includes KVBM integration changes. Review the updated connector API if using custom integrations.
• The Multinode Multimodal Guide works only with release v0.5.0. Users requiring multinode multimodal functionality should continue using v0.5.0 until support is restored in a future release.

Looking Forward
This release strengthens Dynamo's production readiness with advanced KV routing, comprehensive health monitoring, and robust request management. The enhanced Kubernetes integration and multinode support enable seamless scaling for enterprise deployments. With improved observability and the new prefill router, teams can now optimize both throughput and latency for diverse workload patterns. These capabilities set the stage for even more sophisticated routing strategies and performance optimizations in future releases!

Release Assets
Python Wheels:

• ai-dynamo v0.5.1
• ai-dynamo-runtime v0.5.1
  Rust Crates:
• dynamo-runtime v0.5.1
• dynamo-async-openai v0.5.1
• dynamo-llm v0.5.1
• dynamo-parsers v0.5.1

Containers:

• TensorRT-LLM Runtime: nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:0.5.1 NGC
• vLLM Runtime: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.5.1 NGC
• SGLang Runtime: nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.5.1 NGC
• Dynamo Kubernetes Operator: nvcr.io/nvidia/ai-dynamo/kubernetes-operator:0.5.1 NGC

Helm Charts:

• dynamo-crds-0.5.1
• dynamo-platform-0.5.1
• dynamo-graph-0.5.1

Contributors
We welcome new contributors in this release: @blarson-b10, @lixuwei2333, @GavinZhu-GMI, @nv-hwoo
Full Changelog: v0.5.0...v0.5.1

Dynamo Release v0.5.0

Dynamo 0.5.0 Release Notes

Dynamo is a high-performance, low-latency inference framework designed to serve generative AI models—across any framework, architecture, or deployment scale. It's an open-source project under the Apache 2.0 license. Dynamo is available for installation via pip wheels and containers from NVIDIA NGC.

Dynamo supports multiple large language model (LLM) inference engines (see Support Matrix for details).

• NVIDIA TensorRT-LLM
• vLLM
• SGLang

Release Highlights

This release introduces TRT-LLM integration for KV cache management, supports gRPC support and tool calling capabilities. We also delivered major improvements to system reliability, with request cancellation features and improved observability.

---

Major Features and Improvements

1. Fault Tolerance & Observability

• Implemented End to End request cancellation (#2158, #2500) with Python context propagation
• Implemented DRT shutdown on vLLM engine failures (#2698)
• Added fast-fail validation for NATS JetStream requirements to prevent silent failures (#2590)
• Unified metrics across all components with model labels for vLLM (#2474), TensorRT-LLM (#2666), and SGLang (#2679)
• Standardized Prometheus metrics naming and sanitization with KvStats integration (#2733, #2704)
• Added automatic uptime tracking and auto-start of metrics collection upon NATS service creation (#2587, #2664), improving observability readiness

2. Kubernetes Deployments

• Integrated Grove and KAI scheduler into Dynamo Cloud Helm chart for multi-node deployments (#2755)
• Implemented auto-injection of kai-scheduler annotations and labels with parent DGD Kubernetes name support (#2748, #2774)
• Deployed Dynamo EPP-aware gateway with prevention of double-tokenization for optimized routing (#2633, #2559)
• Integrated Model Express client for optimized model downloads with URL injection support (#2574, #2769)

3. KV Cache Management & Transfer

• Integrated Dynamo KVBM connector API with TensorRT-LLM for G2-G3 offloading and onboarding (#2544)
• Added support for user selection among multiple KV transfer connectors (nixl, kvbm, lmcache) (#2517)
• Added detailed KV Block Manager metrics for match, offload, and onboard operations (#2626, #2673)

4. Planning & Routing

Router

• Separated frontend and Router, through Python bindings for KvPushRouter, so the Router and frontend can be scaled independently (#2658, #2548)
• Implemented warm restarts via durable KV event consumers and radix snapshotting for router persistence (#2756, #2740, #2800)

Planner

• Added comprehensive tests for replica calculation and planner scaling with automated Kubernetes deployment validation (#2525)
• Added SLA planner dry-run mode with a CLI to simulate workloads, generate plots, and expose optional Prometheus metrics (#2557)

5. Others

Tool Calling

• Introduced parsers library (#2542) supporting multiple reasoning and tool-calling formats.
• Implemented multiple tool-calling parsers, including Pythonic (#2788), Harmony (#2796), and JSON-based parsers with normal text parsing alongside tool calls (#2709)
• Added support for separating reasoning from visible text (#2555) along with GPT-OSS reasoning parser integration (#2656)
• Added support for custom logits processors in the TensorRT-LLM backend, enabling in-place logits modification during generation (#2613, #2702)

Multimodal Support Expansion

• Added complete multimodal deployment examples for Llava and Qwen, with video support using vLLM v1 (#2628, #2694, #2738)
• Added Encode Worker and NIXL support for TensorRT-LLM multimodal disaggregated flows (#2452)

Infrastructure & Performance

• Added comprehensive KServe gRPC support for industry-standard model inference protocol (#2638)
• Enhanced Hugging Face integration with HF_HOME and HF_ENDPOINT environment variable support (#2642, #2637)

Developer Experience

• Added Devcontainer improvements with enhanced documentation and SGLang-specific setup (#2255, #2578, #2741)
• Added logging setup for Kubernetes with Loki integration and Grafana dashboards (#2699)
• Added benchmarking guide with GenAI-Perf integration and automated performance comparison (#2620)
• Updated TensorRT-LLM to 1.0.0rc6 and simplified Eagle model configuration (#2606, #2661)

Bug Fixes

• Improved Hugging Face download speeds with better API client configuration (#2566)
• Added missing Prometheus to runtime images for SGLang and general runtime (#2565, #2689)
• Fixed kv-event-config command line respect and environment variable overrides (#2627, #2640)
• Enhanced pytest robustness and parsing errors with proper timeout handling (#2676, #2572)
• Resolved metrics registration timing issues and prevented early returns from affecting measurements (#2664, #2576)

Documentation

• Created SNS aggregated Kubernetes example and simplified sphinx build process (#2773, #2519)
• Streamlined cloud installation documentation and deployment guides (#2818)
• Updated benchmarking framework documentation with comprehensive deployment guides (#2620)
• Updated supported models documentation for multimodal and SGLang container build instructions (#2651, #2707)

Build, CI, and Test

• Added replica calculation and planner scaling tests with automated Kubernetes deployment validation (#2525)
• Added vLLM sanity testing support on GitHub Actions with build optimizations (#2526)
• Optimized CI job execution for docs-only changes and Rust-specific changes (#2775)
• Enabled KVBM in vLLM container with improved virtual environment handling (#2763)
• Enhanced test reliability with proper KVBM test exclusions and determinism testing (#2611)
• Fixed concurrency settings to prevent main branch run cancellations (#2780)
• Improved container build process with default dev builds for vLLM (#2837)

Migration Notes

• Parser Integration: New parsing capabilities require updated CLI flags for reasoning and tool calling features
• Container Updates: Runtime images now include Prometheus by default - review monitoring configurations

Looking Forward

This release sets the stage for more features in our H2 roadmap, including benchmarking KVBM performance, E2E performance, and improved fault tolerance and request rejection at every level. We will focus on significantly updating documentation and examples for a better experience and include Kubernetes benchmark scripts for most popular models.

Release Assets

Python Wheels:

• ai-dynamo v0.5.0
• ai-dynamo-runtime v0.5.0

Rust Crates:

• dynamo-runtime v0.5.0
• dynamo-async-openai v0.5.0
• dynamo-llm v0.5.0
• dynamo-parsers v0.5.0

Containers:

• nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.5.0
• nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:0.5.0
• nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.5.0
• nvcr.io/nvidia/ai-dynamo/kubernetes-operator:0.5.0

Helm Charts:

• dynamo-crds-0.5.0
• dynamo-platform-0.5.0
• dynamo-graph-0.5.0

---

Contributors

We welcome new contributors in this release:
@jasonqinzhou, @michaelfeil, @ahinsutime, @bhuvan002, @WaelBKZ, @hhk7734, @Michaelgathara, @KavinKrishnan, @michaelshin

Full Changelog: v0.4.1...v0.5.0

Dynamo Release v0.4.1

Dynamo is a high-performance, low-latency inference framework designed to serve generative AI models—across any framework, architecture, or deployment scale. It's an open-source project under the Apache 2.0 license. Dynamo is available for installation via pip wheels and containers from NVIDIA NGC.

Dynamo supports multiple large language model (LLM) inference engines (see Support Matrix for details)

• NVIDIA TensorRT-LLM
• vLLM
• SGLang

Release Highlights

This release brings substantial performance improvements for Deepseek R1, improved fault tolerance capabilities with high availability router testing, and groundbreaking KV cache management features. We've also significantly enhanced our Kubernetes deployment story with Grove integration and the new Inference Gateway, while expanding multimodal support across multiple backends.

---

Major Features and Improvements

1. Model Performance Breakthroughs

• Achieved significant Deepseek R1 wideEP performance with both SGLang (#2223) and TRT-LLM (#2387)
• Added in TRT-LLM support for variable sliding window attention (VSWA) for Gemma3 models (#2134)
• Launched Day0 support + deployment guide for GPT-OSS 120B on Blackwell GPUs (#2297)

2. Fault Tolerance & Observability Improvements

• Introduced testing for multiple KV routers and frontends for high availability (#2324)
• Completed end-to-end request migration testing with vLLM (#2177), ensuring seamless failover
• Added router-level request rejection (#2465) for better resource management under load
• Unified NATS, DRT & component metrics (#2292) for comprehensive system monitoring
• Made health checks more flexible with parameterized /health and /live endpoints (#2230)

3. Enhanced Kubernetes Deployments

Grove

• Unlocked multi-node support through Grove integration (#2269, #2405)
• Provided workaround for component scaling when using Grove (#2531)

Inference Gateway

• Launched Dynamo integration with API Gateway featuring EPP customization (#2345)

4. Advanced KV Cache Management & Transfer

KV Block Manager

• First release of KV Block Manager (KVBM) with vLLM, supporting tiered storage across HBM (G1), host memory (G2), and local disk (G3) (#2258)

LMCache integration

• Successfully integrated LMCache for improved cache efficiency (#2079)

5. Intelligent Planning & Routing

Router

• Enabled router replicas with state-sharing for improved scalability (#2264)

Planner

• Extended SLA Planner integration to support SGLang dense models (#2421)

6. Others

Multimodal model support

• Shipped multimodal examples with vLLM v1 (#2040)
• Added comprehensive Llava model deployment example with vLLM v1 (#2628)
• Brought multimodal support to TRT-LLM backend (#2195)

Guided decoding

• Implemented frontend support for Structured Output and Guided Decoding (#2380)

Frontend improvements

• Added capability to serve multiple models from a single endpoint (#2418)
• Introduced LLM metrics for non-streaming requests (#2427)

Bug fixes

• Resolved metrics collection timeout issues (#2480, #2506)
• Standardized component metric names to dynamo_component_* pattern, preventing Kubernetes label collisions (#2180)
• Fixed runtime error propagation in endpoint.rs (#2156)
• Corrected processor/router unit queuing behavior with NATS (#1787)
• Added missing dependencies to SGLang runtime build (#2279)
• Improved HuggingFace token handling in preprocessor tests (#2321)
• Implemented detokenize stream functionality (#2413)

Documentation

• Created comprehensive TRT-LLM deployment examples for Kubernetes (#2133)
• Authored SGLang deployment guide (#2238)
• Developed MetricsRegistry API guides (#2159, #2160)
• Published guide for collecting and viewing Dynamo metrics in Kubernetes (#2271)
• Released Dynamo Inference Gateway documentation (#2257, #2260)
• Created SGLang hicache example and guide (#2388)

Build, CI, and Test

• Implemented KV routing tests for SGLang (#2424)
• Completed request migration end-to-end testing with vLLM (#2177)
• Converted vLLM multimodal example to pytest framework (#2451)
• Added ZMQ library support for TRT-LLM's UCX connection establishment (#2381)
• Created unit tests for SLA planner's interpolator (#2505)

Migration Notes

Component metric names have been standardized to the dynamo_component_* pattern. Users monitoring these metrics should update their dashboards and alerting rules accordingly.

Looking Forward

This release sets the foundation for even more ambitious features in our H2 roadmap. The new KV cache management capabilities and multi-node support open doors for larger-scale Dynamo deployments, while our enhanced observability features ensure you can confidently run Dynamo in production.

Release Assets

Python Wheels:

• ai-dynamo v0.4.1
• ai-dynamo-runtime v0.4.1

Rust Crates:

• dynamo-runtime v0.4.1
• dynamo-async-openai v0.4.1
• dynamo-llm v0.4.1

Containers:

• nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.4.1
• nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:0.4.1
• nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.4.1
• nvcr.io/nvidia/ai-dynamo/kubernetes-operator:0.4.1

Helm Charts:

• dynamo-crds-0.4.1
• dynamo-platform-0.4.1
• dynamo-graph-0.4.1

---

Contributors

We welcome new contributors in this release:
@qimcis, @yinggeh, @da-x, @elyasmnvidian, @ryan-lempka, @JesseStutler, @nate-martinez, @suzusuzu

Full Changelog: v0.4.0...v0.4.1

Dynamo Release v0.4.0

Dynamo 0.4.0 Release Notes

Dynamo is a high-performance, low-latency inference framework designed to serve generative AI models—across any framework, architecture, or deployment scale. It's an open-source project under the Apache 2.0 license. Dynamo is available for installation via pip wheels and containers from NVIDIA NGC.

As a vendor-neutral serving framework, Dynamo supports multiple large language model (LLM) inference engines to varying degrees:

• NVIDIA TensorRT-LLM
• vLLM
• SGLang

---

Major Features and Improvements

Increasing Framework Support

• vLLM Updates

  • Added E2E integration tests (#1935) and multimodal example with Llama4 Maverick (#1990)
  • Prefill-aware routing for improved performance (#1895)
  • Configurable namespace support for vLLM examples (#1909)
  • Routing via ApproxKvIndexer with use_kv_events flag (#1869)
  • Updated all vLLM examples to new UX (#1756)

• SGLang Updates

  • Receive KV metrics from scheduler (#1789)
  • Disaggregated deployment examples (#2137)
  • Launch and deploy examples added (#2068)

• TRT-LLM Updates

  • New/speculative decoding example: Llama-4 + Eagle-3 (#1828)

• Routing Performance

  • Removed router hot-path lock for faster request handling (#1963)
  • Added radix tree dumps as router events (#2057)

---

UX Updates

• Migration to New Python UX

  • Updated all Python launch flows to the new UX structure (#2003), including refactoring vLLM backend integration (#1983).
  • Removed outdated examples that relied on the old UX (#1899).

• CLI and Packaging Enhancements

  • Added Python bindings for Dynamo CLI tools (#1799).
  • Updated Python packaging to align with the new UX (#2054).
  • Introduced a Python frontend/ingress node for easier deployment integration (#1912).
  • Added a convenience script to uninstall Dynamo Deploy CRDs (#1933).

• Kubernetes Deployment UX

  • Enhanced Helm chart flexibility:
    • Added ability to override any podSpec property (#2116).
    • Enabled Helm upgrade via deploy script for smoother iteration (#1936).
    • Added Grove scheduling support to the graph Helm chart (#1954).
  • Introduced Kubernetes deployment examples for vLLM, SGLang, and TRT-LLM (#2062, #2133).
  • New Hello World Kubernetes deployment example (#1854).

• Examples & Docs Overhaul

  • Hello World Python binding example (#2083).
  • Documentation updated for UX (#2070), reorganized example READMEs (#2174), and refactored core README structure (#2141).

---

Deployment, Kubernetes, and CLI

• Helm and Graph Deployments

  • Liveness/readiness probes in graph Helm chart (#1888)
  • Added ability to override any podSpec property (#2116)
  • Support for Grove scheduling in Helm (#1954)

• Planner and Profiling

  • Deploy SLA profiler and SLA planner to Kubernetes (#2030, #2135)

---

Performance and Observability

• Structured Logging Improvements

  • Enhanced structured JSONL logs with span start/close events, trace ID/span ID injection, duration formatting in microseconds, and improved context capture for distributed tracing workflows (PR #2061).

• Tokenizer & Runtime

  • De-tokenize performance improved by ~50% (#1868)
  • Runtime now uses all available parallelism (#1858)

• Metrics

  • Hierarchical Prometheus metrics registry (#2008)
  • Generic ingress handler metrics (#2090)

---

Bug Fixes

• Fixed GPU resource specifications in LLM deployments (#1812)
• Corrected vLLM, SGLang, and TRTLLM deployment issues, including container builds, runtime packaging, and helm chart updates (#1942, #2062, #1825)
• Addressed port conflicts, deterministic port assignments, and health check improvements (#1937, #1996)
• Improved error handling for empty message lists and invalid configurations (#2067, #2071)
• Fixed nil pointer dereference issues in the Dynamo controller (#2299, #2335)
• Locked dependencies to avoid breaking changes (e.g., Triton 3.4.0 w/ TRT-LLM 1.0.0) (#2233)

---

Documentation

• Guides and Examples

  • New hello world Python binding example (#2083)
  • Added multinode, disaggregated, and Grove deployment guides (#2155, #2086)
  • Added AKS/EKS deployment guides (#2080)

• Docs Restructuring

  • Updated for new Python UX (#2070)
  • Refactored README and reorganized examples (#2141, #2174)

---

Build, CI, and Test

• Added support for sGLang runtime image builds (#1770)
• Optional TRTLLM dependency and custom build support (#2113)
• New end-to-end router tests with mockers (#2073)
• Fixed vLLM builds for Blackwell GPUs (#2020)

---

Release Assets

Python Wheels:

• ai-dynamo v0.4.0
• ai-dynamo-runtime v0.4.0

Rust Crates:

• dynamo-runtime v0.4.0
• dynamo-llm v0.4.0

Containers:

• nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.4.0
• nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:0.4.0
• nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.4.0
• nvcr.io/nvidia/ai-dynamo/kubernetes-operator:0.4.0

Helm Charts:

• dynamo-crds-0.4.0
• dynamo-platform-0.4.0
• dynamo-graph-0.4.0

---

Open Issues

• x86 TRT-LLM container image not compatible out of the box with B200. Dev container still works for B200/GB200

---

Contributors

We welcome new contributors in this release:
@umang-kedia-hpe, @Ethan-ES, @messiaen, @galletas1712, @mc-nv, @zaristei, @jhaotingc, @saurabh-nvidia.

For the full list of changes, see the changelog.

Dynamo Release v0.3.2

Dynamo is a high-performance, low-latency inference framework designed to serve generative AI models—across any framework, architecture, or deployment scale. It's an open source project under the Apache 2.0 license. Dynamo is available for installation via pip wheels and containers from NVIDIA NGC.

As a vendor-neutral serving framework, Dynamo supports multiple large language model (LLM) inference engines to varying degrees:

• NVIDIA TensorRT-LLM
• vLLM
• SGLang

Major Features and Improvements

Engine Support and Routing

• Example standalone router for use outside of dynamo (#1409).
• The new SLA-based planner dynamically manages resource allocation based on service-level objectives (#1420).
• Data-parallel vLLM worker setups are now supported (#1513).
• SGLang support was extended for DeepEP deployments (#1120).
• Clean shutdown is now available for vllm_v1 and SGLang engines (#1562, #1764).
• Experimental support for WideEP with EPLB aggregation and disaggregation is now available for TRTLLM (#1652, #1690).
• Approximate KV cache residency and predicted active KV blocks for improved routing efficiency (#1636, #1638, #1731).

Observability and Metrics

• Native DCGM and Prometheus integration enables hardware metrics collection and export. Optional Grafana dashboards are provided (#1488, #1701, #1788).
• New Grafana dashboards offer composite software and hardware system visibility (#1788).
• Batch /completions endpoint and speculative decoding metrics are now supported for vLLM (#1626, #1549).

Deployment, Kubernetes, and CLI

• The Kubernetes operator now supports custom entrypoints, command overrides, and simplified graph deployments (#1396, #1708, #1877, #1893).
• Example manifests for multimodal and minimal deployments were added (#1836, #1872).
• Graph Helm chart logic, resource requests, and health probes were improved (#1877, #1888).
• Two new Helm charts are introduced in this release: dynamo-platform, and dynamo-crds, enabling modular and robust Kubernetes deployments for a variety of topologies and operational requirements.
• The dynamo-run command line interface now supports the --version flag and improved error handling and validation (#1596, #1674, #1623).
• Docker and Kubernetes deployment workflows were streamlined. Helm charts and container images were improved (#1742, #1796, #1840, #1841).

Developer Experience

• Embedding request handling was improved with frontend tokenization (#1494).
• OpenAI API request validation is now available (#1674).
• Batch embedding and parallel tokenization improve efficiency for batch inference and embedding (#1657).
• The /responses endpoint and additional API features were added (#1694).

Bug Fixes

• Issues related to GPU resource specifications in deployments, container builds, and runtime were fixed (#1826, #1792, #1546).
• Helm chart logic, resource requests, and health probes were corrected (#1877, #1893).
• Error handling and model loading were improved for multimodal and distributed deployments (#1545).
• Metrics publishing and logging were fixed for vLLM, SGLang, and OpenAI endpoints (#1864, #1649, #1639).
• Process cleanup issues were resolved in tests (#1801).

Documentation

• Documentation updates include new guides for Ray setup, architecture diagrams, and deployment modes (#1947, #1697).
• Benchmarking, troubleshooting, and advanced usage scenario documentation was enhanced.
• Notes were added to deprecate outdated connectors (#1964, #1959).

Build, CI, and Test

• Dependency upgrades include protobuf, nats, and etcd (#1876, #1744).
• CI coverage now includes GPU-based and multi-engine tests.
• Container builds now use distroless images for improved security and efficiency (#1570, #1569).
• Fault tolerance tests #1444

Known Issues

• KVBM is supported only with Python 3.12.

Release Assets

Python Wheels:

• ai-dynamo v0.3.2
• ai-dynamo-runtime v0.3.2
• ai-dynamo-vllm v0.8.4.post4

Rust Crates:

• dynamo-runtime v0.3.2
• dynamo-llm v0.3.2

Containers:

• nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.3.2
• nvcr.io/nvidia/ai-dynamo/tensorrtllm-runtime:0.3.2
• nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.3.2
• nvcr.io/nvidia/ai-dynamo/kubernetes-operator:0.3.2

Helm Charts:

• dynamo-crds-0.3.2
• dynamo-platform-0.3.2

Contributors

Thank you to all contributors for this release. For a full list, refer to the changelog.

Dynamo Release v0.3.1

Dynamo is an open source project under the Apache 2.0 license. The primary distribution is done through pip wheels with minimal binary size. The ai-dynamo GitHub organization hosts two repositories: Dynamo and NIXL. Dynamo is designed as the next-generation inference server, building upon the foundation of NVIDIA® Triton Inference Server™. While Triton focuses on single-node inference deployments, we're integrating its robust capabilities into Dynamo over the next several months. We'll maintain support for Triton while providing a clear migration path for existing users once Dynamo achieves feature parity.

As a vendor-neutral serving framework, Dynamo supports multiple large language model (LLM) inference engines to varying degrees:

• NVIDIA TensorRT-LLM
• vLLM
• SGLang

Dynamo v0.3.1 features:

• Functional DeepSeek R1 disaggregated serving with wide EP using SGLang
• Functional EPD disaggregation with video model (Llava video 7B)
• Proof of concept inference gateway support
• Prebuilt Dynamo + vLLM container
  • We plan to release these pre-built containers in the coming days
• Amazon Linux support

Future plans
Dynamo Roadmap

Known Issues

• KVBM is supported only with python 3.12

What's Changed

🚀 Features & Improvements

• feat: expose estimated kv cache hit in dynamo-run by @tedzhouhk in #1246
• feat: KVBM async Python bindings and Layer class by @kthui in #1141
• feat: add critical task execution handle by @ryanolson in #1268
• feat: Initial Granite support by @grahamking in #1271
• feat: Restructure kv manager block registration by @jthomson04 in #1093
• feat: Publish events and metrics when using kv routing by @tanmayv25 in #1262
• feat(dynamo-run): Use llama.cpp as the default engine for GGUF by @grahamking in #1276
• feat: populate default image name by @biswapanda in #1255
• feat: flatten out dynamo cloud helm chart by @julienmancuso in #1258
• refactor: Refactor kv event publishers by @jthomson04 in #1287
• refactor: rename KvMetricsPublisher to WorkerMetricsPublisher by @alec-flowers in #1284
• feat: all blocks cleared event by @jain-ria in #1279
• perf: Create default sampling params only once during initialization by @krishung5 in #1294
• feat: expose router configurations to dynamo-run by @tedzhouhk in #1259
• feat: Make llama.cpp Gnu OpenMP dependency optional by @grahamking in #1331
• feat: set env variables in Dynamo deployments from secrets by @hhzhang16 in #1325
• feat: Add DSR1 configurations by @ptarasiewiczNV in #1298
• feat: add more metrics to rust frontend by @tedzhouhk in #1315
• feat: Enable disagg support in trtllm standalone script by @tanmayv25 in #1355
• feat: Integrate KVBM with CriticalTaskHandle by @jthomson04 in #1321
• feat: add implementation for embeddings by @t-ob in #1290
• feat: refactor docker registry secret management in operator by @julienmancuso in #1337
• feat: set model specific prompt templates in the multimodal config files, add documentation for multimodal example deployment by @hhzhang16 in #1366
• feat: add result of fluid experiment by @julienmancuso in #1379
• feat: Update container with better EFA/RDMA support by @aranadive in #1333
• feat: Support larger Gemma 3 models by @grahamking in #1359
• refactor: Rename CompletionRequest to NvCreateCompletionRequest by @paulhendricks in #1383
• feat: decouple bento dependency by @biswapanda in #1266
• feat: data synthesizer based on prefix statistics by @PeaBrane in #1087
• feat: introduce abstract classes to dynamo services by @mohammedabdulwahhab in #924
• feat: KVBM dynamo runtime + event manger by @oandreeva-nv in #1195
• feat: Utilities for distributed leader-worker barriers by @jthomson04 in #1429
• feat: Restructure the KVBM WriteTo trait by @jthomson04 in #1363
• feat: KVBM prometheus monitoring by @jthomson04 in #1211
• feat: Improved offload queueing and block eviction ordering by @jthomson04 in #1425
• feat: generate random texts from hashes using lorem ipsum by @PeaBrane in #1458
• refactor: use comment filed in annotated to pass metric-related information by @tedzhouhk in #1385
• feat: generalize VLM embedding extraction by @hhzhang16 in #1388
• refactor: move kv store to runtime by @ryanolson in #1459
• feat: add endpoint to clear all kv blocks in vllm v1 by @jain-ria in #1384
• feat: Video support with Dynamo by @indrajit96 in #1443
• feat: add build --push command by @hhzhang16 in #1485
• feat: FT downed worker instance tracking and skipping by @kthui in #1424
• feat: add dynamo pipeline example using inf-gw by @biswapanda in #1512
• refactor: Log subprocess stderr as WARN (#1563) by @rmccorm4 in #1574

🐛 Bug Fixes

• fix: cherry-pick of attributions from 0.2.1 release branch by @saturley-hall in #1267
• fix: resolve local dev container build issues by @t-ob in #1269
• fix: Renamed event publisher classes and configuration by @alec-flowers in #1273
• fix: Only check model name on etcd-registered endpoints by @jthomson04 in #1263
• fix: Fix mypy errors on trtllm examples by @tanmayv25 in #1277
• fix: remove sglang hash for pyproject by @ishandhanani in #1281
• fix: copy workspace as part of ci-min stage by @nv-anants in #1291
• fix: resources naming by @biswapanda in #1302
• fix: wait until probing on vllm examples to prevent timeouts by @mohammedabdulwahhab in #1293
• fix: Fix vllm v0 None*int error when not using kv aware router by @tedzhouhk in #1304
• fix: Update breaking change to enable_overlap_scheduler field from TRTLLM commit b4e5df0e by @rmccorm4 in #1310
• fix: make imagePullSecrets optional when installing dynamo cloud by @julienmancuso in #1324
• fix: Properly set VLLM_NIXL_SIDE_CHANNEL_HOST in multi-node by @ptarasiewiczNV in #1327
• fix: Allow building only llamacpp or only mistralrs engine. by @grahamking in #1328
• fix: allow custom annotations in api-store service by @julienmancuso in #1329
• fix: Flatten pytorch_backend_config section to address breaking change to trtllm config by @rmccorm4 in #1326
• fix: update profile script by @tedzhouhk in #1336
• fix: Use min of max tokens or context length by @abrarshivani in #1322
• fix: add ingress to llm example by @hhzhang16 in #1349
• fix(dynamo-run): For internal comms use a random endpoint instead of hard coded by @grahamking in #1335
• fix: dockerhub registry issues in dynamo operator by @mohammedabdulwahhab in #1350
• fix: add speculative decoding config to dynamo serve + trtllm by @richardhuo-nv in #1356
• fix: prefillqueue stream name in load-planner by @tedzhouhk in #1377
• fix: take into account number of workers from config by @julienmancuso in #1365
• fix: Fix link path for dynamo_run doc by @krishung5 in #1382
• fix: fix dynamo cloud helm chart by @julienmancuso in #1376
• fix: mismatch GAP and PA version by @tedzhouhk in #1386
• fix: remove unused arg in planner by @tedzhouhk in #1390
• fix: Use Ru...

Dynamo Release v0.3.0

Dynamo is an open source project under the Apache 2.0 license. The primary distribution is done through pip wheels with minimal binary size. The ai-dynamo GitHub organization hosts two repositories: Dynamo and NIXL. Dynamo is designed as the next-generation inference server, building upon the foundation of NVIDIA® Triton Inference Server™. While Triton focuses on single-node inference deployments, we're integrating its robust capabilities into Dynamo over the next several months. We'll maintain support for Triton while providing a clear migration path for existing users once Dynamo achieves feature parity.

As a vendor-neutral serving framework, Dynamo supports multiple large language model (LLM) inference engines to varying degrees:

• NVIDIA TensorRT-LLM
• vLLM
• SGLang

Dynamo v0.3.0 features:

• Dynamo run with KV routing and multiple model support! guide
• Vllm v1 engine support! example
• Sglang with DP attention! example
• SLA based planner! guide
• Optimized embedding transfer for multi-modal! example
• Dynamo deploy update command! guide
• Model caching using Fluid! guide
• Fluxcd guide to managing custom resources guide

Future plans
Dynamo Roadmap

Known Issues

• KVBM is supported only with python 3.12

What's Changed

🚀 Features & Improvements

• feat: kv block manager by @ryanolson in #965
• feat(sglang): disaggregated support by @ishandhanani in #976
• feat(dynamo-run): Print HTTP routes on startup by @grahamking in #1010
• feat(dynamo-run): KV-aware routing by @grahamking in #1064
• feat: KV Cache Manager block offloading by @jthomson04 in #1030
• feat: Add ignore_eos/nvext support for legacy completions by @rmccorm4 in #1080
• feat: Use existing Tokio runtime in components by @abrarshivani in #941
• feat: add vLLM V1 PD disagg example by @ptarasiewiczNV in #1013
• feat: Add OpenAI Embeddings interface in rust lib by @t-ob in #1110
• feat: add update deployment to dynamo deploy API and CLI by @hhzhang16 in #1048
• feat: KV Block Manager Python bindings by @kthui in #1022
• feat: Add LWS to Dynamo Operator by @nvrohanv in #998
• feat: Add support for SSD offloading in block manager by @jthomson04 in #1115
• feat: Support multiple models on single ingress node by @grahamking in #1127
• feat: adding outer dimension to isolate k/v blocks by @ryanolson in #1126
• feat: SLA Profiling and Recommending Parallelization Mapping by @tedzhouhk in #1114
• feat: vllm mock workers, Rusty skeleton by @PeaBrane in #1033
• feat: rename dynamo decorator by @biswapanda in #1133
• feat(dynamo-run): Allow setting context-length by @grahamking in #1157
• feat: Various KVBM improvements by @jthomson04 in #1134
• feat: Add TTFT and ITL Interpolation to Profiling Script by @tedzhouhk in #1159
• feat(dynamo-run): Allow setting KV cache block size by @grahamking in #1175
• feat: Add standalone script for TRTLLM integration into dynamo-run by @tanmayv25 in #1162
• feat: adding arena allocator for storage objects by @ryanolson in #1178
• feat: support k8s target in dynamo deploy command by @hhzhang16 in #1104
• feat: add dynamo operator overview doc by @julienmancuso in #688
• feat: add dynamo-run example for vllm v0 by @tedzhouhk in #1186
• feat: kvbm offload fixes and tests by @jthomson04 in #1191
• feat: Add metrics and event publishers by @tanmayv25 in #1192
• feat: NIXL Based RDMA Support w/ Multimodal Example by @whoisj in #1060
• feat: Add Hello World Multinode example by @kylehh in #624
• feat(sglang): add dockerfile/pyproject toml entry + steps to run dsr1 disagg by @ishandhanani in #1193
• feat(http): add health check endpoint by @ishandhanani in #1037
• feat: document model caching using Fluid by @julienmancuso in #1218
• feat: portable dynamo build by @biswapanda in #1215
• feat: fluxcd guide to managing custom resources by @mohammedabdulwahhab in #1220
• feat: Enable dynamo-run out=trtllm by @tanmayv25 in #1223
• feat(dynamo-llm): Remove bring-your-own-engine by @grahamking in #1216
• feat: remove bento cloud deploy target, set deployment target to kubernetes by default by @hhzhang16 in #1247
• feat: Support OAI frontend format and add async image handing by @krishung5 in #1214
• feat: add KV Event Publishing to vLLM v1 by @alec-flowers in #1181

🐛 Bug Fixes

• fix(bindings): serve_endpoint no longer takes a lease by @grahamking in #1014
• fix(deps): sglang install must be done manually by @ishandhanani in #1019
• fix: dynamo_serve and scv config inject/get by @tedzhouhk in #1017
• fix: pin click dependency to old releases by @nv-anants in #1042
• fix: use correct lease id for kv router by @tedzhouhk in #1035
• fix: update nixl setup for arm builds by @nv-anants in #1061
• fix: downgrade CUDA image use to work around PyNccl timeout in vLLM Ray use case by @GuanLuo in #1065
• fix: read 'workers' to set deployments 'replicas' by @julienmancuso in #1040
• fix: add maxage to nats stream by @wxsms in #1053
• fix: fix broken links in deployment docs by @biswapanda in #1084
• fix: Fix default RouterMode value by @grahamking in #1092
• fix: planner fixes by @mohammedabdulwahhab in #1055
• fix: use resource and workers hints from decorators and service args by @biswapanda in #1044
• fix: add planner path in devcontainer by @biswapanda in #1113
• fix: remove lib.real from LD_LIBRARY_PATH by @alec-flowers in #1117
• fix(sglang): allow for disaggregation_bootstrap_port for multinode deployment by @ishandhanani in #1119
• fix: Disable block manager by default in Python bindings by @kthui in #1128
• fix: Incrementally decode token to reduce the overhead from Processor by @tanmayv25 in #1129
• fix: set gpus as strings in config files by @julienmancuso in #1123
• fix: Fix the protocol in the example by @tanmayv25 in #1146
• fix: register model after engine load by @nnshah1 in #1145
• fix: make component type a simple string by @mohammedabdulwahhab in #1144
• fix(llmctl): Use ModelWatcher instead of direct etcd operations by @grahamking in #1150
• fix(dynamo-run): Don't exit interactive chat on error by @grahamking in #1155
• fix(llmctl): Add back the model_type in remove by @grahamking in #1158
• fix: Enable Dynamo HTTP servers to run on IPv6-only hosts by @jmswen in #1166
• fix: typo in planner doc and log by @tedzhouhk in #1165
• fix: Fix race condition in kv_router unit test by @grahamk...

Dynamo Release v0.2.1

Dynamo is an open source project with Apache 2 license. The primary distribution is done via pip wheels with minimal binary size. The ai-dynamo github org hosts 2 repos: dynamo and NIXL. Dynamo is designed as the ideal next generation inference server, building upon the foundations of the Triton Inference Server. While Triton focuses on single-node inference deployments, we are committed to integrating its robust single-node capabilities into Dynamo within the next several months. We will maintain ongoing support for Triton while ensuring a seamless migration path for existing users to Dynamo once feature parity is achieved. As a vendor-agnostic serving framework, Dynamo supports multiple LLM inference engines including TRT-LLM, vLLM, and SGLang, with varying degrees of maturity and support.

Dynamo v0.2.1 features:

• KV Block Manager! intro
• Improved vLLM Performance by avoiding re-initializing sampling params
• SGLang support! README.md
• Multi-Modal E/P/D Disaggregation! README.md
• Leader Worker Set K8s!
• Qwen3, Gemma3 and Llama4 in Dynamo Run!

Future plans

Dynamo Roadmap

Known Issues

• Benchmark guides are still being validated on public cloud instances (GCP / AWS)

What's Changed

🚀 Features & Improvements

• feat: Qwen3, Gemma3 and Llama4 support by @grahamking in #1002
• feat: Remove vllm and sglang from cargo build command by @hhzhang16 in #1003
• feat: deploy planner in operator by @julienmancuso in #921
• refactor: use primary lease + self-contained graceful shutdown trigged by SIGINT/SIGTERM by @tedzhouhk in #1001
• feat: Add AWS EFA support by @aranadive in #999
• feat(sglang): aggregated support by @ishandhanani in #937
• feat: decoupling dynamo serve by @biswapanda in #905
• feat: allow adding auth to etcd by @wxsms in #980

🐛 Bug Fixes

• fix: Extract tokenizer from GGUF for Qwen3 and Gemma3 arch by @grahamking in #1011

Other Changes

• docs: add docs for dynamo build by @mohammedabdulwahhab in #714
• docs: fix typo in disagg perf tuning guide by @tedzhouhk in #859
• feat: Adding completions endpoint support to dynamo run in=http by @oandreeva-nv in #777
• docs: update editable install to include planner by @nv-anants in #860
• chore: add docs around how runtime reconfiguration works by @ishandhanani in #861
• feat: replace async queue with async iter and double decorator by @biswapanda in #858
• docs: fix typo in planner documentation by @AndyDai-nv in #864
• feat: Add unified x86 / aarch64 (ARM) build for VLLM image by @rmccorm4 in #839
• refactor: move logging config to runtime by @ishandhanani in #863
• feat: support multiple endpoints by @biswapanda in #857
• build: Add Olga as a Rust reviewer by @grahamking in #872
• fix: change the processor number to 5 to reduce the tokenization bottleneck by @richardhuo-nv in #865
• refactor: change trtllm example kv routing use python bindings | deal with trtllm partial blocks | trtllm event change by @ziqif-nv in #866
• fix: change environment variable to support local mount by @nnshah1 in #885
• fix: manylinux tag in ai-dynamo-vllm wheel by @nv-anants in #884
• chore: Split PushRouter from Client by @grahamking in #817
• chore: add fastapi depenedncy in pyproject.toml by @biswapanda in #888
• docs: update pythonpath for starting planner by @tedzhouhk in #890
• fix(http): Make ModelDeploymentCard optional by @grahamking in #891
• feat: Add request template support for default inference parameters by @abrarshivani in #841
• fix: endless map in nixl.py by @wxsms in #852
• feat: remove dynamoComponentRequest CRD by @julienmancuso in #856
• docs: Fixes to dynamo deploy docs by @mohammedabdulwahhab in #902
• feat: label component CR for planner by @julienmancuso in #901
• feat: allow users to add env vars to dynamo deployment by @hhzhang16 in #862
• chore: unified logging, added informative warnings for KV router example by @PeaBrane in #912
• docs: add an example on how to use --service-name flag to spin up a standalone service by @ishandhanani in #915
• fix: trtllm example by @biswapanda in #909
• fix: add dedicated llmapi config for trtllm disagg kv routing example by @ziqif-nv in #916
• chore: reduce code repetition in processor by @PeaBrane in #919
• feat: Support hf:// URLs in dynamo run by @abrarshivani in #917
• feat: Add check for version info in container build script by @abrarshivani in #774
• docs: update examples in document by @biswapanda in #897
• chore(dynamo-llm): Move the pre-processor to ingress side by @grahamking in #903
• fix: default docker username and password are empty by @hhzhang16 in #926
• feat: Add multimodal example with aggregated serving by @krishung5 in #709
• docs: Add multi-node TRTLLM steps to README by @rmccorm4 in #930
• feat: Update to support completion endpoint in TRTLLM by @tanmayv25 in #837
• fix: use primary lease for NixlMetadataStore by @tedzhouhk in #928
• chore: merge in support matrix and nixl commit hash by @saturley-hall in #944
• feat: allow to set http port by @julienmancuso in #931
• feat: automatically reserve port for assigning port number to endpoint and pubsub by @richardhuo-nv in #946
• feat: multi-thread (via asyncio.task) in processor by @tedzhouhk in #904
• fix: remove requirement for istio in doc by @julienmancuso in #950
• feat: dynamo-run <-> python interop by @grahamking in #934
• refactor: refactor dynamo deploy subfolder by @hhzhang16 in #927
• ci: lock cuda at 12.8 by @hhzhang16 in #957
• chore: Two-line copyright check by @grahamking in #958
• chore: Add John as Codeowner by @jthomson04 in #962
• feat(dynamo-run): vllm and sglang subprocess engines by @grahamking in #954
• docs: add drt doc by @tedzhouhk in #951
• feat: Migrate NATS Queue to Rust (#669) by @jthomson04 in #961
• fix: create k8s service for main component only by @julienmancuso in #953
• fix: fix missing num_remote_prefill_groups in vLLM patch by @ptarasiewiczNV in #981
• fix: Create default sampling params only once during initialization by @ptarasiewiczNV in #982
• chore: Remove embedded Python vllm and sglang engines by @grahamking in #966
• fix: increase ulimit nofile for container/run.sh by @ajcasagrande in #969
• docs: add fix for Zsh globbing error with pip install .[all] by @Chasing1020 in #945
• build: Cleans the TensorRTLLM + Dynamo container build by @tanmayv25 in #968
• feat: add interface for deployment manager by @biswapanda in #987
• fix: Check nvext for ignore_eos and set min_tokens for benchmark consistency by @rmccorm4 in #988
• fix: Fix vllm/sglang engine model name if using HF repo by @grahamking in #986
• feat: Add multimodal example with disaggregated serving by @krishung5 in #811
• feat: clea...

Dynamo Release v0.2.0

Dynamo is an open source project with Apache 2 license. The primary distribution is done via pip wheels with minimal binary size. The ai-dynamo github org hosts 2 repos: dynamo and NIXL. Dynamo is designed as the ideal next generation inference server, building upon the foundations of the Triton Inference Server. While Triton focuses on single-node inference deployments, we are committed to integrating its robust single-node capabilities into Dynamo within the next several months. We will maintain ongoing support for Triton while ensuring a seamless migration path for existing users to Dynamo once feature parity is achieved. As a vendor-agnostic serving framework, Dynamo supports multiple LLM inference engines including TRT-LLM, vLLM, and SGLang, with varying degrees of maturity and support.

Dynamo v0.2.0 features:

• GB200 support with ARM builds (Note: currently requires a container build)
• Planner - new experimental support for spinning workers up and down based on load
• Improved K8s deployment workflow
  • Installation wizard to enable easy configuration of Dynamo on your Kubernetes cluster
  • CLI to manage your operator-based deployments
  • Consolidate Custom Resources for Dynamo Deployments
  • Documentation improvements (including Minikube guide to installing Dynamo Platform)

Future plans

Dynamo Roadmap

Known Issues

• Benchmark guides are still being validated on public cloud instances (GCP / AWS)
• Benchmarks on internal clusters show a 15% degradation from results displayed in summary graphs for multi-node 70B and are being investigated.
• TensorRT-LLM examples are not working currently in this release - but are being fixed in main.

What's Changed

• fix: fix max_local_prefill_length not being printed out in disagg router log by @tedzhouhk in #628
• docs: Add instructions to install git lfs by @tanmayv25 in #627
• fix: add DYNAMO_HOME env var to vLLM docker image by @nv-anants in #629
• fix: Account for Metrics.decode() changes by @rmccorm4 in #619
• fix: Update test_report by @pvijayakrish in #641
• fix: serviceArgs in config was not getting set for workers by @mohammedabdulwahhab in #640
• fix: adding conversion to string for notif id comparison by @nnshah1 in #638
• docs: Add documentation for UCX KV cache transfer in TRTLLM by @tanmayv25 in #639
• build: Define UCX env var to use NVLink when available by @tanmayv25 in #631
• feat: ETCD prefix watcher + python binding + runtime reconfiguration for router and disagg router by @tedzhouhk in #581
• fix: dynamo build should work with link syntax by @mohammedabdulwahhab in #646
• fix: change trtllm kv_router default block_size to 32 by @ziqif-nv in #642
• fix: signal handlers to clean up zombie vllm processes by @ishandhanani in #545
• feat: add .devcontainer based off images in container/ by @alec-flowers in #497
• fix: devcontainer mounts and vllm c api by @alec-flowers in #663
• fix: deploy command should support passing config by @mohammedabdulwahhab in #626
• feat(dynamo-run): improve available engines list in --help by @XueSongTap in #664
• feat: add dynamoDeployment CR finalizer by @julienmancuso in #623
• fix: set correct parent_hash for each kv block when publish kv events by @ziqif-nv in #671
• docs: Use the same term for dynamo base image across code snippets and text by @hutm in #670
• docs: move deploy docs to docs/guides by @hhzhang16 in #674
• fix: frontend and http server signal handling by @alec-flowers in #677
• fix: check for resource in pipeline helm chart by @julienmancuso in #687
• fix: ensure VLLM_LOGGING_LEVEL=xyz followsDYN_LOG=xyz by @ishandhanani in #692
• feat: replace dynamo server with dynamo cloud by @hhzhang16 in #696
• feat: base Dynamo docker image improvements and fixes by @hhzhang16 in #658
• fix: fix pipeline helm chart by @julienmancuso in #698
• docs: Benchmarking guide updates by @kthui in #678
• feat: bump vLLM version to v0.8.4 by @ptarasiewiczNV in #690
• chore: Replace TRD->Dynamo in llmctl help output by @rmccorm4 in #710
• fix: allow for an empty dynamo config file by @hhzhang16 in #712
• fix: cli version by @ishandhanani in #716
• docs: Remove outdated python-wheels directory reference by @rmccorm4 in #719
• fix: direct clients vs dependancies by @ishandhanani in #704
• feat: adding dynamo-tokens crate by @ryanolson in #718
• fix: bump GAP to r25.03 by @tedzhouhk in #724
• feat: make ingress configurable in operator by @julienmancuso in #717
• feat: configure logger with detail info by @tlipoca9 in #654
• feat: Add disagg skeleton example by @kylehh in #683
• fix: dynamo deploy helm chart cleanup by @mohammedabdulwahhab in #727
• docs: add dedicated minikube guide by @mohammedabdulwahhab in #735
• feat(dynamo-engine-vllm): vllm 0.8.X support by @grahamking in #728
• feat: gracefully shutdown endpoint by revoking etcd lease + python binding by @tedzhouhk in #730
• fix: Add missing deps for '--framework none' build by @rmccorm4 in #738
• chore: Remove TRT-LLM C++ engine in favor of Python one by @grahamking in #747
• docs: Support matrix post release. by @pvijayakrish in #736
• docs: add aggregated deployment guide for multi-node sized model by @GuanLuo in #713
• feat: make the model name to be the same as the HF repo name for dynamo-run by @AndyDai-nv in #749
• feat: add additional packages to log filters by @abrarshivani in #752
• chore(dynamo-run): Fix echo_core for EOS tokens by @grahamking in #759
• feat: add custom lease to worker components by @ishandhanani in #748
• chore: Add roadmap to main README.md by @harryskim in #763
• feat: MLA disaggregation support to vLLM patch by @ptarasiewiczNV in #745
• fix: Fix cancellation flow in python component graph by @pankajroark in #765
• fix: give the user ownership permissions of /opt/dynamo/venv by @hhzhang16 in #767
• docs: deployment docs improvements by @hhzhang16 in #753
• feat: add option to configure separate docker registry for pipelines docker images by @julienmancuso in #744
• chore: Update bug report to use dynamo env for collecting environment information by @nv-tusharma in #558
• docs: R1 disaggregation guide by @GuanLuo in #720
• feat: allow to CRUD dynamo pipelines by @julienmancuso in #761
• docs: Custom Backend/Worker Guide by @rmccorm4 in #608
• chore: fix arg name in example by @CormickKneey in #770
• build: add rust binaries in manylinux image by @nv-anants in #783
• feat: remove bento/yatai references by @julienmancuso in #782
• docs: add note to use release branch examples by @nv-anants in #793
• feat: Add log verbosity level flag to dynamo-run cli by @abrarshivani in #780
• feat: rename operator CRDs by @julienmancuso in #795
• feat: Add linux aarch64 support to dynamo-run build by @rmccorm4 in #802
• fix: Update TRTLLM version and fix disagg workflow by @tanmayv25 in #804
• chore: Increase sleep tim...

Dynamo Release v0.1.1

Dynamo is an open source project with Apache 2 license. The primary distribution is done via pip wheels with minimal binary size. The ai-dynamo github org hosts 2 repos: dynamo and NIXL. Dynamo is designed as the ideal next generation inference server, building upon the foundations of the Triton Inference Server. While Triton focuses on single-node inference deployments, we are committed to integrating its robust single-node capabilities into Dynamo within the next several months. We will maintain ongoing support for Triton while ensuring a seamless migration path for existing users to Dynamo once feature parity is achieved. As a vendor-agnostic serving framework, Dynamo supports multiple LLM inference engines including TRT-LLM, vLLM, and SGLang, with varying degrees of maturity and support.

Dynamo v0.1.1 features:

• Benchmarking guides for Single and Multi-Node Disaggregation on H100 (vLLM)
• TensorRT-LLM support for KV Aware Routing
• TensorRT-LLM support for Disaggregation
• ManyLinux and Ubuntu 22.04 Support for wheels and crates
• Unified logging for Python and Rust

Future plans

• Instructions for reproducing benchmark guides on GCP and AWS
• KV Cache Manager as a standalone repository under the ai-dynamo organization. This release will provide functionality for storing and evicting KV cache across multiple memory tiers, including GPU, system memory, local SSD, and object storage.
• Searchable user guides and documentation
• Multi-node instances for large models
• Initial Planner version supporting dynamic scaling of P / D workers. We will include an early version of the Dynamo Planner, another core component. This initial release will feature heuristic-based dynamic allocation of GPU workers between prefill and decode tasks, as well as model and fleet configuration adjustments based on user traffic patterns. Our vision is to evolve the Planner into a reinforcement learning platform, which will allow users to define objectives and then tune and optimize performance policies automatically based on system feedback.
• vLLM 1.0 support with NIXL and KV Cache Events

Known Issues

• Benchmark guides are still being validated on public cloud instances (GCP / AWS)
• Benchmarks on internal clusters show a 15% degradation from results displayed in summary graphs for multi-node 70B and are being investigated.

What's Changed

• docs: Benchmarking guide updates (#678) by @kthui in #699
• docs: Update support matrix by @pvijayakrish in #691
• fix: change trtllm kv_router default block_size to 32 (#642) by @tanmayv25 in #694
• fix: set correct parent_hash for each kv block when publish kv events by @tanmayv25 in #693
• fix: Remove kv connector from agg config by @ptarasiewiczNV in #655
• fix: Account for Metrics.decode() changes (#619) by @rmccorm4 in #619
• fix: update to match latest nixl notifications as bytes @nnshah1 in #645
• docs: Update support matrix by @pvijayakrish in #633
• docs: Add instructions to install git lfs (#627) by @tanmayv25 in #627
• fix: add DYNAMO_HOME env var to vLLM docker image (#629) by @nv-anants in #629
• feat: TRT-LLM disaggregated serving using UCX (#562) by @tanmayv25 in #562
• docs: Update support matrix by @pvijayakrish in #604
• docs: Guide for multi-node benchmarking (#561) by @kthui in #561
• fix: remove api-store from container by @mohammedabdulwahhab in #617
• docs: Guides for single node benchmarking (#509) by @kthui in #509
• fix: set worker env before worker process spawn by @ishandhanani in #614
• docs: Move trtllm dynamo run doc from example to dynamo run guide (#578) by @tanmayv25 in #578
• chore: update ai-dynamo-vllm wheel version (#598) by @nv-anants in #598
• fix: bump bento to 1.4.8 (#579) by @mohammedabdulwahhab in #579
• fix: update yum install in wheel-builder image (#605) by @nv-anants in #605
• docs: update dynamo serve trtllm agg example yaml files (#600) by @ziqif-nv in #600
• chore: use latest nixl for docker builds by @nv-anants in #596
• chore: update versions to 0.1.1 by @nv-anants in #552
• docs: Updated dynamo run instructions by @cdgamarose-nv in #555
• feat: Add manylinux support for Dynamo by @pvijayakrish in #536
• docs: Clarify the --max-local-prefill-length help description by @kthui in #554
• feat: Add dynamo env CLI option to provide information about user environment by @nv-tusharma in #533
• docs: add disagg tuning guide by @tedzhouhk in #413
• fix: let dynamo run pass --help to dynamo-run by @ziqif-nv in #547
• chore: Update TRTLLM version. Fix router. by @tanmayv25 in #527
• fix: unify and enable dynamo logging by @ishandhanani in #520
• feat(dynamo-run): Basic routing choice by @grahamking in #524
• fix: clean unused bento pieces from serve.py and serving.py by @ishandhanani in #532
• docs: update close-deployment in dynamo_serve.md by @tlipoca9 in #535
• feat: update operator README by @julienmancuso in #544
• fix: mypy error by @ishandhanani in #543
• feat: cleanup operator code by @julienmancuso in #529
• chore: Fixed file headers. Added attributions. by @dmitry-tokarev-nv in #530
• fix: Remove api-server code by @mohammedabdulwahhab in #526
• docs: hello world and vllm process docs by @ishandhanani in #525
• feat: KV recorder for dumping router events into a jsonl by @PeaBrane in #505
• chore: cleaner required workers check (don't spam print) by @PeaBrane in #521
• docs: dynamo-run clarify engine list by @grahamking in #522
• chore: Upgrade Rust to 1.86 by @grahamking in #518
• chore: Add devops in more CODEOWNERS by @grahamking in #512
• feat: Python decorator dynamo_worker takes optional static parameter without etcd by @grahamking in #494
• fix: broken link to dynamo run by @lkm2835 in #517
• docs: add 405b disaggregated serving documentation by @ishandhanani in #496
• refactor: migrate engines to standalone crates by @ryanolson in #453
• feat: Add TensorRT-LLM example for dynamo serve/run by @tanmayv25 in #456
• docs: Remove invalid link by @grahamking in #506
• docs: add instruction to copy dynamo-run in container setup by @hanweisen in #508
• chore: Add libclang-dev to CI for llamacpp by @grahamking in #507
• chore: rename duration to timeout by @tlipoca9 in #503
• fix: adding missing file by @ryanolson in #501
• feat: allow replicas to be set in DynamoDeployment CR by @julienmancuso in #486
• chore: Disable blank issue creation for default issues template by @nv-tusharma in #492
• chore: Remove <> from title + add labels for default issues template. by @nv-tusharma in #491
• feat: Sets the code of conduct for the repository by @saturley-hall in #454
• fix: Consolidate dynamo start and dynamo serve commands by @mohammedabdulwahhab in #405
• feat: improve serve commands and expose DYNAMO_HOME env var by @jon-chuang in #436
• feat: kv aware router executable by @ryanolson in #399
• feat: deploy and use buildkit to build dynamo images by @julienmancuso in #450
• feat(serve): Enhance multi-node deployment and worker configuration by @ishandhanani in #457
• chore: Add default issue template for bug & feature requests by @nv-tusharma in #471
• feat: unified logging by @ryanolson in #472
• feat: add devcontainer to dynamo for Ubuntu 24.04 use by @h...