v0.11.1

What's Changed

🚀 Features

• [ascend] support dptp by @tangzhiyi11 in #4218
• Support Deepseek v32 by @grimoire in #4026

💥 Improvements

• Improve metrics by @CUHKSZzxy in #4178
• reserve blocks for dummy inputs by @grimoire in #4157
• Add vision id for Qwen3-VL by @CUHKSZzxy in #4183
• [Enhance]: Return routed experts when request canceled by @RunningLeon in #4197
• Add mm processor args for Qwen3-VL by @CUHKSZzxy in #4196
• support chat_template_kwargs in v1/chat/completions by @lvhan028 in #4201
• Refactor scheduler and engine.py by @grimoire in #4163
• update dp timeout by @grimoire in #4204
• Improve Qwen3-VL by @CUHKSZzxy in #4207

🐞 Bug fixes

• [Fix]: Split routed experts with query lens by @RunningLeon in #4180
• [Maca] fix ray and memory sync by @wanfengcxz in #4164
• Build block trie in prefill and add hit rate by @RunningLeon in #4184
• fix fope by @CUHKSZzxy in #4191
• fix hf modules read/write conflicts by multi processors by @lvhan028 in #4188
• Some Minor fix by @windreamer in #4185
• fix insecure deserialization when calling torch.load() by @lvhan028 in #4202
• Fix processor args by @CUHKSZzxy in #4200
• remove get_model_config to avoid pickle hf_config error in rpc calling by @lvhan028 in #4217
• Fix quant scale-fmt by @grimoire in #4212
• Fix requests of mix return_logprobs by @RunningLeon in #4222
• fix fillkv quant8 by @grimoire in #4229
• fix scale-fmt by @grimoire in #4230

📚 Documentations

• [Docs]: Add guide for VLMEvalKit by @CUHKSZzxy in #4156

🌐 Other

• Add FA3 by @CUHKSZzxy in #4166
• Add distributed test cases by @littlegy in #4161
• Add generate test by @littlegy in #4181
• [ci] add mllm eval by @zhulinJulia24 in #4194
• [ascend] refactor code by @yao-fengchen in #4176
• install serve.txt when building the docker image by @lvhan028 in #4219
• bump version to v0.11.1 by @lvhan028 in #4221

Full Changelog: v0.11.0...v0.11.1

v0.11.0

What's Changed

🚀 Features

• add endpoint /abort_request by @lvhan028 in #4092
• Qwen3 next by @grimoire in #4039
• Support Qwen3-VL by @CUHKSZzxy in #4093
• Support sync weights with flattened bucket tensor by @RunningLeon in #4109
• Support group router for moe models by @RunningLeon in #4120
• [Feature]: return routed experts to reuse by @RunningLeon in #4090
• support context parallel by @irexyc in #3951
• fope by @grimoire in #4043
• [Feature]: Support speculative decoding by @RunningLeon in #3945
• Moe bf16 ep by @grimoire in #4144

💥 Improvements

• Enlarge gc threshold by @grimoire in #4076
• remove num_tokens from EngineOutput by @lvhan028 in #4088
• revert masking vocab_size by @lvhan028 in #4089
• feat: add json_object support in response_format by @windreamer in #4080
• support image_data input to /generate endpoint by @irexyc in #4086
• [Fix] all RayEngineWorker actors created at node 0 in RL training by @CyCle1024 in #4107
• Optimize sleep level=1 for turbomind backend by @irexyc in #4074
• [Feat] enable ascend update_params by @CyCle1024 in #4111
• Enhance request checker by @lvhan028 in #4104
• Refactor dp tp by @grimoire in #4004
• fix kernel numerical error by @grimoire in #4133
• free ray put by @grimoire in #4137
• Reduce experts cache when resize by @RunningLeon in #4138
• support interleave text and image in messages by @lvhan028 in #4141
• optimize rms norm by @grimoire in #4153
• fix evict policy by @Tsundoku958 in #4127

🐞 Bug fixes

• fix type hint by @grimoire in #4078
• Fix inputs split by @RunningLeon in #4083
• add missing update_model_meta by @jinminxi104 in #4099
• Fix update_params for pytorch backend when loading vl model by @irexyc in #4101
• workaround for issue "TypeError argument 'tokens': 'NoneType' object cannot be converted to 'PyString" by @lvhan028 in #4103
• fix bug: schedule ratio support prefix-caching by @Tsundoku958 in #4100
• remove prefill free ratio threshold by @grimoire in #4110
• fix key error: api_server node might be removed by @lvhan028 in #4112
• Incorrectly judging the request as a bad request by @lvhan028 in #4121
• fix dist config keys by @grimoire in #4125
• proxy server miss media_type in streaming mode by @lvhan028 in #4130
• Fix logprobs to_tensor by @RunningLeon in #4132
• Fix cli help by @RunningLeon in #4139
• fix and optimize fill_kv_cache_quant by @grimoire in #4140
• fix: fix package deprecation introduced by CUDA 13 by @windreamer in #4117
• yield empty list for token_ids when it runs out of tokens by @lvhan028 in #4148
• Fix interns1 routed experts outputs by @RunningLeon in #4149
• fix qwen3-30-a3b lcb-code score by @yao-fengchen in #4142
• Fix ep deployment issues by @CUHKSZzxy in #4084
• Fix dllm to not use fa3 decoding by @RunningLeon in #4159
• fix: handle non-tuple decoder outputs during Qwen-2.5 quantization by @chengyuma in #4158
• fix cu11 docker build by @CUHKSZzxy in #4165
• Fix model config by @CUHKSZzxy in #4170
• fix lora by @grimoire in #4172
• fix cmake logic detect sm70, sm75 by @tuilakhanh in #4175

📚 Documentations

• Update model evalution guide by @lvhan028 in #4094
• [Docs]: Add guide for update weights by @RunningLeon in #4151

🌐 Other

• add dockerfile to build dev image by @lvhan028 in #4091
• add ascend_a3 Dockerfile by @yao-fengchen in #4097
• [ci] refactor longtext benchmark by @zhulinJulia24 in #4087
• enable metrics by default by @lvhan028 in #4108
• Replace pynvml with nvidia-ml-py in requirements by @myhloli in #4118
• [ci] add free disk before build test whl package and add session_len args in benchmark script by @zhulinJulia24 in #4136
• Add prefixcache functionality and performance testing by @littlegy in #4119
• [ci] modify pipeline.close and add more case into pr_test by @zhulinJulia24 in #4150
• bump version to v0.11.0 by @lvhan028 in #4155

New Contributors

• @myhloli made their first contribution in #4118
• @tuilakhanh made their first contribution in #4175

Full Changelog: v0.10.2...v0.11.0

v0.10.2

What's Changed

🚀 Features

• add /generate api by @irexyc in #4019
• Guided decoding with xgrammar for TurboMind by @windreamer in #3965
• Reimplement guided decoding with xgrammar for PyTorch Engine by @windreamer in #4028

💥 Improvements

• [ascend] support aclgraph by @yao-fengchen in #4063
• Leverage incremental output between the inference and async engines to improve performance by @lvhan028 in #4054
• Optimize multinomial sampling by @grimoire in #4056

🐞 Bug fixes

• zmqrpc localhost only by @grimoire in #4017
• fix bug: dp+tp warmup by @Tsundoku958 in #3991
• fix dllm long-context by @grimoire in #4012
• Fix GPT-OSS streaming tool call parsing by @QwertyJack in #4023
• move releasing resource from async_engine to inference engine by @lvhan028 in #4041
• fix: fix tokenizer parsing bug for guided decoding by @windreamer in #4044
• Fix message content field handling for tool calls and multimodal input by @QwertyJack in #4029
• fix builder for kimi-k2 by @CUHKSZzxy in #4069
• Skip unnecessary sampling and fix the random offset by @grimoire in #4068
• fix duplicated stop_token_string when ignore_special_tokens is False by @irexyc in #4077

🌐 Other

• Drop CUDA 11.8 build support, upgrade CI/CD to CUDA 12.6/12.8 by @windreamer in #4013
• remove profile_generation.py and its testcases by @lvhan028 in #4027
• [ci] refactor eval into api eval and add h800 eval workflow by @zhulinJulia24 in #4008
• Add Docker image for NVIDIA Jetson by @windreamer in #3834
• [ci] refactor api evaluate test into llm judger evaluation by @littlegy in #4046
• Check color logger by @grimoire in #4060
• Update API testing with HLE and LCB datasets by @littlegy in #4061
• update ascend requirements by @yao-fengchen in #4066
• bump version to v0.10.2 by @lvhan028 in #4062

Full Changelog: v0.10.1...v0.10.2

v0.10.1

What's Changed

🚀 Features

• Add ROCm support: installation guide and FlashAttention compatibility for AMD GPUs by @Vivicai1005 in #3925
• support gpt-oss basic output by @irexyc in #3956
• Add FP8*(B)F16 GEMM by @lzhangzz in #3960
• Support GLM-4.5 by @CUHKSZzxy in #3863
• [Refactor]: Remove tokenizer when building engine by @RunningLeon in #3978
• Support InternVL3.5-Flash by @CUHKSZzxy in #3952
• support gpt-oss function/reasoning in /v1/chat/completions by @irexyc in #3962
• support returning stop_str in output by @lvhan028 in #3984
• Support SDAR by @grimoire in #3922

💥 Improvements

• specify installation on GeForce RTX 50 series by @lvhan028 in #3947
• cherry pick PR-3708 to return token_id by @lvhan028 in #3976
• Optimize AsyncEngine generation method by @shell-nlp in #3982
• Use blocking sync when TP engine is idling by @lzhangzz in #3974
• add openai_harmony to requirements by @irexyc in #4006

🐞 Bug fixes

• fix bugs with triton3.4.0 by @grimoire in #3946
• fix longrope by @grimoire in #3968
• Fix tm rl usage in xtuner by @irexyc in #3912
• Disable prefix caching when serving a VLM model by @lvhan028 in #3990
• remove NCCL_LAUNCH_MODE by @irexyc in #3994
• return the last token's logprobs, logits and last_hidden_states if include_stop_str_in_output is requested by @lvhan028 in #4000
• [Fix] device args in chat cli when using pytorch engine by @CyCle1024 in #3999
• fix internvl by @CUHKSZzxy in #3997
• fix not-returned iterator in SequenceManager::Erase by @irexyc in #4001
• fix cudagraph without warmup by @grimoire in #4005
• fix internvl flash long context acc by @CUHKSZzxy in #4003

🌐 Other

• [ci] update daily testcase by @zhulinJulia24 in #3944
• [maca] change kv layout from pagedattn to flashattn by @yuchiwang in #3958
• remove cudnn by @irexyc in #3969
• build(pypi): add cuda 12.8 support for wheels by @windreamer in #3948
• [CI] add ascend test by @littlegy in #3959
• update serve requirement by @RunningLeon in #3986
• [ci] add h800 function test workflow by @zhulinJulia24 in #3985
• bump version to v0.10.1 by @lvhan028 in #3989

New Contributors

• @Vivicai1005 made their first contribution in #3925
• @shell-nlp made their first contribution in #3982
• @littlegy made their first contribution in #3959

Full Changelog: v0.10.0...v0.10.1

v0.10.0

What's Changed

🚀 Features

• support offloading weights & kv_cache for turbomind by @irexyc in #3798
• Add PPU backend support by @guozixu2001 in #3807
• Add turbomind metrics by @lvhan028 in #3811
• PytorchEngine support gpt-oss bf16 by @grimoire in #3820
• support sleep/wakeup for pt engine by @irexyc in #3687
• [ascend] run intern-s1 on A3 by @yao-fengchen in #3831
• Initial gpt-oss support for turbomind by @lzhangzz in #3839
• Support GLM-4-0414 and GLM-4.1V by @CUHKSZzxy in #3846
• support internvl3.5 by @lvhan028 in #3886
• Update turbomind communication library by @lzhangzz in #3736
• MXFP4 support for turbomind GEMM library by @lzhangzz in #3927
• Dispatch MXFP4 weight conversion for sm70 & sm75 by @lzhangzz in #3937

💥 Improvements

• fix: turbomind backend config in cli serve by @PeymanRM in #3784
• remove deprecated codes by @lvhan028 in #3759
• Refactor FP8 MoE GEMM by @lzhangzz in #3795
• Fix build rope params by @grimoire in #3760
• Optimize rmsnorm with head_dim=128 by @grimoire in #3814
• Simplify GEMM interface by @lzhangzz in #3818
• Optimize create_model_inputs and schedule_decoding by @grimoire in #3766
• add remote logs;optimize forward lock by @grimoire in #3737
• support deepgemm new api by @grimoire in #3827
• remove serving with gradio by @lvhan028 in #3829
• Deprecate interactive mode from api_server by @lvhan028 in #3830
• build(docker): Try to optimize docker by @windreamer in #3779
• Make a common chat.py to replace each engine's by @lvhan028 in #3836
• Ray mp engine backend by @grimoire in #3790
• [Feat] support using external ray pg with bundles by @CyCle1024 in #3850
• Remove unused code in PT Engine by @grimoire in #3858
• support logprobs by @grimoire in #3852
• optimize prefill preprocess by @grimoire in #3869
• fix flash-attn bc by @grimoire in #3873
• Graph warmup by @grimoire in #3851
• Improve turbomind's prefix cache by @lvhan028 in #3835
• Support OpenAI compatible parameter max_completion_tokens by @Huarong in #3876
• [ascend] add env to set rt visable by ray and disable warmup by @tangzhiyi11 in #3894
• support cache_max_entry_count >= 1 for Turbomind backend by @lh9171338 in #3913
• adjust default values by @lvhan028 in #3921
• [refactor][chat_template][1/N] adopt tokenizer's apply_chat_template by @lvhan028 in #3845
• use FA 2.8.3 which is compatible with torch 2.8.0 by @lvhan028 in #3936
• refactor ascend Dockerfile by @yao-fengchen in #3926

🐞 Bug fixes

• fix gemma3 by @grimoire in #3772
• fix head_dim=None by @grimoire in #3793
• fix user-specified max_session_len by @grimoire in #3785
• remove 'lmdeploy convert' from CLI by @lvhan028 in #3813
• Fix EP with large batch size by @grimoire in #3808
• fix internvl disable_vision_encoder by @grimoire in #3800
• Align response behavior across both engines by @lvhan028 in #3821
• fix: set text_config.tie_word_embedding = False in qwen2vl by @zenosai in #3824
• Fix v1 comp protocol by @CUHKSZzxy in #3828
• [dlinfer] fix get_backend err by @yao-fengchen in #3847
• Update internvl.py to fix #3528 by @zodiacg in #3837
• fix partial rotary factor by @CUHKSZzxy in #3861
• fix: duplicated token usage in /chat/completions stream mode by @Huarong in #3859
• fix chatting with VLM model via CLI by @lvhan028 in #3862
• fix inference on windows platform by @irexyc in #3865
• fix prebuild on cuda12.8 by @lvhan028 in #3857
• Fix uninitialized members in cuBLAS wrapper by @lzhangzz in #3874
• fix flashmla build for cuda12.4 by @CUHKSZzxy in #3872
• [Fix] ray mp engine on ascend platform by @CyCle1024 in #3877
• fix bug: leaves empty by @Tsundoku958 in #3868
• Fix side effect brought by gpt-oss support by @lvhan028 in #3880
• fix pytorch metrics in mp engine by @CUHKSZzxy in #3882
• Fix stream assert error when wakeup 30+ times by @CyCle1024 in #3883
• fix batched prefill by @grimoire in #3887
• fix side effect brought by #3821 by @lvhan028 in #3888
• check_env in multiprocess by @grimoire in #3879
• fix cli serve --help by @RunningLeon in #3895
• 1. [PD Disaggregation] Some Bug Fix (adapte p2p_initialize, metrics, uniexecutor with pd disagg) by @JimyMa in #3893
• Resolve a crash in the sleep endpoint by casting the level parameter from string to int by @irexyc in #3897
• Fix nccl for docker cu11 by @RunningLeon in #3896
• disable check_env in multiprocess on dlinfer devices by @tangzhiyi11 in #3914
• [dlinfer] fix nn layout typo and scale t by @yuchiwang in #3915
• fix chat and warmup of lora adapter by @grimoire in #3911
• build(acsend): try to fix acsend CI docker build by @windreamer in #3906
• fix internvl3 hf by @CUHKSZzxy in #3932
• build(docker): fix ascend tag name by @windreamer in #3939
• put eot_token to stop_words by @lvhan028 in #3941

📚 Documentations

• update proxy docs by @CUHKSZzxy in #3796
• add missing docs by @CUHKSZzxy in #3871
• fix docs by @CUHKSZzxy in #3885
• update news and citation by @lvhan028 in #3889

🌐 Other

• add prometheus client by @CUHKSZzxy in #3792
• fix: add dummy_prefill guard for PD connection operations by @FirwoodLin in #3803
• minor fix about the log level and logs by @lvhan028 in #3758
• assert PytorchEngineConfig block size by @Tsundoku958 in #3826
• [ci] change restful api into openai and add more testcase by @zhulinJulia24 in #3866
• remove ppu backend by @yao-fengchen in #3904
• [ci] remove flash attn installation in ete test workflow by @zhulinJulia24 in #3908
• dlinfer backend support ray by @yao-fengchen in #3903
• style(types): fix return type annotation for get_all_requests by @xiaoajie738 in #3919
• upgrade torch to 2.8.0 and triton 3.4.0 by @lvhan028 in #3930
• Dlinfer readme by @jinminxi104 in #3938
• bump version to v0.10.0 by @lvhan028 in #3933

New Contributors

• @PeymanRM made their first contribution in #3784
• @FirwoodLin made their first contribution in #3803
• @zenosai made their first contribution in https://github.com/In...

v0.9.2.post1

What's Changed

• Fix interns1 LLM mapping for turbomind engine by @lvhan028 in #3848
• bump version to v0.9.2.post1 by @lvhan028 in #3849

Full Changelog: v0.9.2...v0.9.2.post1

v0.9.2

What's Changed

🚀 Features

• [Feature] metrics support by @CUHKSZzxy in #3534
• Relax FP8 TP requirement by @lzhangzz in #3697
• FA3 by @zhaochaoxing in #3623
• support qwen2/2.5-vl in turbomind by @irexyc in #3744
• feat: add pytorch_engine_qwen2_5vl_sm120 by @kolmogorov-quyet in #3750
• Internvl pt by @RunningLeon in #3765
• Improve internvl for turbomind engine by @lvhan028 in #3769

💥 Improvements

• Refactor linear by @grimoire in #3653
• remove python3.8 support and add python3.13 support by @lvhan028 in #3638
• refactor vl inputs split by @grimoire in #3699
• [Fix]: Replace mutable default with default_factory for scheduler_stats by @ConvolutedDog in #3730
• Fix the logic of calculating max_new_tokens and determining finish_reason by @lvhan028 in #3727
• Override HF config.json via CLI by @CUHKSZzxy in #3722
• feat(build): Integrate and build turbomind backend directly in setup.py by @windreamer in #3726
• Generate the benchmark output filename with given arguments by @lvhan028 in #3740
• Make loading llm without vlm as an option by @grimoire in #3745

🐞 Bug fixes

• add ray to ascend requirements by @sigma-plus in #3713
• fix accessing undefined attribute seq_aux of deepseek-r1-0528 by @lvhan028 in #3728
• [Fix]: Avoid quantize qk norm for qwen3 dense models by @taishan1994 in #3733
• fix py313 env creation failed when building lmdeploy-builder image by @lvhan028 in #3739
• [Fix]: kernel meta retrieval for SM7X does not work by @xiaoajie738 in #3746
• limit max_session_len by @grimoire in #3751
• fix internvl norm by @grimoire in #3756
• support qwen3 moe yarn and vlm hf_overrides by @grimoire in #3757
• [PD Disaggregation] fix double unshelf by @JimyMa in #3762
• fix(build): fix version parse regex to support post-release versions by @windreamer in #3764
• adapt transformers>=v4.52.0 to loading qwen2.5-vl with turbomind by @irexyc in #3771
• fix chat template with tool call by @RunningLeon in #3773
• fix vl nothink mode by @RunningLeon in #3776

📚 Documentations

• update reward model docs by @CUHKSZzxy in #3721

🌐 Other

• update twomicrobatch by @SHshenhao in #3651
• [CI]: Upgrade to py310 for ut by @RunningLeon in #3718
• [ci] update dailytest environment and scripts by @zhulinJulia24 in #3716
• Preliminary Blackwell (sm_120a, RTX 50 series) support by @lzhangzz in #3701
• [ci] add fp8 evaluation workflow by @zhulinJulia24 in #3729
• Add VRAM bandwidth utilization stat to attention test by @lzhangzz in #3731
• doc: fix dead links to MindX DL to recover CI. by @windreamer in #3741
• fix free cache in MPEngine branch by @JimyMa in #3670
• fix: make RelWithDebInfo default cmake build type by @windreamer in #3774
• bump version to v0.9.2 by @lvhan028 in #3770

New Contributors

• @sigma-plus made their first contribution in #3713
• @ConvolutedDog made their first contribution in #3730
• @windreamer made their first contribution in #3726
• @taishan1994 made their first contribution in #3733
• @xiaoajie738 made their first contribution in #3746
• @kolmogorov-quyet made their first contribution in #3750

Full Changelog: v0.9.1...v0.9.2

v0.9.1

What's Changed

🚀 Features

• feature: enable tool_call and reasoning_content parsing for qwen3 by @ywx217 in #3615
• Support Mooncake migration backend for PD disaggregation by @Risc-lt in #3620
• Support load fused moe weights by @RunningLeon in #3672
• Seperate api_server and pytorch engine into different processors by @grimoire in #3627
• add reward model api by @CUHKSZzxy in #3665

💥 Improvements

• [ascend]import patch at initiazing time by @JackWeiw in #3662
• [ascend]use custon transdata in python kernel by @JackWeiw in #3671
• move import transformers in patch by @grimoire in #3660
• set ray envs by @grimoire in #3643
• raise ImportError when enable ep and not install dlblas by @zhaochaoxing in #3636
• Reduce sampling memory usage by @lzhangzz in #3666

🐞 Bug fixes

• fix dockerfile by @lvhan028 in #3657
• Fix top-p only sampling with padded vocab size by @lzhangzz in #3661
• fix pt engine stop & cancel by @irexyc in #3681
• Fix convert bf16 to numpy by @RunningLeon in #3686
• disable torch.compile in cuda graph runner by @grimoire in #3691
• fix reward model api by @CUHKSZzxy in #3703

📚 Documentations

• add reward model documents by @CUHKSZzxy in #3706

🌐 Other

• upgrade torch and triton by @grimoire in #3677
• support do_preprocess=False for chat.completions by @irexyc in #3645
• [ci] change flash atten installation in pr test by @zhulinJulia24 in #3688
• fix profile_throughput.py by @irexyc in #3692
• fix profile_generation.py by @irexyc in #3707
• update dlblas version in dockerfile by @CUHKSZzxy in #3711
• bump version to v0.9.1 by @lvhan028 in #3685

New Contributors

• @ywx217 made their first contribution in #3615
• @Risc-lt made their first contribution in #3620

Full Changelog: v0.9.0...v0.9.1

v0.9.0

What's Changed

🚀 Features

• LMDeploy Distserve by @JimyMa in #3304
• allow api server terminated through requests from clients by @RunningLeon in #3533
• support update params for pytorch backend from api server by @irexyc in #3535
• support eplb for Qwen3-MoE by @zhaochaoxing in #3582
• support update params for turbomind backend by @irexyc in #3566
• Quantize Qwen3 MoE bf16 model to fp8 model at runtime by @grimoire in #3631
• [Feat]: Support internvl3-8b-hf by @RunningLeon in #3633
• Add FP8 MoE for turbomind by @lzhangzz in #3601

💥 Improvements

• reduce ray memory usage by @grimoire in #3487
• use dlblas by @zhaochaoxing in #3489
• internlm3 dense fp8 by @CUHKSZzxy in #3527
• random pad input ids by @grimoire in #3530
• ray nsys profile support by @grimoire in #3448
• update blockedfp8 scale name by @CUHKSZzxy in #3532
• start engine loop on server startup event by @grimoire in #3523
• update two microbatch by @SHshenhao in #3540
• [ascend]set transdata dynamic shape true by @JackWeiw in #3531
• ray safe exit by @grimoire in #3545
• support update params with dp=1 for pytorch engine by @irexyc in #3562
• Skip dp dummy input forward by @grimoire in #3552
• Unclock mutual exclusivity of argument: tool-call-parser and reasoning-parser by @jingyibo123 in #3550
• perform torch.cuda.empty_cache() after conversion by @bltcn in #3570
• pipeline warmup by @irexyc in #3548
• Launch multiple api servers for dp > 1 by @RunningLeon in #3414
• support awq for Qwen2.5-VL by @RunningLeon in #3559
• support qwen3 /think & /no_think & enable_thinking parameter by @BUJIDAOVS in #3564
• Eplb by @zhaochaoxing in #3572
• Update benchmark by @lvhan028 in #3578
• block output when prefetch next forward inputs. by @grimoire in #3573
• support both eplb and microbatch simultaneously by @zhaochaoxing in #3591
• Add log_file and set loglevel in launch_servers by @RunningLeon in #3596
• 1. add migration flow control by @JimyMa in #3599
• sampling on the tokenizer's vocab by @grimoire in #3604
• update deepgemm version by @grimoire in #3606
• [Ascend] set default distrbuted backend as ray for ascend device by @JackWeiw in #3603
• Blocked fp8 tma by @grimoire in #3470
• [PDDisaggreagtion] Async migration by @JimyMa in #3610
• move dp loop to model agent by @grimoire in #3598
• update some logs of proxy_server and pt engine by @lvhan028 in #3621
• improve loading model performance by shuffling the weight files by @irexyc in #3625
• add benchmark scripts about pipeline api and inference engines according to the config file by @lvhan028 in #3622

🐞 Bug fixes

• [ascend] fix recompile on different rank by @jinminxi104 in #3513
• fix attention sm86 by @grimoire in #3519
• fix stopwords kv cache by @grimoire in #3494
• [bug fix] fix PD Disaggregation in DSV3 by @JimyMa in #3547
• fix proxy server heart beat by @irexyc in #3543
• fix dp>1 tp=1 ep=1 by @grimoire in #3555
• fix mixtral on new transformers by @grimoire in #3580
• [Fix]: reset step after eviction by @RunningLeon in #3589
• fix parsing dynamic rope param failed by @lvhan028 in #3575
• Fix batch infer for gemma3vl by @RunningLeon in #3592
• Fix symbol error when dlBLAS is not imported by @zhaochaoxing in #3597
• read distributed envs by @grimoire in #3600
• fix side-effect caused by PR 3590 by @lvhan028 in #3608
• fix bug in qwen2 by @LKJacky in #3614
• fix awq kernel by @grimoire in #3618
• fix flash mla interface by @grimoire in #3617
• add sampling_vocab_size by @irexyc in #3607
• fix for default quant by @grimoire in #3640
• Fix log file env in ray worker by @RunningLeon in #3624
• fix qwen3 chat template by @lvhan028 in #3641
• fix vlm runtime quant by @grimoire in #3644
• Fix 'Namespace' object has no attribute 'num_tokens_per_iter' when serving by gradio by @lvhan028 in #3647
• Synchronize weight processing by @lzhangzz in #3649
• Fix zero scale in fp8 quantization by @lzhangzz in #3652

🌐 Other

• update doc for ascend 300I Duo docker image by @jinminxi104 in #3526
• simulate EPLB for benchmark only by @lvhan028 in #3490
• [ci] add test workflow for 3090 machine by @zhulinJulia24 in #3561
• [ci] fix transformers version in prtest by @zhulinJulia24 in #3584
• [Misc] minor api_server and tm loader, and upgrade docformatter to resolve lint error by @lvhan028 in #3590
• [ci] add qwen3 models into testcase by @zhulinJulia24 in #3593
• update Dockerfile by @CUHKSZzxy in #3634
• check in lmdeploy-builder on cuda 12.4 and 12.8 platform by @lvhan028 in #3630
• fix blocked fp8 overflow by @grimoire in #3650
• Bump version to v0.9.0 by @lvhan028 in #3609

New Contributors

• @JimyMa made their first contribution in #3304
• @jingyibo123 made their first contribution in #3550
• @bltcn made their first contribution in #3570
• @BUJIDAOVS made their first contribution in #3564
• @LKJacky made their first contribution in #3614

Full Changelog: v0.8.0...v0.9.0

v0.8.0

What's Changed

🚀 Features

• Torch dp support by @grimoire in #3207
• Add deep gemm with tma pre allocated by @AllentDan in #3287
• Add mixed DP + TP by @lzhangzz in #3229
• Add Qwen3 and Qwen3MoE by @lzhangzz in #3305
• [ascend] support multi nodes on ascend device by @tangzhiyi11 in #3260
• [Feature] support qwen3 and qwen3-moe for pytorch engine by @CUHKSZzxy in #3315
• [ascend]support deepseekv2 by @yao-fengchen in #3206
• add deepep by @zhaochaoxing in #3313
• support ascend w8a8 graph_mode by @yao-fengchen in #3267
• support all2all ep by @zhaochaoxing in #3370
• optimize ep in decoding stage by @zhaochaoxing in #3383
• Warmup deepgemm by @grimoire in #3387
• support Llama4 by @grimoire in #3408
• add twomicrobatch support by @SHshenhao in #3381
• Support phi4 mini by @RunningLeon in #3467
• [Dlinfer][Ascend] support 310P by @JackWeiw in #3484
• support qwen3 fp8 by @CUHKSZzxy in #3505

💥 Improvements

• Add spaces_between_special_tokens to /v1/interactive and make compatible with empty text by @AllentDan in #3283
• add env var to control timeout by @CUHKSZzxy in #3291
• refactor attn param by @irexyc in #3164
• Verbose log by @grimoire in #3329
• optimize mla, remove load v by @grimoire in #3334
• support dp decoding with cudagraph by @grimoire in #3311
• optimize quant-fp8 kernel by @grimoire in #3345
• refactor dlinfer rope by @yao-fengchen in #3326
• enable qwenvl2.5 graph mode on ascend by @jinminxi104 in #3367
• Add AIOHTTP_TIMEOUT env var for proxy server by @AllentDan in #3355
• disable sync batch on dp eager mode by @grimoire in #3382
• fix for deepgemm update by @grimoire in #3380
• Add string before hash tokens in blocktrie by @RunningLeon in #3386
• optimize moe get sorted idx by @grimoire in #3356
• use half/bf16 lm_head output by @irexyc in #3213
• remove ep eager check by @grimoire in #3392
• Optimize ascend moe by @yao-fengchen in #3364
• optimize fp8 moe kernel by @grimoire in #3419
• ray async forward execute by @grimoire in #3443
• map internvl3 chat template to builtin chat template internvl2_5 by @lvhan028 in #3450
• Refactor turbomind (low-level abstractions) by @lzhangzz in #3423
• remove barely used code to improve maintenance by @lvhan028 in #3462
• optimize sm80 long context by @grimoire in #3465
• move partial_json_parser from ’serve.txt‘ to ‘runtime.txt‘ by @lvhan028 in #3493
• support qwen3-dense models awq quantization by @lvhan028 in #3503
• Optimize MoE gate for Qwen3 by @lzhangzz in #3500
• Pass num_tokens_per_iter and max_prefill_iters params through in lmdeploy serve api_server by @josephrocca in #3504
• [Dlinfer][Ascend] Optimize performance of 310P device by @JackWeiw in #3486
• optimize longcontext decoding by @grimoire in #3510
• Support min_p in openai completions_v1 by @josephrocca in #3506

🐞 Bug fixes

• fix activation grid oversize by @grimoire in #3282
• Set ensure_ascii=False for tool calling by @AllentDan in #3295
• fix sliding window multi chat by @grimoire in #3302
• add v check by @grimoire in #3307
• Fix Qwen3MoE config parsing by @lzhangzz in #3336
• Fix finish reasons by @AllentDan in #3338
• remove think_end_token_id in streaming content by @AllentDan in #3327
• Fix the finish_reason by @AllentDan in #3350
• set cmake policy minimum version as 3.5 by @lvhan028 in #3376
• fix dp cudagraph by @grimoire in #3372
• fix flashmla eagermode by @grimoire in #3375
• close engine after each benchmark-generation iter by @grimoire in #3269
• [Fix] fix image_token_id error of qwen2-vl and deepseek by @ao-zz in #3358
• fix stopping criteria by @grimoire in #3384
• support List[dict] prompt input without do_preprocess by @irexyc in #3385
• add rayexecutor release timeout by @grimoire in #3403
• fix tensor dispatch in dynamo by @wanfengcxz in #3417
• fix linting error by upgrade to ubuntu-latest by @lvhan028 in #3442
• fix awq tp for pytorch engine by @RunningLeon in #3435
• fix mllm testcase fail by @caikun-pjlab in #3458
• remove paged attention autotune by @grimoire in #3452
• Remove empty prompts in benchmark scripts by @lvhan028 in #3460
• failed to end session properly by @lvhan028 in #3471
• fix qwen2.5-vl chat template by @CUHKSZzxy in #3475
• Align forward arguments of deepgemm blockedf8 by @RunningLeon in #3474
• fix turbomind lib missing to link nccl by exporting nccl path by @lvhan028 in #3479
• fix dsvl2 no attr config error by @CUHKSZzxy in #3477
• fix flash attention crash on triton3.1.0 by @grimoire in #3478
• Fix disorder of ray execution by @RunningLeon in #3481
• update dockerfile by @CUHKSZzxy in #3482
• fix output logprobs by @irexyc in #3488
• Fix Qwen2MoE shared expert gate by @lzhangzz in #3491
• fix replicate kv for qwen3-moe by @grimoire in #3499
• fix sampling if data overflow after temperature penalty by @irexyc in #3508

📚 Documentations

• update qwen2.5-vl-32b docs by @CUHKSZzxy in #3446

🌐 Other

• bump version to v0.7.2.post1 by @lvhan028 in #3298
• [ci] add think function testcase by @zhulinJulia24 in #3299
• merge dev into main by @lvhan028 in #3348
• [ci] add vl models into pipeline interface testcase by @zhulinJulia24 in #3374
• merge dev to main branch by @lvhan028 in #3378
• opt experts memory and permute by @zhaochaoxing in #3390
• Revert "opt experts memory and permute" by @zhaochaoxing in #3406
• merge dev to main by @lvhan028 in #3400
• add Hopper GPU dockerfile by @CUHKSZzxy in #3415
• optimize internvit by @caikun-pjlab in #3433
• fix stop/bad words by @irexyc in #3492
• [ci] testcase bugfix and add more models into testcase by @zhulinJulia24 in #3463
• bump version to v0.8.0 by @lvhan028 in #3432

New Contributors

• @zhaochaoxing made their first contribution in #3313
• @ao-zz made their first contribution in #3358
• @wanfengcxz made their first contribution in #34...