vllm

v0.33.0

Highlights

This release features 927 commits from 389 contributors (122 new)!

• New models: Reflection Beam-1.0 (501B MoE, 23B active) with its DFlash2 drafter (#62074), Yandex AliceAI-Foundation-80B-A3B (#57963), Recursive Flint (40B MoE, 2.5B active) (#62938), Muse Spark Open (#61879), Step-5 Preview (#61727, #63147), NVIDIA Alpamayo 2 Super (#62957), Nemotron-Labs Dory1.0-4B (#60149), Nemotron Parse 3.5 (#61651), MiniCPM-OCR 1.0 (#63095), Domyn Edge (#62544) and the ColUltraVBERT visual document retriever (#62237).
• PyTorch 2.14: vLLM is upgraded to PyTorch 2.14.0, with torchvision 0.29.0 and Triton 3.8.0 (#52183).
• CUDA 12.9 dropped: release wheels and images are CUDA 13.0 only. The CUDA 12.9 wheels and the -cu129 / -cu129-ubuntu2404 images are no longer published (#62575); use the default CUDA 13.0 artifacts instead.
• libtorch ABI stable CUDA wheel: the CUDA wheel is now ABI stable with libtorch.
• AWS EFA image: a new vllm/vllm-openai:v0.33.0-efa image (x86_64 and aarch64) bundles the AWS EFA libraries and the EFA build of the Mooncake transfer engine, for KV transfer and multi-node serving on EFA-enabled EC2 instances (#58768).
• vllm/vllm image alias: every release image is now also published under the shorter vllm/vllm name, e.g. vllm/vllm:v0.33.0, vllm/vllm:v0.33.0-efa and vllm/vllm:v0.33.0-rocm.
• Speculative decoding: the new XPress refiner for block-diffusion drafters such as DFlash and DSpark (#54448); FLy, an opt-in lossy verification policy with 1.04-1.31x decode throughput on MI355X and B300 (#53987); DFlash/DSpark no longer reserve draft slots in the token budget, taking DeepSeek-V4.1-Flash + DSpark on GB300 from 1373 to 3390 output tok/s in the PR's benchmark (#59468); quantized draft lm_head (draft_lm_head_quantization, +18% decode at concurrency 1) (#59973); lookahead block hashes so EAGLE/MTP prefix-cache hits keep their last block, on by default (#61589); and MTP drafts in the Transformers modeling backend (#61270).
• Faster decode on NVIDIA: opt-in latency-specialized decode on GB300 for Kimi-K3 (1.5-2x at TP8/TP16, #61867) and DeepSeek-V4.1-Flash (18.3% faster decode step at batch 1, #61868); DeepSeek-V4/V4.1 MXFP4 experts run as block FP8 through Humming on Hopper (+9-21% decode on 4x H200, #61875); FA4 sparse MLA decode on Blackwell (#55866); FlashInfer DCP prefill on trtllm-gen with 39-48% lower mean TTFT at 100k input on 8xB200 (#62029); and decode-only full CUDA graphs under MRV2 prefill context parallelism (2.6x over eager, #62377).
• AMD ROCm: the default ROCm image and wheels move to ROCm 10.0, with ROCm 7.2 still published as -rocm72 (#58761); MonoKernel decode via --monokernel for DeepSeek-V4.1-Flash on MI300X/MI325X (2.32x P50 interactivity at concurrency 1, #60731) and on every MI355X layer (#61082), and for MiniMax-M3-MXFP4 on MI355X (16.6% lower TPOT, #62712, #62994); the ROCM_OCTAVE 3-4 bit KV cache (4.6x the KV tokens of fp16 on RX 7900 XTX, #59774); and gfx1250 enablement for gpt-oss-120b, DeepSeek-R1, DeepSeek-V4-Flash and MiniMax-M3 (#58800).
• API and serving: the OpenAI-compatible /v1/decisions endpoint, on by default for supported generation models (#60465, #61550); token embeddings with optional late chunking on /v1/embeddings (#61772, #63051, #60164); output_mode="derender" on /inference/v1/generate (#60082); a large Rust frontend parity push, including no more HTTP 500 on long conversations (#62374) and unsupported flags failing at startup (#62431); concurrent worker startup with TP > 1 (#60028); and a watchdog that dumps stack traces from stuck engine and worker processes (#55700).
• Breaking changes: Medusa and the MLP speculator removed (#61701); the legacy Ray compiled-graph executor removed (#61712, #62228); --mamba-cache-mode deprecated (#60043); requests sent while generation is paused now get HTTP 503 (#51488); the NIXL connector needs NIXL 1.5 or newer (#61032); LM Format Enforcer and --ir-op-priority deprecated (#63423, #63371).

Release Artifacts

Python Wheels

| Platform | Install |

|---|---|

| PyPI (CUDA 13.0) | pip install vllm |

| PyPI (CUDA 13.0, uv) | uv pip install vllm --torch-backend=auto |

| ROCm 10.0 | pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.33.0/rocm100 |

| ROCm 7.2 | pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.33.0/rocm72 |

| XPU | uv pip install vllm --extra-index-url https://wheels.vllm.ai/0.33.0/xpu --extra-index-url https://download.pytorch.org/whl/xpu --index-strategy unsafe-best-match |

Docker Images

| Platform | Docker Image |

|---|---|

| CUDA 13.0 (Default) | docker pull vllm/vllm-openai:v0.33.0 |

| CUDA 13.0 + AWS EFA | docker pull vllm/vllm-openai:v0.33.0-efa |

| ROCm 10.0 | docker pull vllm/vllm-openai-rocm:v0.33.0 |

| ROCm 7.2 | docker pull vllm/vllm-openai-rocm:v0.33.0-rocm72 |

| CPU | docker pull vllm/vllm-openai-cpu:v0.33.0 |

| XPU | docker pull vllm/vllm-openai-xpu:v0.33.0 |

Every image is also available under the shorter vllm/vllm name: vllm/vllm:v0.33.0, vllm/vllm:v0.33.0-efa, vllm/vllm:v0.33.0-rocm (-rocm100, -rocm72), vllm/vllm:v0.33.0-cpu and vllm/vllm:v0.33.0-xpu. latest works the same way, e.g. vllm/vllm:latest-rocm.

Other Artifacts

Pre-built release artifacts are available in the Assets section at the bottom of this page, including:

• Source distribution tarball
• CUDA 13.0 Python wheels for x86_64 and arm64
• CPU Python wheels for x86_64, arm64, and macOS
• XPU Python wheel for x86_64

Model Support

• New models: Reflection Beam-1.0 (BeamForCausalLM) day-0 support with its DFlash2 drafter and fused Blackwell kernels (#62074, #62042), plus beam reasoning and tool parsers (#62052); Yandex AliceAI-Foundation-80B-A3B (Kimi Delta Attention hybrid, 512-expert MoE, 262K context) (#57963); Recursive Flint BF16 and NVFP4 GGUF checkpoints with small-batch Blackwell decode kernels (#62938); Muse Spark Open (#61879) with reasoning and tool parsers (#62207); Step-5 Preview (#61727, #62126, #62132), running on SM100 (#63147) with LoRA (#62498); NVIDIA Alpamayo 2 Super (#62957); Nemotron-Labs Dory1.0-4B (#60149); Nemotron Parse 3.5 with an MTP proposer on MRV2 (#61651); MiniCPM-OCR 1.0 (#63095); Domyn Edge (#62544); ColUltraVBERT (#62237); LFM2.5-Embedding-350M (Lfm2BidirectionalModel, #62201); Kolibri 1 through the Transformers backend (#60827); and Decision 3.0 models (#61565, #61778) and Cloudflare Clef-Flash (#61308) on /v1/systemone.
• DeepSeek-V4 and V4.1: length-aware decode indexer block selection (about 850 us saved per step at 1M context, #61039); MegaMoE experts loaded directly in the DeepGEMM layout so reload_weights works (#60096); fused_wqa_wkv split by token across TP on SM90 (#58872); MegaAttn wo_a writes MXFP8 input for wo_b (#60118); native small-head FlashMLA sparse prefill on SM90 (#61615); FlashMLA interleaved KV layout (#61371); redundant EPLB expert slots loaded (#59611); GPU Engram tables read from peer HBM, fixing DEP decode hangs on GB200 (#62091); and fixes for Engram under PP (#62208), Engram all-to-all hangs (#62269), prefill index-workspace NaNs (#60613), C128 compressed-KV writes under PIECEWISE graphs (#59709), routed-expert rows under replay (#61643, #62652) and the updated 1017 reference encoding (#62199, #62200).
• GLM: GLM-5.3-Flash defaults to an FP8 KV cache (9.8% higher end-to-end throughput in the PR's benchmark, #61228) and gets mHC TileLang decode kernels on Blackwell (#62745); GLM-5.3 per-token-group quant (2.8% ITL, #60927), in-place draft decode metadata (#61873) and skipped concat_and_cache_mla scales (1.2-3.0% TPOT on B300, #62102); FP8 indexer WK pairs kept across load_weights calls, fixing thinking loops with runai_streamer (#59223).
• Kimi-K3: KDA RecoverSSM CuTe DSL verify/commit kernels on SM10x, with verify records moved out of the KV cache so KV connectors work (#63045); online quantization of NVIDIA projections (#62980).
• Qwen: Qwen3.8-Flash-Next FP8 QSA KV cache on Ampere (#60602, #60992), packed NVFP4 PLE embeddings that fit Qwen3.8-Flash-Next-NVFP4 on one DGX Spark (#56273) and PLE short-conv prefill checkpoints (#61601); FlashInfer GDN prefill checkpoints for Qwen3-Next / Qwen3.5 (#60050).
• Other models: Cohere2MoE attention/MoE dual stream, single TP all-reduce and fused add + RMSNorm (1.14-1.36x TPOT, on by default, #62101); Beam TP all-reduce fused with norms and MoE finalize (#62366); MiniMax-M3 CUTLASS MSA decode crossover (#55838); Granite MoE as an EAGLE3/DFlash/DSpark target (#61110); NemotronHMTP sharing target embeddings (#62194); LoRA for BERT cross-encoders and token classification (#61279, #61391).
• Multimodal: PaddleOCR-VL batched vision encoder (34-85% more throughput, #61458) and compile_mm_encoder (#61838); DeepSeek-VL2 ViT full CUDA graphs (#46005); Gemma4Unified batched embedding (#60643); Pixtral real-valued RoPE under compile (#63145); and fixes for EXAONE-4.5 images (broken since v0.29.0, #61104), Gemma 4 encoder graphs (#61365, #60687, #63538), DeepSeek-VL2 encoder cache and PP (#63120, #63192) and Gemma 4 placeholders (#61196).
• Transformers modeling backend: MTP speculative drafts (#61270), fused post-top-k MoE routers (#57410), graph caching instead of AOT caching (#58918) and dynamic-RoPE CUDA graphs (#58064).

Engine Core

• Speculative decoding: XPress (#54448), FLy (#53987), no draft-slot budget reservation for DFlash/DSpark (#59468), quantized draft lm_head (#59973), lookahead block hashes (#61589), MRV2 skipping drafts no request can use (up to 7.3% prefill throughput with DFlash, #62784), Triton split-K verify (#56148), varlen Mamba2 decode for adaptive verification (#53036), and fixes for stop-trimmed token counts (#58668), block verification randomness (#61161), MTP block tables (#61174), grammar masks (#52477), EAGLE SWA prefix hits (#62206), DFlash/DSpark Mamba checkpoint reuse (#54163) and PP warmup (#62129).
• Model Runner V2: batch-sharded sampling for GPT-OSS (#61257) with adaptive verification (#58466), 31-120x faster long-prompt staged writes on GB200 (#62698), encoder towers skipped on later PP stages (0.58-1.27 GiB per stage, #60238), async scheduling kept for ngram (#62521), input_ids on later PP stages (#61016), and memory-profiling fixes (#57926, #58243, #61509, #63431).
• Sampling: Triton top-k/top-p skips the cutoff search for masked logits (up to about 8x, #61213); small-batch split top-p (1.22-2.70x on B300, #61996); and fixes for huge logits and boundary ties (#58231, #62046).
• Startup and reliability: concurrent local worker startup with TP > 1 (about 50s saved at TP8, #60028); shared Triton JIT compile pool (9-28% faster cold warmup, #61422); engine and worker watchdog (#55700); libgomp fork deadlock fixed (#62884); Fast Start with expert parallelism (#55490) and daemon fixes (#62583, #63451, #63088, #62734, #63506); and SHM broadcast reader memory growth fixed (#63165).
• Memory and performance: UVA-offloaded MoE experts on the sorted fused-MoE path (+7-19%, #62054); DispatchPooler bypass for homogeneous pooling (up to 10.3%, #62940); linear-time encoder cache free (#62850); sliding-window block tables null freed entries (#60767).
• LoRA: expert LoRA for MXFP4 MoE on flashinfer_trtllm (#61979), text-only LoRAs on multimodal wrappers (#62224), unsupported models rejected at validation (#61884), and fixes for load_inplace reloads (#41482), max_loras accounting (#60306), SM90 lora_expand NaNs (#61906) and wake-up from level-2 sleep (#62358).
• RL: per-choice return_weight_versions (#61347), NCCL M2N transfer with per-parameter meshes (#60934), parameter metadata sent once (#35911), the ParamMeta / iter_full() / iter_local() weight-transfer API (#63256), and a silent MoE weight corruption fix with packed NCCL transfer plus layerwise reload (#62555).
• Batch invariance: compressed-tensors W4A16/W8A16 (#60524), GateLinear routers (#59985), online FP8 MoE (#57092), and inductor batch-invariant mode under VLLM_BATCH_INVARIANT (#61118).

Large Scale Serving

• Context parallelism: MRV2 PCP decode full CUDA graphs (#62377); PCP for DeepSeek-V4.1-Flash with prefix caching and SWA bounded replay (#59857, #62183, #62384); FlashInfer DCP prefill on trtllm-gen (#62029); direct DCP all-to-all with strided LSE (#60373); FlashMLA sparse on the a2a DCP backend (#61285).
• HiSparse: direct reads of GPU-resident KV, fixing OOM with long prompts and a 3.3x slowdown on 1M-token prefill (#60417); faster host pool prefault (#60869); and P/D correctness fixes (#61659, #62739, #63827).