vLLM releases v0.34.0 with extensible KV cache enabled by defaultMachine translation
vLLM 发布 v0.34.0,默认启用可扩展 KV 缓存
vLLM v0.34.0 adds support for 10 models and sizes the KV cache using memory actually free after warmup. The change can improve throughput and reduce time to first token at high concurrency, but per-token decode latency can rise on some models. The older GPU runner has also been removed, so existing deployments need a compatibility check before upgrading.
The complete source text is not yet available.
Read at the original source