A year ago, most inference traffic was still chat: one prompt in, one response out. Now more and more requests come from multi-step agents and reinforcement-learning rollouts, which fan out across several tools, share very long prefixes and arrive in bursts. Put this traffic on a single machine, or behind an ordinary round-robin load balancer, and the replicas recompute the same prompts, accelerator memory fills up with duplicate KV cache, and one bursty agent loop can stall interactive users' requests. llm-d 1.0, released at the PyTorch Conference by Google together with Red Hat, IBM and others, is an open-source distributed inference stack that runs on Kubernetes and organizes a set of vLLM and SGLang model servers into one serving pool: the router knows which replicas hold each cached prefix and sends a session's requests back to them; when accelerator memory runs short, KV cache is moved to host memory and storage; high-priority requests can jump the queue. On an agentic coding workload on Ironwood TPU, prefix-aware routing plus host-memory offload cut median time-to-first-token (TTFT) by more than 85% and delivered 2.4x the total throughput of a plain Kubernetes Service. Most of these gains come from no longer recomputing prefixes that have already been computed, not from faster kernels.
This is a Google Cloud product blog, read together with IBM Research's introduction from the same day. Below, I first explain the problem llm-d solves, then break down four components, routing, KV-cache tiering, prefill/decode disaggregation and flow control, and finally look at the production numbers the blog gives. All numbers come from these two blogs, and most of them come without a full experimental setup.
1. The problem: why agent traffic breaks ordinary load balancing
LLM inference has two steps: prefill reads the whole prompt once and computes the KV cache for every token; decode then generates new tokens one by one from that cache. The compute of prefill grows with prompt length. An agent calls the model many times within one task, and every prompt contains the same system instructions, code and conversation history, with only a few new turns appended at the end.
If the second call is sent round-robin to a different replica, that replica has no cache from the first call and has to prefill the whole prefix again. The more replicas there are, the lower the chance of landing on the original one. IBM's blog puts it directly: if every step requires the model to reread the same material, you are repeatedly paying for work the system has already done.
The second problem is mixed load. Interactive requests need low latency, background agent loops and RL rollouts need throughput, and demand swings by the minute. Without priorities, interactive requests queue behind a large batch of background requests.
2. Method: four components
Cache-aware routing. The llm-d router is built on the Gateway API Inference extension for Kubernetes. It tracks which replicas hold each cached prefix and, when choosing a replica, weighs that information together with replica load and predicted latency. Version 1.0 adds session-aware routing and turn-priority scheduling, and stores Responses API state natively at the routing layer so agents can reuse context across turns. In IBM's words, multi-turn requests stay pinned to the appropriate replicas so that their prefix caches remain usable.
Tiered KV-cache offloading. Once accelerator memory is full, llm-d does not simply drop old cache; it uses vLLM's OffloadingConnector, Mooncake or SGLang's HiCache to move it to host memory, and further down to SSD or Google Cloud Managed Lustre. The point is to avoid many prefixes being pushed out of accelerator memory at once under multi-turn, long-context load and then recomputed all together; the blog calls this a prefix-eviction storm.
Prefill/decode disaggregation. Prefill is compute-heavy and decode is bound by memory bandwidth, so running both on the same replicas makes each wait on the other. llm-d splits prefill, decode and multimodal encode into separate pools, and uses wide expert parallelism to spread large MoE models across nodes. DisaggregatedSet, new in 1.0 and built on LeaderWorkerSet, the Kubernetes API for multi-host inference, manages these pools as one unit: during rolling updates, prefill and decode move together to matching versions, while scaling stays independent for each.
Flow control and async processing. Flow control queues requests by priority and fairness, so latency-sensitive requests and bursty agent workloads can share one set of hardware without starving each other. Version 1.0 also adds llm-d-async-processor, which pulls latency-tolerant requests from queues such as Pub/Sub or Redis, routes them alongside real-time traffic, runs them only on spare capacity, and enforces strict priority lanes and quotas.
This figure also shows where llm-d sits. It does not replace vLLM or SGLang; it sits in front of them and does the scheduling. Teams can keep their existing inference engine, models and cluster, and plug in only the components they need; the order the blog recommends is to first add prefix-aware routing in front of an existing vLLM deployment, then add flow control, KV-cache offloading and prefill/decode disaggregation.
3. What is new in 1.0
IBM's blog points out that the five techniques, prefix-cache-aware routing, KV-cache offloading, batch inference optimization, prefill/decode disaggregation and wide expert parallelism, existed before 1.0; what changes in 1.0 is that they can run long-term as a critical service. Specifically:
- Multi-tenant quality of service: flow control by priority and fairness.
- Stability: v1 APIs and a hardened request path, validated on multiple clouds and accelerator types.
- Safer upgrades: router high availability, zero-downtime updates, and the DisaggregatedSet described above.
- Agent-aware serving: session-aware routing, turn-priority scheduling and native Responses API state.
- TPU support: TorchTPU runs PyTorch models natively on TPUs inside vLLM and SGLang, with cooperative time-slicing.
Time-slicing targets idle time in RL training. The blog says that during rollout generation or weight updates, accelerators sit idle 40% to 60% of the time; llm-d fits async inference into these gaps, raising aggregate accelerator duty cycles from a baseline of about 40% to more than 70%.
4. Experiments and production numbers
The blog's numbers fall into two kinds: Google's own tests and production traffic, and user reports. Each set gives only the comparison and the result, without a full experimental configuration; below they are organized by what they aim to show, what they compare against and the result.
| Scenario | Compared against | Result |
|---|---|---|
| Agentic coding workload on Ironwood TPU (TPU 7x), prefix-aware routing + host-memory offload | Plain Kubernetes Service | Median TTFT down more than 85%, total throughput 2.4x |
| Gemini platform (formerly Vertex AI) production traffic through GKE Inference Gateway | Not stated in the blog | Prefix-cache hit rate from 35% to 70%; TTFT for coding agents down more than 35%; P95 TTFT under bursty chat traffic down more than 50% |
| 17,000-node multi-region GKE deployment across the US and Europe serving an MoE model | Not stated in the blog (result is for three clusters combined) | Throughput scaled 2.9x across three clusters, 99.9% success rate, routing overhead below 1% |
| A3 Mega cluster serving a 70B model, 50k-token prompts, KV cache offloaded to both Managed Lustre and host memory | Offload to host memory only | TTFT improved about 40%, end-to-end latency down 30%, GPU-hour requirement down nearly 60% |
| Agentic coding workload test (IBM) | Upper limit of reusable cache | Prefix-aware router captured 97% of the available cache reuse |
| 3,000 concurrent software-engineering agents, self-hosted deployment (IBM and Red Hat) | Buying the same open model as a service | Cost 5 to 10 times lower |
The first row answers how much cache-aware routing itself brings: on the same hardware, simply replacing round-robin with prefix-based routing and adding host-memory offload cut median TTFT by more than 85%. The second row is a check on production traffic: the prefix-cache hit rate doubled, consistent in direction with the drop in TTFT. The third row tests scale: routing overhead below 1% shows that the router's own computation did not become a bottleneck. The fourth row compares two KV-cache offloading schemes; with long 50k-token prompts, adding a shared storage tier has a clearer effect.
Among user reports, Snap reached prefix-cache hit rates of up to 75–80% with prefix-cache-aware routing; ZenteiQ raised daily generated tokens per 1,000 TPU v6e accelerators from 1.47 billion to 5.97 billion, more than 4x higher aggregate throughput; Mistral uses flow control to enforce strict priorities across its commercial tiers, automatically holding or shedding lower-priority requests when pools saturate.
5. Limitations
The blog itself does not discuss limitations. Two facts to add: first, most numbers in the table come with only a comparison and a result, so the baseline configuration, workload mix and measurement method need to be checked in the original sources linked from the blog; second, many capabilities (GKE Inference Gateway's managed routing, cross-region failover, the Managed Lustre storage tier) are Google Cloud managed services, and on other clouds or self-built clusters you need to deploy the corresponding open-source components yourself.
6. Practical takeaways
- If you serve agents or multi-turn chat with vLLM or SGLang and still have round-robin load balancing in front, check the prefix-cache hit rate first. If it is low, prefix-based routing is the step with the smallest change and the most direct gain.
- When interactive requests and batch jobs share the same GPUs, flow control and async queues let batch jobs use only spare capacity, so interactive latency is not affected by batch work.