4.25
LEAD STORY ·
GoogleGoogle Agent traffic is multi-step, shares long prefixes and arrives in bursts. Behind an ordinary round-robin load balancer, replicas recompute the same prompts and accelerator memory fills up with duplicate KV cache. llm-d 1.0, released at the PyTorch Conference by Google together with Red Hat, IBM and others, is an open-source distributed inference stack that runs on Kubernetes and organizes a set of vLLM and SGLang model servers into one serving pool: the router tracks which replicas hold each cached prefix and picks a replica using load and predicted latency; when accelerator memory fills up, KV cache is moved first to host memory and then to SSD; prefill and decode are split into two pools. Version 1.0 adds flow control that queues requests by priority and fairness, and rolling upgrades that do not interrupt sessions. On an agentic coding workload on Ironwood TPU, prefix-aware routing plus host-memory offload cut median time-to-first-token by more than 85% and delivered 2.4x the total throughput of a plain Kubernetes Service. The gain comes mainly from sending a session's requests back to the replica that already holds its context, which removes repeated prefill.
3.80
DrugCLIP embeds protein pockets and small molecules into one vector space, which turns drug screening into vector retrieval: one campaign queries 500M molecules with over 20K pockets, keeps 500K candidates per pocket, and needs 99% mean recall relative to exact search. With answers this large, an IVF index must probe almost every list and a graph index must track at least K candidates, so little of the library can be skipped, and the bottleneck becomes maintaining each query's K candidates. SweepANN scans instead: INT8-compressed molecule vectors stay resident in the memory of 4 GPUs, and each block is read only once for a whole batch of pockets; it first takes each pocket's top K over a sample of blocks and uses the lowest of those scores as a cutoff, after which only rows above the cutoff enter the buffer. Four RTX PRO 6000 GPUs finish 26,562 pockets in about 61 seconds at 99.6% mean recall; GPU IVF configurations that reach the recall target take 8.2–24x as long.
3.35
IBMIBM Research's introduction to llm-d 1.0 focuses on day-to-day operations after deployment. The five techniques llm-d has always worked on are prefix-cache-aware routing, KV-cache offloading across GPU, CPU and storage, batch inference optimization, prefill/decode disaggregation and wide expert parallelism; 1.0 adds no new technique, and the change is that they are integrated to the point where they can be operated long-term: a stable API, multiple tenants sharing the same hardware by priority, and upgrades that do not interrupt service. Multi-turn requests are pinned to the same replicas so that their prefix caches stay usable. In one agentic coding workload test, the prefix-aware router captured 97% of the available cache reuse; IBM and Red Hat ran 3,000 software-engineering agents concurrently on a self-hosted deployment at one-fifth to one-tenth of the price of buying the same open model as a service. Agents keep returning to the same code, documents and instructions, so pinning a session to the same replica pays off more than in chat.
3.35
AmazonAmazon released TorchNeuron, which lets existing PyTorch code run on AWS Trainium chips by changing a single line of configuration, open-sourced under the Apache 2.0 license. Amazon says that when the same training script ran on a conventional chip and on Trainium, the model learned the same things at the same rate, and the tools used for testing, debugging and scaling need no changes. The voice AI company Cartesia says small, fast ops go through the native path directly, so its team only has to hand-write NKI kernels for the layers that are called repeatedly and drive end-to-end latency. Amazon is also contributing changes upstream to PyTorch to remove code that ties the framework to specific chips. For inference teams, what it lowers is the cost of rewriting and revalidating code when switching hardware; the article gives no performance comparison between Trainium and GPUs.
3.30
HHysen Labs
FastLLM is an LLM inference engine written in C++ that does not depend on PyTorch, reads Hugging Face weights directly, and serves OpenAI- and Anthropic-compatible endpoints. Its core design is that model weights do not have to fit entirely in GPU memory: ordinary layers and MoE expert layers can be placed separately on GPU, CPU, NUMA memory or disk, in proportions the user chooses. The author says that on a dual 9004/9005 server with one GPU, full-precision DeepSeek reaches 20 tokens/s at single concurrency, the INT4 version 30 tokens/s, and more than 60 under multiple concurrent requests; these are the author's own numbers, and the repository has no benchmark data to verify them. It targets machines whose GPU memory cannot hold the model but which have plenty of host memory and SSD, and when running MoE models it gives precise control over which layers live where.
3.20
AlibabaThis is a vector-database selection article on the Alibaba Cloud developer community that introduces TairVector, the vector engine built into Tair (Redis Enterprise Edition): it supports HNSW and FLAT indexes, hybrid retrieval that combines vector search with BM25 full-text search, and millisecond-level retrieval latency. The article argues that RAG systems no longer need to bring in Milvus or Qdrant, since one Tair deployment can handle caching, vector retrieval and state management at once, and it reports 510,000 QPS on a single node. The main selling point of this approach is one less standalone vector database to maintain; the summary does not state the vector dimension, data size or recall behind the 510,000 QPS figure, so a selection decision should re-test on your own data.
3.10
This is a measurement paper from Dell's Office of the CTO on how to choose a parallelism scheme for LLM inference. Inference has two phases, prefill and decode: prefill determines time-to-first-token (TTFT), and decode determines the latency between consecutive tokens (ITL). The authors ran Qwen2.5-14B with vLLM on 8 H200 GPUs, compared tensor parallelism (TP), pipeline parallelism (PP) and hybrid configurations, and plotted Pareto frontiers of TTFT against ITL. The more prefill-heavy the workload, the more the best configuration shifts from TP-dominant to PP-dominant; under strict TTFT requirements PP-dominant configurations do better, and as the SLO is relaxed, TP-dominant configurations win because of their higher throughput; as context grows longer, PP-dominant configurations scale better. The best parallelism scheme changes with workload mix, SLO and context length, so it is better treated as a parameter that can be tuned at runtime.
3.05
BBaseCompute
Superfluid is a local LLM server open-sourced by BaseCompute (Apache-2.0) for the case where several agents run on one machine at the same time; it is compatible with the OpenAI, Anthropic and Ollama APIs and can use llama.cpp, MLX or its own BaseRT engine underneath. Interactive requests preempt running batch work at the next scheduler tick; every token is written to a log before it is returned, so after a worker crash a session resumes where it stopped; 32 requests sharing the same system prompt need only one prefill. Running Qwen3.8-27B at 4 bits on an Apple M5 Pro with 8 agents working in the background, a chat request's time-to-first-token is 1.5 s, versus 291 s for llama-server. The gap comes from priority preemption: with the same model and runtime, the chat request no longer waits behind batch work.