INFRA · 2026-10-24

Infra — Paper Daily, 2026-10-24

Six of today's 8 papers are about LLM inference serving. llm-d 1.0 handles agent traffic with prefix-cache-aware routing and separate prefill and decode. FastLLM and Superfluid are local inference servers that can place weights in CPU memory or even on disk. Dell compares the latency trade-offs of tensor and pipeline parallelism, and Amazon's TorchNeuron runs PyTorch code on Trainium with a one-line configuration change. The other two cover vector search infrastructure: SweepANN keeps half a billion molecule vectors in GPU memory and scans them directly, and Tair builds a vector engine into Redis.

4.25
LEAD STORY · Google

1. llm-d 1.0, an open source stack for distributed inference

Google

More and more inference requests now come from multi-step agents and reinforcement-learning rollouts, which share very long prefixes and arrive in bursts. Put this traffic behind an ordinary round-robin load balancer, and the replicas recompute the same prompts while accelerator memory fills up with duplicate KV cache. llm-d 1.0 is an open-source distributed inference stack that runs on Kubernetes and organizes a set of model servers into one serving pool. The router knows which replicas hold each cached prefix and sends a session's requests back to them. When accelerator memory runs short, KV cache is moved to host memory and storage. On an agentic coding workload on Ironwood TPU, median time-to-first-token fell by more than 85%, and total throughput was 2.4x that of a plain Kubernetes Service. Most of these gains come from no longer recomputing prefixes that have already been computed, not from faster kernels.

📖 Read the deep dive →
3.80

2. SweepANN: A GPU-Accelerated Large-K Approximate Vector Search for Drug Discovery

Drug screening needs hundreds of thousands of best-matching candidates from hundreds of millions of molecules for each protein pocket, and for such searches a full scan with a score cutoff beats building an approximate index. SweepANN splits the compressed molecule vectors across several GPUs, and one matrix multiplication scores a data block for a whole batch of pockets. The cutoff is set like this: each GPU first scans a few sample blocks and keeps, for each pocket, as many top-scoring molecules as will finally be returned. Merging across GPUs keeps the same number again, and the lowest of those scores becomes the cutoff, sent back to every GPU just once. After that only molecules above the cutoff are written into fixed-capacity buffers, and if a batch of pockets overflows, that whole batch is rescanned exactly. On four Blackwell GPUs, retrieving 500,000 candidates for each of more than twenty thousand pockets takes about 61 seconds once the library is in memory, with mean recall of 99.6% of exact search.

3.35

3. llm-d 1.0 hardens open-source AI inference for production

IBM

Whether self-hosting open models for agents can cost less than buying a managed service comes down to cache reuse, because agent requests share large amounts of the same context. llm-d is an open-source inference stack built jointly by Google, IBM and Red Hat, and its stable release just shipped. It sends requests carrying the same context to the server that already holds that context, and lets the cache move between GPU, CPU and storage. IBM and Red Hat ran 3,000 coding agents at once at 5 to 10 times lower cost than a managed service. In another test, 97% of the reusable cache was actually reused.

3.35

4. Amazon targets Trainium gains with new TorchNeuron developer tool

Amazon

When moving to a new AI chip, the biggest cost is often not the hardware but rewriting and revalidating code. Amazon's new tool TorchNeuron lets existing PyTorch code run on its own Trainium chips by changing a single line of configuration. Amazon ran the same training script on conventional hardware and on Trainium, and the learning process behaved the same on both, showing that switching chips did not change the training result. The voice company Cartesia says its inference pipeline ran with almost no code changes; it only hand-wrote kernels for a few key layers that determine latency. Amazon is also contributing the related changes back to the official PyTorch codebase.

3.30

5. FastLLM: a C++ LLM inference engine

Hysen Labs

For running a large MoE model without enough GPU memory, the C++ inference engine FastLLM lets weights be split layer by layer across GPU memory, CPU memory and even disk, but there is no benchmark data yet to check the speeds it advertises. The repository says it can run an unpruned DeepSeek model, without naming the version, on any GPU with more than 10 GB, and on a dual-socket server generate 20 tokens per second at full precision with one request at a time. It does not depend on PyTorch, reads weights straight from Hugging Face, and offers OpenAI- and Anthropic-compatible APIs. The review notes that the repository has no releases and no benchmark file to check these figures against. Its niche is narrow: machines whose RAM or SSD is much larger than GPU memory and that need to run MoE models.

3.20

6. Choosing a Vector Retrieval Database: The Built-in Vector Engine of Tair (Redis Enterprise Edition)

Alibaba

An Alibaba Cloud product-selection article argues that a RAG application already using Redis as a cache does not need a separate Milvus or Qdrant deployment and can use TairVector, the vector engine built into Tair, Alibaba Cloud's enterprise edition of Redis. It supports two indexes, the HNSW graph index, which is fast but approximate, and FLAT, which compares every vector for exact results. It combines vector search, BM25 full-text search and tag filtering in one query, keeps all data in memory and updates indexes in real time. The article says a single node reaches 510 thousand QPS on a performance-enhanced instance. A case study in the article says an e-commerce search team cut vector search latency from 15 ms to 3 ms after switching to Tair. The article also concedes that pure vector search beyond a billion vectors is still better served by a dedicated vector database. Teams already on Redis can drop a separate vector store, but very large data still calls for a dedicated one.

3.10

7. Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off

Should LLM inference use pipeline parallelism or tensor parallelism? This paper's conclusion is that it depends on how tight the first-token deadline is, not on throughput alone. The authors measured various parallel configurations on H200 and multiplied throughput by the share of requests that get their first token on time, giving the throughput that actually counts. When the deadline is very tight (5 to 10 seconds), pipeline parallelism, which assigns layers to different GPUs in order, is best; as the deadline loosens, the advantage shifts to tensor parallelism, which splits each layer across several GPUs that compute together. The longer the input, the more pipeline-heavy configurations come out ahead.

3.05

8. Introducing Superfluid: the local-AI native LLM server for multi-agent inference

BaseCompute

When several agents run on one local machine, the request a person is waiting for in the chat box often queues behind batch work. Superfluid lets interactive requests jump ahead at the next scheduler tick, and in the launch blog's demo it answered about 30 times faster. In the demo, eight agents on one Mac are working through long documents when the user asks a question. Superfluid answers in 0.36 seconds, while llama-server with the same model takes more than 10 seconds. Servers like llama-server are built for throughput, so a new chat request waits behind running batches; Superfluid lets the agents' tasks resume afterwards. Requests that share a system prompt also get only one prefill. These figures come from a demo in the launch blog, not an independent benchmark. When several local agents share a machine, scheduling should put the request a person is waiting for first.