INFRA · 2026-10-21
Infra — Paper Daily, 2026-10-21
Community search over short user-generated posts, questions and comments. Key metrics: relevance (nDCG) and click-through rate. Focus: semantic matching, query understanding, retrieval, ranking and LLM applications.
4.50
RemoteANN co-designs vector indexing and query execution for cloud object storage. Quantized inverted lists, reordered data, merged range reads and local SSD caching reduce remote I/O. At matched target recall, the authors report up to 5.9× higher throughput than representative baselines.
Key insight: The useful optimization target is remote requests per query. Data locality makes request merging and caching effective; faster distance computation alone cannot remove object-store latency.
4.45
TokenGate trains an existing model to skip less useful tokens in later layers. Token-gated attention supplies a learning signal for token selection. Across the tested tasks, the authors report up to 11.3× faster time to first token without measured accuracy loss.
Key insight: Token selection can be learned during lightweight adaptation instead of imposed as a fixed heuristic. This targets prefill latency and memory, with an adaptation step required before deployment.
4.20
KVCompass adds a discovery layer for reusable prompt-prefix caches across regions. It separates cache identity from placement and routes requests using reuse, load and network distance. Simulations and a three-continent testbed show up to a 14× reduction in P99 time to first token.
Key insight: A prefix-cache directory must track changing holders rather than assume fixed placement. Cross-region routing also needs distance and load; a remote cache hit is not automatically cheaper than local recomputation.
4.20
MOSAIC searches per-layer KV-cache budgets instead of allocating the same capacity everywhere. A single search maps the quality–memory tradeoff and transfers its allocation across total budgets. On the tested models, it matches uniform-allocation quality with up to 75% less cache.
Key insight: Budget allocation is a separate design choice from token eviction. The learned plans favor a few middle layers and can sit on top of existing eviction methods.
4.20
GoogleGoogle introduces native PyTorch TPU backends for vLLM and SGLang. The engines retain framework schedulers and APIs while replacing lower-level kernels and cache transfers. Google reports more than 700 tokens/s for a single user with Kimi K3 on Ironwood in a specific configuration.
Key insight: A platform plugin can preserve the serving control plane while specializing kernels and cache movement. The reported single-user speed is configuration-specific; concurrency and request lengths remain essential comparison inputs.
4.10
SigmaTrace records individual RDMA operations below communication libraries and exposes GPU-initiated transfers. In the authors’ 128-GPU NCCL AllReduce test, host-side tracing adds 1.3% overhead.
Key insight: Observability below the library layer can serve multiple frameworks. Per-operation traces distinguish traffic skew from synchronization delays; the quoted overhead applies specifically to host-side tracing.
4.10
PACT accelerates frozen projections during LoRA training with native NVFP4 matrix operations and stores selected backward activations at low precision. Sensitive nonlinear operations keep higher-precision inputs. Evaluation spans multiple models and tasks on a 16GB RTX 5080; the abstract reports improvements without a single aggregate speedup figure.
Key insight: LoRA reduces trainable state but does not eliminate frozen-weight compute or saved activations. PACT treats computation precision and activation-storage precision as separate choices.
3.85
This study tests when better draft acceptance actually improves speculative-decoding throughput. It retrains six drafter recipes on shared data and evaluates them across ten benchmarks with matched serving conditions. The preferred drafter and budget depend on task uncertainty and execution cost.
Key insight: Accepted tokens per round is an intermediate metric. Compare useful tokens against round latency, and hold training data and inference settings constant before attributing gains to a new drafter.
3.85
Google Google’s TorchTitan TPU integration brings collectives and full training steps into the compiler graph, then applies kernel and memory optimizations. In a weak-scaling experiment with fixed per-chip batch size, per-chip throughput drops about 8% as chip count grows 128-fold.
Key insight: Communication becomes easier to overlap when it is visible inside the compiled graph. The scale result uses a growing global batch, so it measures framework scalability rather than fixed-batch training efficiency.
3.40
Google Google previews a managed RAG search stack that runs entirely inside an air-gapped GDC deployment. Local models, embeddings, PostgreSQL and object storage handle documents and persistent conversations. The stated per-file limit is 100MB; the announcement provides no comparative retrieval-quality benchmark.
Key insight: Air-gapped deployment requires identity, ingestion, retrieval and conversation state to fit inside the same boundary. This is an architectural and capacity announcement, with limited comparative performance evidence.