INFRA · 2026-11-11

Infra — Paper Daily, 2026-11-11

Today's thread in this track is recalculating inference costs around the real shape of each workload. RecoGEM finds that the cost of generative recommendation concentrates in short sequences over large candidate sets, which calls for a dedicated serving system, while Meta's TLX lets ML engineers write kernels close to the hardware themselves. PACE shows that a long-session agent has to keep tokens close to the query as well as keep them at all. Firepool and NVIDIA Snapshot both shorten the wait for GPU startup, AdaOffload and Hibana schedule expert weights on memory-constrained machines, and Detent stops other requests in a batch from changing a request's output bits.

4.90

1. RecoGEM: A Serving System for Production-Scale Generative Recommendation

Generative recommendation has the model generate an item's SID token by token like an LLM, so LLM serving stacks look reusable, yet they run poorly because they assume every step has the same shape and attention cost sits in long sequences. This Kuaishou paper on RecoGEM spells out the difference: each candidate reads only a few SID tokens, but beam expansion creates a huge number of independent short sequences, so the fixed overhead of long-sequence kernels cannot be spread out.

4.50

2. Firepool: Brokering Warm GPU Capacity for Agentic and RL Workloads at LinkedIn

Agents and RL training issue thousands of GPU jobs that run for seconds, yet starting a Pod takes twenty to thirty seconds even with the image cached, so waiting outweighs the work. Common fixes each miss something: caching images does not remove scheduling and initialization, a persistent Ray cluster holds GPUs idle between steps, and a warm pool of ready Pods often goes unused because the next job needs a different environment.

4.30

3. How We Built TLX: A Hardware-Native GPU Compiler for Production ML at Scale

Meta

The performance of recent GPUs depends more and more on how groups of threads divide the work, for example one group moving data while another multiplies matrices. CUDA can express this division but needs large amounts of architecture-specific orchestration code. Triton is quick to write but hides the division inside the compiler, so with each hardware generation engineers often wait for the compiler. Meta added a layer called TLX on top of Triton, so engineers keep writing Python and spell out thread-group roles and synchronization only where it matters. For recommendation engineers, the most concrete example is user features at scoring time: the same features are typically copied dozens of times to match the candidates, wasting memory bandwidth and compute with a cost that grows linearly in the number of candidates. Meta moved that copy inside the kernel and changed the model and inference runtime along with it, cutting compute-intensive net latency by up to two thirds. Kernel optimization is becoming a routine tool for ML engineers, and the more candidates a model scores, the earlier this kind of change deserves a look.

4.15

5. Multi-GPU MoE Inference with Adaptive Expert Offloading

Each token in an MoE model activates only a few experts, so most expert weights can live in CPU memory while the GPU caches a subset, and a missed expert either has its weights fetched to the GPU or runs on the CPU. This is called expert offloading, and existing systems mostly target a single GPU or large offline batches. A fixed division of work between CPU and GPU can even make throughput fall as GPUs are added. AdaOffload shards each expert evenly across all GPUs, shards separately on the CPU according to NUMA layout, and combines offline hardware profiles with expert loads observed online to decide in real time what share of missed experts to fetch to the GPU. Across three open-weight models and five platforms, average decode throughput on reasoning workloads is 1.9 to 15.5 times that of the strongest baseline. The best division of work between CPU and GPU changes with hardware and load, and a fixed policy cannot keep up. Search and recommendation teams hosting their own models can rent fewer GPUs when memory is short.

4.15

6. PACE: Keeping the Right Tokens in the KV Cache Is Not Enough for Long-Horizon LLM Agents

The KV cache of a long-session agent does not fit on one GPU, so the system keeps only a small share of tokens and evicts the rest. Eviction methods compete on what to keep, assuming that keeping the needed tokens is enough. This paper finds it is not: in long coding sessions, with almost all needed tokens kept, the pass rate still falls from 93% to 64% as definitions move further from where they are used. The tokens are still there, but their rotary-position distance to the query is so large that the model uses them poorly. PACE has the agent's planner tell the cache what later steps will read, then renumbers the kept tokens so they always stay close to the query. In a pre-registered evaluation it passes 97% of hidden unit tests, and each of its two parts alone falls far short. This result uses the true dependency graph and synthetic coding sessions. Besides keeping tokens, a KV cache has to keep them close to the query. A search or recommendation agent with a task plan can hand it to the cache.

4.15

7. Hibana: Fast Agentic Inference on Unified-Memory Devices

Unified-memory devices such as the DGX Spark let the CPU and GPU share one memory and can hold large models. Memory bandwidth is limited, so decode is often stuck reading weights, and agent session state competes with those weights for memory. vLLM and SGLang mainly target datacenter GPUs, while local engines underuse this memory layout. Hibana changes three things in response: requests in a batch share one set of experts to cut weight reads, inactive session state moves to SSD, and speculative decoding generates several tokens per step with the length adjusted from acceptance rates. Replaying recorded agent sessions on one DGX Spark, it completes up to 3.4 times as many tasks per hour as other frameworks, and peak decode throughput at each system's best concurrency reaches up to 3.9 times. The fast profile is lossy, and three-bit weights and relaxed verification cost a few points of accuracy on some tasks. Local inference has to save the memory traffic behind each accepted token. When running several agents on a device, choose an engine by concurrency and cache hits together.

4.15

8. MACaQ: Fast, Model Agnostic Calibrated Quantization for Low-Precision Weights and Activations

Compressing both weights and activations to 4 bits saves memory and compute, and a common trick is a basis transform that makes inputs and weights easier to quantize. Inputs need the same transform at inference, and prior work absorbs it into earlier layers to avoid that cost, which couples layers together, requires new adaptation for each architecture and slows quantization. MACaQ goes the other way and keeps each layer's scaling and small-block rotation online at inference time, so every layer is quantized independently and only one layer needs to be in memory at a time. The online cost is small, adding only 2.3% end to end on Qwen3-32B. A greedy refinement step then moves individual FP4 codes to adjacent values, giving lower perplexity than DuQuant++ and MR-GPTQ, and it matches their best quality with 11 times less quantization time. The online cost gets diluted as layers grow, which makes it a good price for decoupling the layers. Teams deploying new architectures no longer need to adapt quantization for each one.

3.65

9. Meituan LongCat-2.5 Officially Open-Sourced: From Seeing to Arriving

Meituan

Long-sequence training and fine-tuning of a trillion-parameter MoE model on domestic Chinese accelerators runs into a practical limit of under 80GB of memory per card, with tight bandwidth too, so architecture and parallelism have to be traded off repeatedly. Meituan's open-source LongCat-2.5 has 1.6 trillion parameters in total, and this post explains how the team coped. Training uses FSDP double sharding and flattened buffers to lower memory use, keeps parameter, expert token and KV exchanges inside the high-bandwidth interconnect of a supernode where possible, and overlaps communication with computation. Inference quantizes expert weights and activations together to 4 bits, and one INT4 representation runs through expert communication and computation, which cuts data movement and conversion. On long-sequence SFT tasks, this FSDP configuration raises per-card throughput by 86% over a DDP baseline. Memory and bandwidth limits are fixed, so only architecture, operators and parallel strategy can change. Search and recommendation teams training large models on domestic accelerators can take this configuration as a starting point.

3.65

10. Start and Restore GPU Workloads in Seconds with NVIDIA Snapshot

NVIDIA

When inference traffic spikes or a replica fails, idle GPUs still cannot take the work, because a cold start often takes minutes and holds up both scaling and recovery. NVIDIA's open-source Snapshot saves a fully initialized workload together with its CPU and GPU memory as a checkpoint, and restores it on a node with the same GPU model and driver, skipping weight loading, compilation and graph capture. On vLLM, SGLang and TensorRT LLM, restoring is 8 to 36 times faster than a cold start, every configuration comes back within 43 seconds, and for large MoE models the slowest stage is writing GPU state back to GPU memory. In every tested configuration, one checkpoint plus one restore takes less time than a single cold start. These times exclude pulling the image and setting up the container runtime. The first restored replica already pays back the cost of the checkpoint. Teams serving large models for search or recommendation can cut the scaling wait from minutes to under a minute.