INFRA · 2026-11-11
Infra — Paper Daily, 2026-11-11
Today's thread in this track is recalculating inference costs around the real shape of each workload. RecoGEM finds that the cost of generative recommendation concentrates in short sequences over large candidate sets, which calls for a dedicated serving system, while Meta's TLX lets ML engineers write kernels close to the hardware themselves. PACE shows that a long-session agent has to keep tokens close to the query as well as keep them at all. Firepool and NVIDIA Snapshot both shorten the wait for GPU startup, AdaOffload and Hibana schedule expert weights on memory-constrained machines, and Detent stops other requests in a batch from changing a request's output bits.
4.90
Generative recommendation has the model generate an item's SID token by token like an LLM, so LLM serving stacks look reusable, yet they run poorly because they assume every step has the same shape and attention cost sits in long sequences. This Kuaishou paper on RecoGEM spells out the difference: each candidate reads only a few SID tokens, but beam expansion creates a huge number of independent short sequences, so the fixed overhead of long-sequence kernels cannot be spread out.
4.50
Agents and RL training issue thousands of GPU jobs that run for seconds, yet starting a Pod takes twenty to thirty seconds even with the image cached, so waiting outweighs the work. Common fixes each miss something: caching images does not remove scheduling and initialization, a persistent Ray cluster holds GPUs idle between steps, and a warm pool of ready Pods often goes unused because the next job needs a different environment.
4.20
The same request can produce different output bits just because the other requests in its batch changed, so evaluations cannot be reproduced and an RL trainer recomputing a rollout cannot match it. The engines' own deterministic modes keep one conservative kernel per operator, with latency reaching 10.3 times the standard settings.
4.15
Each token in an MoE model activates only a few experts, so most expert weights can live in CPU memory while the GPU caches a subset, and a missed expert either has its weights fetched to the GPU or runs on the CPU. This is called expert offloading, and existing systems mostly target a single GPU or large offline batches. A fixed division of work between CPU and GPU can even make throughput fall as GPUs are added. AdaOffload shards each expert evenly across all GPUs, shards separately on the CPU according to NUMA layout, and combines offline hardware profiles with expert loads observed online to decide in real time what share of missed experts to fetch to the GPU. Across three open-weight models and five platforms, average decode throughput on reasoning workloads is 1.9 to 15.5 times that of the strongest baseline. The best division of work between CPU and GPU changes with hardware and load, and a fixed policy cannot keep up. Search and recommendation teams hosting their own models can rent fewer GPUs when memory is short.
4.15
The KV cache of a long-session agent does not fit on one GPU, so the system keeps only a small share of tokens and evicts the rest. Eviction methods compete on what to keep, assuming that keeping the needed tokens is enough. This paper finds it is not: in long coding sessions, with almost all needed tokens kept, the pass rate still falls from 93% to 64% as definitions move further from where they are used. The tokens are still there, but their rotary-position distance to the query is so large that the model uses them poorly. PACE has the agent's planner tell the cache what later steps will read, then renumbers the kept tokens so they always stay close to the query. In a pre-registered evaluation it passes 97% of hidden unit tests, and each of its two parts alone falls far short. This result uses the true dependency graph and synthetic coding sessions. Besides keeping tokens, a KV cache has to keep them close to the query. A search or recommendation agent with a task plan can hand it to the cache.
4.15
Unified-memory devices such as the DGX Spark let the CPU and GPU share one memory and can hold large models. Memory bandwidth is limited, so decode is often stuck reading weights, and agent session state competes with those weights for memory. vLLM and SGLang mainly target datacenter GPUs, while local engines underuse this memory layout. Hibana changes three things in response: requests in a batch share one set of experts to cut weight reads, inactive session state moves to SSD, and speculative decoding generates several tokens per step with the length adjusted from acceptance rates. Replaying recorded agent sessions on one DGX Spark, it completes up to 3.4 times as many tasks per hour as other frameworks, and peak decode throughput at each system's best concurrency reaches up to 3.9 times. The fast profile is lossy, and three-bit weights and relaxed verification cost a few points of accuracy on some tasks. Local inference has to save the memory traffic behind each accepted token. When running several agents on a device, choose an engine by concurrency and cache hits together.
4.15
Compressing both weights and activations to 4 bits saves memory and compute, and a common trick is a basis transform that makes inputs and weights easier to quantize. Inputs need the same transform at inference, and prior work absorbs it into earlier layers to avoid that cost, which couples layers together, requires new adaptation for each architecture and slows quantization. MACaQ goes the other way and keeps each layer's scaling and small-block rotation online at inference time, so every layer is quantized independently and only one layer needs to be in memory at a time. The online cost is small, adding only 2.3% end to end on Qwen3-32B. A greedy refinement step then moves individual FP4 codes to adjacent values, giving lower perplexity than DuQuant++ and MR-GPTQ, and it matches their best quality with 11 times less quantization time. The online cost gets diluted as layers grow, which makes it a good price for decoupling the layers. Teams deploying new architectures no longer need to adapt quantization for each one.
3.65
NVIDIAWhen inference traffic spikes or a replica fails, idle GPUs still cannot take the work, because a cold start often takes minutes and holds up both scaling and recovery. NVIDIA's open-source Snapshot saves a fully initialized workload together with its CPU and GPU memory as a checkpoint, and restores it on a node with the same GPU model and driver, skipping weight loading, compilation and graph capture. On vLLM, SGLang and TensorRT LLM, restoring is 8 to 36 times faster than a cold start, every configuration comes back within 43 seconds, and for large MoE models the slowest stage is writing GPU state back to GPU memory. In every tested configuration, one checkpoint plus one restore takes less time than a single cold start. These times exclude pulling the image and setting up the container runtime. The first restored replica already pays back the cost of the checkpoint. Teams serving large models for search or recommendation can cut the scaling wait from minutes to under a minute.