2026-11-02 ~ 11-08 · Infra
Generative · Personalization · AI Search · Infra · Embodied AI
- 1. TableTune: Measured, Self-Correcting Draft-Length Tables for Dynamic Speculative Decoding · 📖 Deep dive
Speculative decoding only pays off while the GPU has spare compute: the more requests share a batch, the fewer tokens the small drafter should guess for the big model each step…
Appeared in: 11-02
- 2. FlatCake: A Flat KVCache Storage Architecture for LLM Serving · 📖 Deep dive
LLM serving usually treats host memory as a KV cache layer in front of the SSD, but after analyzing eight production request traces the authors find that this layer barely filters any reads…
Appeared in: 11-02
- 3. ConfluxServe: Reconciling Resource Utilization and Parallelism Efficiency across Prefill and Decode in LLM Serving · 📖 Deep dive
In LLM serving, running prefill and decode together lets their resource needs complement each other, since prefill uses compute and decode uses memory bandwidth. But the two want different parallelism…
Appeared in: 11-02
- 4. Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI
Open models built for agents often have to choose between capability and inference cost. NVIDIA's Nemotron 3.5 Super optimizes both. It is a hybrid Mamba-Transformer mixture-of-experts model that activates only about a…
Appeared in: 11-02
- 5. Rimuru: Unified Memory Management for Modern LLM Serving via Elastic Expert Offloading
To run a large MoE model on a machine with limited GPU memory, the usual approach keeps some expert weights in host memory and moves them onto the GPU when needed. Even so…
Appeared in: 11-02
- 6. SGLang security update: improving request isolation
In one of Cloudflare's production model deployments, a request received content from another user's session. The SGLang team traced it to the path where prefill and decode run on separate workers…
Appeared in: 11-02
- 7. The Endgame of FSDP Communication: Layout-Aware Collectives on Copy Engines, RDMA, and NVSwitch
In FSDP2 training, parameters and gradients are stored one tensor per parameter, but communication wants one flat buffer ordered by GPU, so every step copies data back and forth: gathered results are copied into…
Appeared in: 11-02
- 8. A Unified Hessian-Guided Framework for Post-Training Quantization
Post-training quantization has to settle three things: each layer's value range, what transform to apply to weights and activations before quantizing, and which way to round each weight…
Appeared in: 11-02
- 9. DualQuant: Dual-scale Calibration-free Weight-and-Activation Quantization for LLMs
To get the full speedup of new GPUs with native low-bit math, both weights and activations must be quantized to 4 bits. A few activation channels carry very large values, and the usual fix collects calibration data…
Appeared in: 11-02
- 10. DWISE: Denoising Workload-Informed Scheduling Engine
Diffusion language models denoise many tokens in parallel each round and promise much faster generation than producing one token at a time, but serving frameworks copy the scheduling of autoregressive models: first…
Appeared in: 11-02