INFRA · 2026-10-20
Infra — Paper Daily, 2026-10-20
Community search over short UGC posts, questions and comments, with relevance, CTR and model-serving cost as priorities.
4.35
Relaxed speculative verification can inflate completions enough to erase apparent decoding gains. Across matched prompts, the study tracks completion length, verifier rounds, runtime and accuracy. At the most relaxed settings, accepted tokens per round rise 1.34× while completions grow 1.54×.
Key insight: Acceptance length is an incomplete speed metric when verification changes the output distribution. Track request completion time, output length and answer accuracy together; longer reasoning can consume the budget needed to return an answer.
4.10
Reducing experts during prompt processing saves compute, but a few prompt positions account for much of the quality loss. Token-level interventions favor preserving full expert computation near task interfaces over random document positions. Time to first token falls about 14% on long prompts.
Key insight: Allocate expert compute by prompt position rather than uniformly. Instructions and answer boundaries deserve attention, but the sensitive positions also occurred at prompt ends, so these experiments do not distinguish interface-aware rules from simply preserving both ends.
4.10
DLite removes explicit historical context from the speculative drafter by reusing contextual hidden states from the target model. A lightweight sequential head restores dependencies within each proposed block. The paper reports improvements of up to 70% over DSpark.
Key insight: The drafter need not maintain its own copy of the entire conversation history. Reusing target-model states shifts its job toward local continuation, reducing duplicated state and cache maintenance rather than merely shrinking the draft model.
4.10
Accelerated Dataflow coordinates autoscaling across heterogeneous worker pools with prefix-aware batching for streaming LLM pipelines. It addresses inference-heavy workloads that conventional streaming engines handle poorly. Reported throughput improves 1.5–2× at similar latency.
Key insight: Batch formation and autoscaling are coupled decisions. Adding accelerators alone can leave upstream supply or queueing bottlenecks intact. For continuous UGC processing, shared-prefix reuse and stage-level queueing provide useful scheduling signals.
4.10
This study transfers group-wise calibration gains between representations of the same pretrained model without optimizing the recipient. Quantized integer codes and storage layout stay fixed. On the tested Qwen3 pair, transfer retains 76.9–90.5% of recipient-specific text-loss gains.
Key insight: The reusable object includes the assignment of gains to groups, not just their numerical distribution. Shuffling assignments hurts both recipients. Preserving that mapping may reduce repeated calibration work across precision variants.
3.95
GoogleAlloyDB combines vector and full-text retrieval through a native SQL interface with reciprocal rank fusion, avoiding application-side score normalization and joins. It also supports additional text indexes and external search sources. The example uses one declarative SQL call; this is an integration result, not a throughput multiplier.
Key insight: Rank-based fusion avoids calibrating incompatible scores. Combine exact keyword matches with semantic retrieval for short-text search, while accounting for the storage and build costs of position-aware text indexes.
3.75
HydraLLM converts a pretrained checkpoint into nested subnetworks whose embedding, attention, feed-forward and output widths follow the active head count. Weight reordering and distillation train the elastic family. At one-quarter of the heads, token-generation throughput rises 75.3% over full width.
Key insight: Width can become a deployment resource setting when coupled layers shrink together. This flexibility requires conversion and subnetwork training; disabling heads in an ordinary checkpoint does not reproduce the trained elastic architecture.
3.70
SP²ec adapts self-speculative draft configurations during generation rather than fixing one for every multimodal prompt. Its bandit algorithm exploits an observed single-peaked throughput curve and accounts for exploration time. Tested visual and text workloads improve by up to 11% over the underlying methods.
Key insight: Choose draft settings within each request while charging exploration against actual elapsed time. The single-peak assumption narrows the search, making adaptation useful when multimodal inputs have different compute needs.
3.70
This work reorders statements in the compiled program rather than recompiling a reordered graph, which can miss its predicted memory peak. Read/write modeling supports exact accounting under stated allocation conditions. Lower bounds certify optimal schedules for 94% of held-out wrapper graphs.
Key insight: Optimize the program that actually executes. A better high-level order can change during compilation. An optimality certificate is also actionable: it identifies cases where further scheduling search should give way to recomputation or architectural changes.
3.70
CascadeLedger audits whether a budgeted LLM cascade used only information legally available at each decision. It records calls, cache identities and budget transitions, then applies counterfactual checks to frozen traces. The reported detection rate for serialized faults is 99.83%.
Key insight: Final answers and aggregate call counts cannot establish valid routing. Preserve what was observable at each decision and freeze the trace before attaching task scores, so execution validity can be audited separately from answer quality.
EMNLP 2026 Workshop