Detent: Batch-Invariant LLM Serving and RL Rollouts at Near-Stock Speed

Peixu Xin, Runchuan Feng, Jiazhen Guo, Kan Wu, Ming Yin
62 minread4.10scoreNo. 411-11 Infra issue
Contents · 9 sections1Why not just pick one fast kernel and fix it?2Request A's route through Detent3Kernel-choosing code rewritten as a small analyzable program4A minimum cut finds the few values to pin5Filter legal values first, then tune family by family6Bits stay stable, and latency is about a tenth above Stock7One parameter list suits only one kind of traffic8RL trainer and rollouts align bit for bit9When can request A's answer still change?
Rel 4Nov 4Prac 4Evid 4Time 5

Making an LLM service compute exactly the same result for a request every time, no matter which other requests it is batched with, does not require giving up fast kernels (the programs that actually run the computation on the GPU). Detent replaces the batch information the inference engine reads when choosing kernels (such as the batch's total token count) with a few constants fixed before deployment, and median latency (p50) is only about a tenth higher than the engine's default configuration.

Inference engines such as vLLM and SGLang group requests that arrive together into a batch and send it to the GPU. Under the engine's default configuration (the paper calls it Stock), when the same request is batched with different companions, the computed values may differ at the level of binary digits (bits), and the generated text may change with them. Here is a scenario used throughout the article (illustrative): request A is a grade-school word problem; in the morning off-peak it runs almost alone, and at the evening peak it is batched with 31 other requests. The model and the prompt are unchanged and decoding is greedy, yet A's output can still differ. The paper measured this: keeping the model and questions fixed and changing only how many requests are sent at once, Stock SGLang running Qwen3-30B-A3B changed the answers on 33 of GSM8K's 1,319 questions.

This kind of change hits three kinds of work. People running evaluations rerun them and get scores that do not match. A third party who wants to recompute a provider's output to verify it also gets a mismatch. People doing reinforcement learning (RL) have it worse: the inference engine first generates a batch of answers (rollouts), and the trainer then recomputes the log probability (logprob) of each token in them. If the two sides differ in bits, the probabilities used in training are no longer the probabilities used when sampling, and the mathematical premise of training no longer holds.

Why does the same request compute different bits? The root is floating-point addition. Floating-point addition is not associative: for the same few numbers, the order of addition affects rounding. Illustration: suppose only 3 significant digits are kept and we compute 1000 + 4 + 4. Adding left to right, 1000 + 4 = 1004, which rounds back to 1000, and adding 4 again still gives 1000. Computing 4 + 4 = 8 first and then 1000 + 8 = 1008 rounds to 1010. Mathematically it is the same sum, but the results differ by 10, which shows that the order of addition alone can change the result.

Applied to request A, suppose attention has to sum over A's 8 blocks of KV. In the morning the engine uses FlashAttention-3 (FA3) to split these 8 blocks into 4 parts (0–1, 2–3, 4–5, 6–7), computes each part and then merges them. At the evening peak A lands in a busy batch; FA3 sets its split count from the batch, this time choosing 1 part, so the 8 blocks are processed in sequence (the paper's illustration). Once the order of addition changes, A's bits may change. The split count is only one of the ways batch information reaches A: a MoE (mixture-of-experts) layer takes the row of a tuned configuration table closest to the batch's token count to decide how to tile its matrices, and multi-GPU summation (all-reduce) picks its summation algorithm by the number of bytes the batch has to send.

The paper first discusses two existing approaches: making each kernel invariant inside, and the engine's built-in deterministic mode. Applied to request A, the first does not stop the change and the second is too slow. The first is batch-invariant kernels (the approach of Thinking Machines and other teams), which make the order of addition inside each kernel independent of the batch. But the engine still chooses kernels and parameters from batch information (the batch's token count, the bytes to send). In the morning it picks "split into 4", in the evening "split into 1". Each path has a fixed order of addition inside, but the two paths' orders differ from each other, so A's result can still differ.

The second is the engine's built-in deterministic mode (SGLang-Det, vLLM-Det): each operator keeps only one most conservative kernel, attention is not split, all-reduce always uses NCCL tree, and MoE uses one hard-coded configuration. A now always takes the same path, but every request is also forced onto this slow path, with the fast kernels and splitting unavailable. On the paper's test workloads, the latency per output token of SGLang-Det is 1.59–6.34 times Stock's, and of vLLM-Det 2.83–10.3 times.

Why not just pick one fast kernel and fix it?

The most direct idea is to keep FA3's splitting and the tuned MoE configurations, and simply confirm that these fast kernels produce identical bits, so the engine can pick any of them. The trouble is that whether two kernels produce identical bits cannot be seen from the source code; it can only be measured. The paper gives two examples with opposite outcomes, showing that you cannot guess without measuring. With head dimension 64, two tile widths of FA3 (an internal blocking parameter of the same kernel) gave different bits on 31 of 32 logprobs. On Qwen3-30B-A3B, the 18 tuned MoE configurations the authors measured produced bits exactly identical to the one SGLang-Det hard-codes.

The amount of measurement is also too large to finish: FA3 alone has 18,472 configurations at head dimension 128 in BF16, and measuring all of them would occupy a four-GPU node for about 4.8 days.

Detent answers a different question instead. The authors observe that many fast kernels, once chosen and with their parameters set, order their additions based only on the request itself and not on who else is in the batch; that is, they are already batch-invariant. Then batch information has only one way left to affect a request's bits: the engine's step of "which kernel, with which parameters". So there is no need to answer "which kernels agree with each other"; it is enough to guarantee that the same request gets the same choice every time. For A, Detent does not compare whether "split into 4" and "split into 1" compute the same thing; it only guarantees that A takes the same split in the morning and in the evening.

The paper writes this idea as a sufficient condition, where BI (batch invariance: results that do not change with the other requests in the batch) is the property, ∧ means "and", and ⟹ means "implies":

Kernel BI means that each candidate kernel, once its parameters are set, orders its additions based only on the request itself; the user-supplied kernels guarantee this. Selection BI means the choice depends only on the deployment, the model, the parameters fixed in advance and the request itself; Detent is responsible for this. For the two together to imply end-to-end BI there is one more premise: mathematically, each request's output is determined only by its own inputs, so for example there can be no computation that lets requests in a batch see each other.

Request A's route through Detent

Two names are used throughout. A pin is a value in the selection program that Detent replaces with a constant, such as FA3's split count. The parameter list (the paper calls it a commitment tuple) assigns each pin a legal value and each deployment switch a setting. Deployment switches are configuration items that do not read batch information but affect which kernel is used; examples are whether all-reduce uses push or pull and whether matrix multiplication (GEMM) adds padding. The list is fixed before deployment and never changes during serving.

Detent's work splits into a before-deployment part and a serving part:

StepInputOperationOutputWhen it runs
Write selection logicThe engine's original kernel-choosing codeRewrite it in SelIRAn executable selection programOnce per kernel call site
SimplifySelection program, quantities known at deploymentSubstitute known quantities and simplifyA dependency graphOnce per deployment and engine version
Find pinsDependency graphFind the fewest values that cut off batch informationPin listOnce per deployment and engine version
Filter legal valuesCandidate pin valuesRun the program on them and check preconditionsLegal values for each pinOnce per deployment and engine version
TuneLegal values, recorded workloads similar to productionThe tuner measures each kernel family (see below) in turnA parameter listOnce per workload type
ServeRequest, parameter listThe selection program runs with the constantsA fixed kernel choiceEvery kernel call

At the evening peak, A and 31 other requests land in the same batch. At the FA3 call site the engine runs the selection program. The program would have computed the split count from the batch's total KV length; now the split-count selection point takes the constant from the list directly and no longer changes with the total KV length. Illustration: A's 8 KV blocks are still split into 0–1, 2–3, 4–5 and 6–7, and each part adds in a fixed order. The FA3 used here is a version the authors rebuilt: it keeps KV splitting but splits by a fixed stride (split stride), so a request's split boundaries depend only on its own KV length and not on the batch. The MoE and all-reduce call sites work the same way, and all-reduce sums each element in a fixed GPU order. A's bits now depend only on the deployment, the model, the list and A itself.

Kernel-choosing code rewritten as a small analyzable program

Detent has to find where batch information flows in the selection logic, and also guarantee that those places really are replaced with constants in production. So each kernel call site runs a SelIR program directly, in place of the engine's original selection code. The solver (the component that finds pins) analyzes exactly the program the engine runs; when the solver pins a value in the program, the running engine reads that constant. If a separate copy of the logic were analyzed instead, production would still run the engine's original code, and pinned values would have no effect there.

SelIR is a subset of Python, and its programs are pure functions. The user marks inputs that read batch information as Runtime, and the values the program returns are called selection points, such as the split count or which kernel version to use. require and assume state the conditions under which a choice is valid, for example that a kernel supports only certain head dimensions; if a condition fails, the choice is not legal. Here is an illustration with simplified syntax and numbers:

# Illustrative: simplified FA3 selection logic
def select_fa3(head_dim, total_kv: Runtime, stride: knob):
    require(head_dim in (64, 128, 256))
    splits = 4 if total_kv < 4096 else 1   # batch information is read here
    return splits, stride                  # the return values are selection points

The user only marks inputs and does not have to say which value is a pin; that is left to the solver. In the morning the total KV is small and splits is 4; in the evening it is large and splits is 1. The opening problem, A splitting into 4 in the morning and 1 in the evening, is the line that reads total_kv in this program.

Most inputs are known before deployment, such as head dimension, data type, GPU architecture and parallelism. Detent first substitutes and simplifies these, then turns the program into a dependency graph: each node is a value in the program and each edge shows which values a value is computed from, so it becomes visible where batch information flows, including dependencies created by conditional branches. Illustration: the program has a branch if head_dim == 64, and this deployment's head dimension is 128, so the branch disappears together with the batch information it read.

The pins saved in this step are substantial. With constant folding alone (replacing expressions that can be computed directly with their results), FA3 needs 12 pinned values; after full simplification it needs only 3. Every extra pin multiplies the later search space by its number of candidates. Simplification results are cached: an FA3 cache hit takes about 20 ms and a cold solve about 5 s. Simplification happens only at deployment, and a new batch of requests during serving does not trigger it again.

A minimum cut finds the few values to pin

The goal is to keep batch information from reaching any selection point. The simplest way is to pin every selection point that reads batch information, which is sure to block it but costs too many pins.

Illustration: on a MoE dependency graph, the batch's token count first decides which row of the configuration table to use (this value is called the row key), and the row key then decides 4 selection points, each with 5 candidates. Pinning the selection points, the tuner (the component that picks the list) must search 5×5×5×5 = 625 combinations. Pinning only the row key cuts all 4 paths with one value, and only 5 options need searching.

The solver frames this as a minimum cut on the graph: find a set of values such that every path from a batch input to a selection point passes through at least one of them. It solves the minimum cut with a max-flow algorithm, with weights set to prefer fewer pins and, at equal pin counts, positions closer to the selection points. On Llama-3.1-70B, FA3's dependency graph has 8,076 values and 26 selection points, only 3 of which relate to batch information, and the cut falls on those 3 selection points. Qwen3-30B-A3B's MoE is different: the cut falls on the row key inside the program, and this one pin fixes 14 of the 16 batch-dependent selection points.

In the end Llama-3.1-70B pins 3, 3 and 0 values for attention, all-reduce and MoE (Llama-3.1-70B has no MoE layers), Qwen3-30B-A3B pins 3, 3 and 3, and vLLM and SGLang get the same pins. Without the minimum cut, pinning every value that the MoE's batch-dependent selection points read, Qwen3-30B-A3B would need 10 pins instead of 3. Without any analysis, FA3 would have 2^27 candidate combinations. This shows that the minimum cut shrinks the later tuning to a size that can finish, and that where the pins land is decided by the structure of the graph, which differs between FA3 and MoE.

Left: the full FA3 selection program on Llama-3.1-70B; right: the MoE selection program on Qwen3-30B-A3B. A few pins separate kernel selection from batch information.
Left: the full FA3 selection program on Llama-3.1-70B; right: the MoE selection program on Qwen3-30B-A3B. A few pins separate kernel selection from batch information.

Filter legal values first, then tune family by family

Candidate values cannot be filled in arbitrarily. Detent runs the program on a set of candidate values and checks that every require and assume holds; when the range of inputs the program may see (such as context length) is small it checks every value, and when it is large it checks a fixed set of sampled values. On Qwen3-30B-A3B, FA3's 7,168 candidate sets drop to 2,976 after filtering.

During serving the selection program also checks new inputs. Illustration: some value in the list requires the context to be at most 32k tokens, and an over-long request arrives in production; the engine stops with an error and does not quietly switch to a kernel outside the list.

Being legal is not enough; the list also has to be fast, and the fastest list depends on the workload's prefix hit rate, concurrency and prompt length. The number of combinations facing the tuner is daunting. This is where kernel families come in: a family is the set of kernels one SelIR program chooses among, for example all FA3 configurations form one family. On Qwen3-30B-A3B the four families FA3, MoE, all-reduce and GEMM have 2,976, 36, 3 and 5 legal values, so a joint search would try 1,607,040 combinations.

The tuner's approach is to assume that the families' times simply add up. Illustration: two families with 10 values each need 100 joint trials. If the total time equals the sum of the two families' own times, picking the fastest within each family is the fastest overall, and only 20 trials are needed.

This assumption relies on three conditions: kernels of different families run one after another without overlapping; the recorded workload fixes batching and outputs, so the list only changes which kernel each family uses; and the time of a call depends only on that call and its own family's value. If the engine fuses two families, the user must declare them as one family.

With additivity, Qwen3-30B-A3B needs only 2,976 + 36 + 3 + 5 = 3,020 candidates, and 76 were actually measured. On Qwen3-30B-A3B the authors measured that the largest interaction between families was 1.04% of the total GPU time per output token; from this bound it follows that the list picked family by family is within 2.1% of the truly optimal list. On Llama-3.1-70B's exhaustive results, the tuner measured only 8.0% of the candidates and was on average 0.04% from optimal, 0.75% at worst. This shows that the family-by-family search skips most of the measurement with almost no loss against the optimum.

The tuner's output is the list request A reads both in the morning and in the evening:

# Illustrative: the parameter list for one workload; only fields mentioned in the text, values made up
FA3:         split count = 4, split stride = 2 blocks
MoE:         row key = 4096
all-reduce:  workspace = pull
GEMM:        padding = on

Bits stay stable, and latency is about a tenth above Stock

First, are the bits really stable? The paper checks with 26 test cells. In each cell, at temperature 0, a request first runs in a reference batch and then in the same batch or a changed one, and the logprobs of the first 16 output tokens are compared bit for bit. "Changed batches" include replacing A's companions, moving A to another row of the batch, and changing the length of A's prefix hit. Logprobs are compared instead of tokens because a small change in the model's scores need not change the chosen token, and comparing only tokens would miss it.

Detent and both deterministic modes pass all 26 cells, and Detent also passes 9 stricter extra cells. Stock fails 9 cells (Llama-3.1-70B) and 5 cells (Qwen3-30B-A3B) of 26 on SGLang, and 19 and 2 of the 24 and 23 cells that apply on vLLM (the remaining cells do not apply on vLLM: its cache pages hold 16 tokens, so the tests' short prompts get no prefix hit in those cells). This shows the test suite really catches Stock's inconsistency, and Detent's guarantee holds on both engines.

Latency is compared on four H100s with ShareGPT multi-turn chat and arXiv long documents of about 8k tokens. Each workload is run at several load levels by requests sent per second, and the metric is the median latency per output token, p50. The table below gives the geometric mean of the ratios across load levels, with Detent using a list tuned for each level:

EngineDetent / Stock (Llama-3.1-70B)Detent / Stock (Qwen3-30B-A3B)Deterministic mode / Detent (range across both models)
SGLang1.10×1.11×2.4–2.6×
vLLM1.08×1.11×3.4–5.7×

This shows that keeping fast kernels recovers most of the deterministic modes' cost: with equally stable bits, Detent is 2.4–5.7 times faster than the deterministic modes and about a tenth slower than Stock. On individual workloads, Detent's p50 is 1.01–1.22 times Stock's, and the geometric mean of p99 is 1.12–1.22 times.

p50 latency of each engine and mode relative to Stock. Bars are geometric means over the load levels of one workload type, with Detent tuned per level; (a) is Qwen3-30B-A3B and (b) is Llama-3.1-70B.
p50 latency of each engine and mode relative to Stock. Bars are geometric means over the load levels of one workload type, with Detent tuned per level; (a) is Qwen3-30B-A3B and (b) is Llama-3.1-70B.

Part of the deterministic modes' gap comes from queueing. The top load level is set by Stock's maximum throughput; the deterministic modes process more slowly, and when they cannot keep up, requests pile up in the queue. The individual workloads where p50 reaches 6.1 and 9.2 times Detent's all come from such runs. Running each method at 0.67–1.0 times its own maximum throughput, where queues stop growing, vLLM-Det (Llama-3.1-70B on ShareGPT) and SGLang-Det (Qwen3-30B-A3B on arXiv) are still 2.0–2.3 times Detent. This shows that queueing magnifies the gap, but the gap itself remains without queueing.

How much does the tuner contribute to keeping latency only a tenth higher? The untuned default list sets each pin to "the legal value the program most often chooses on this workload when nothing is pinned". On SGLang the default list is up to 103% slower than the tuned one (Llama-3.1-70B), and 1–233% slower on Qwen3-30B-A3B. Both lists can choose from the same kernels, so tuning alone accounts for 1.34 times (Llama-3.1-70B) and 1.67 times (Qwen3-30B-A3B) of SGLang-Det's gap.

Pinning decides whether the bits are stable; the values pinned decide whether it is fast.

Detent's closeness to Stock is not because it swapped in faster kernels either. The paper put Detent's chosen GEMM and all-reduce kernels into Stock: if Detent had caught up by using faster kernels, Stock should get faster, but GPU time per output token instead rose by 0.6% (Qwen3-30B-A3B) and 2.8% (Llama-3.1-70B). The authors point to two sources of the remaining gap: the rebuilt FA3 adds 3 µs per call, and the best split stride is within 1.06 times of Stock's splitting; and one list has to serve both long prefills and small decode batches, while Stock can change its choice every batch.

Next, the answers. Detent and SGLang-Det both guarantee results that do not change with the batch, but they run different kernels, so their output bits can differ, and Detent's accuracy still has to be checked. The paper runs GSM8K's 1,319 questions on SGLang at three load levels, concurrency 8, concurrency 64 and all questions at once, with greedy decoding. Detent's accuracy minus SGLang-Det's is between −0.15 and +0.53 percentage points, with McNemar test p ≥ 0.14. This shows that on these questions the two cannot be told apart in accuracy.

Across the three load levels, Stock's text changed on 790 questions (Qwen3-30B-A3B) and 484 (Llama-3.1-70B), and its answers on 33 and 16. SGLang-Det's text never changed. Detent used different lists at concurrency 64 and all-at-once, and these two lists happen to agree on KV splitting, so the text was identical. The list for concurrency 8 splits KV, so the order of addition differs and the text may differ from the other two levels. So as far as load alone is concerned, only Stock lets answers change.

These results add a condition to A's stability: A's answer follows the list, not the load.

(a) GSM8K accuracy with 95% Wilson intervals, and on the right Detent minus SGLang-Det with its 95% interval. (b) Number of questions whose answer changed across the three load levels; for Detent only concurrency 64 and all-at-once are compared.
(a) GSM8K accuracy with 95% Wilson intervals, and on the right Detent minus SGLang-Det with its 95% interval. (b) Number of questions whose answer changed across the three load levels; for Detent only concurrency 64 and all-at-once are compared.

Another route does not fix kernels but verifies afterwards. LLM-42 decodes with fast kernels first, then replays the computation in a fixed order of addition to verify it, and rolls back and recomputes when they disagree; it can also verify only some requests, making only those requests' results deterministic. The authors ported LLM-42 onto their own SGLang baseline and compared on one GPU with Llama-3.1-8B and ShareGPT. With every request deterministic, Detent's median end-to-end latency is 3.5–5.1% above Stock's. With only 2% of requests deterministic, LLM-42's median latency is still 1.15–1.20 times Detent's.

Offline runs on arXiv long documents go the other way: with a deterministic share below 20%, LLM-42's throughput is 0.4–2.6% higher than Detent's. This shows that when only a few requests need to be deterministic and the work is offline long documents, verify-and-roll-back still has an edge. These numbers come from the authors' own port of LLM-42: the offline check they registered in advance failed by 44%, and after seeing the data the authors changed how the check is done.

One parameter list suits only one kind of traffic

Stock can change its choice every batch; a parameter list cannot. The authors' compromise is one list per workload type. When all load levels of one workload type (ShareGPT, arXiv long documents, arXiv closed-loop sending) share one list, it is only 2.7% (Qwen3-30B-A3B) and 1.1% (Llama-3.1-70B) behind per-level tuning on its own workload.

The wrong list is costly. On Qwen3-30B-A3B, running one load level of arXiv long documents with the ShareGPT list gives 4.17 times the latency of per-level tuning, and median time to first token rises from 238 ms to 4.75 s. The same cell on Llama-3.1-70B is 1.15 times. This shows that when the traffic type changes and the list does not, latency lands in cells like this. When switching to a newly tuned list, requests in the same batch must not be split between the old and new lists, or the batch would again affect the results.

RL trainer and rollouts align bit for bit

RL demands bit consistency even more strictly than evaluation. When generating a rollout, the engine prefills and then decodes, while the trainer recomputes everything in one prefill. Illustration: A is a rollout with a 100-token prompt and 50 generated tokens. During generation the engine prefills the 100 prompt tokens and then decodes the 50 generated tokens one at a time, while in training all 150 tokens are prefilled at once; the keys are grouped differently, so the logprob bits may differ.

Detent's trainer runs the serving kernels directly with the same list, so the only difference left between trainer and rollout is the batch, and the list already blocks that. Sampling randomness is fixed by seeded temperature sampling: like the prompt, the seed is an input of each request and does not depend on who else is in the batch. On the RL path, Detent accepts only this sampling method and refuses all others.

Hoot saysTwo computation paths for the same tokensClose ✕
Same parameters, same tokens, only the batch changes: why do the bits still match? Because the kernel-selection step can no longer read the batch.
The same tokens go down two paths, a server rollout and a trainer recompute, with different batches and pinned kernel selection on both sides; the resulting logprobs are bit-identical and KL is 0. The Stock baseline result is shown below.Same tokensprompt, seed, this step's parametersServer rolloutBatch A: one batch stateTrainer recomputeBatch B: another stateKernel selectionpinned, cannot read batchsame kernels + tupleKernel selectionpinned, cannot read batchsame kernels + tupleRecorded logprobat rolloutRecomputed logprobon the trainerBit-identicalKL is 0Stock baseline (trainer on transformers)same comparison matches only 1.0–47.7% of tokens
The same tokens go down two paths, a server rollout and a trainer recompute, with different batches and pinned kernel selection on both sides; the resulting logprobs are bit-identical and KL is 0. The Stock baseline result is shown below.Same tokensprompt, seed, this step's parametersServer rolloutBatch A: one batch stateTrainer recomputeBatch B: another stateKernel selectionpinned, cannot read batchsame kernels + tupleKernel selectionpinned, cannot read batchsame kernels + tupleRecorded logprobat rolloutRecomputed logprobon the trainerBit-identicalKL is 0Stock baseline (trainer on transformers)same comparison matches only 1.0–47.7% of tokens
  1. The kernels and tuple fix the computation path. The pins block every path from the batch to a selection point, and the user supplies the kernels' own invariance.
  2. The parameters M are the premise of this comparison. The trainer recomputes with the parameters of that step, which is what makes bit equality meaningful; if the parameters change, the logprobs should change anyway, and that is not a difference caused by the batch.
  3. For the seed s, this records only the paper's conclusion and measurement. The paper says admitting only seeded sampling also fixes the sampled tokens; 512 seeded rollouts replay bit for bit under other batches and cache splits.
  4. Admitting only seeded temperature sampling is a separate patch for RL. The serving runs use greedy decoding.
  5. Read the Stock row with care: its trainer runs on transformers, so it differs from the rollout in more than the batch.

In the paper's notation, a request's output is y=F(x,s;M,D,T;B): x is the prompt, s the seed, M the model code and weights, D the deployment, T the tuple and B the batch state. The two paths in the picture differ only in B.

Sourcepaper §3 (Eq. 2, selection batch invariance, the argument that trainer and server differ only in the batch); §7 (the RL sampling patch); §8.1 (serving runs use greedy decoding); §8.6 (seeded rollouts, trainer bit agreement, Stock trains with transformers)

First, do the trainer and rollouts match? The authors follow the test protocol of TBIK (a kernel scheme with a fixed summation tree), running GSM8K on Llama-3.1-8B for 100 training steps. With Detent every token at every step matches bit for bit, and KL is 0. With Stock only 1.0–47.7% of tokens match across steps. SGLang-Det paired with a transformers trainer using batch-invariant operators reaches only 9.9–63.9%. My reading is that these numbers show batch invariance on each side separately does not make the two sides agree; Detent gets agreement by running the same kernels and the same list on both sides.

On time, Detent's rollout costs 1.03 times Stock per generated token and the whole step is within 4%; the combination of SGLang-Det and the transformers trainer is 1.47 times.

Next, does alignment affect training outcomes? The authors ran Qi et al.'s BF16 GRPO recipe on DeepSeek-R1-Distill-Qwen-1.5B, with one GPU serving and one training. Stock's reward fell from 0.733 at step 299 to 0.436 at step 500, and its AIME 2024 Pass@1 fell from 29.2% to 8.5%. Detent's KL stayed 0 through step 600, its reward rose to 0.843, and its time per token was 1.18 times Stock's, mostly spent in training. This shows that on this BF16 recipe the bit-aligned run stayed stable while the Stock run collapsed.

Detent's trainer matches its rollouts bit for bit at a cost close to Stock. (a) Share of tokens matching bit for bit and (b) time per token relative to Stock, both under the TBIK protocol on Llama-3.1-8B. (c) Training reward averaged over 20 steps and (d) AIME 2024 scores, both on DeepSeek-R1-Distill-Qwen-1.5B.
Detent's trainer matches its rollouts bit for bit at a cost close to Stock. (a) Share of tokens matching bit for bit and (b) time per token relative to Stock, both under the TBIK protocol on Llama-3.1-8B. (c) Training reward averaged over 20 steps and (d) AIME 2024 scores, both on DeepSeek-R1-Distill-Qwen-1.5B.

When one batch mixes requests with different sampling methods, the difference is starker. In another setup, the authors used Detent to generate 512 GSM8K rollouts at temperature 1 each on Qwen3-30B-A3B, Llama-3.1-70B and Llama-3.1-8B; regenerating them with different batch companions and different KV cache splits gave bit-identical results every time, and the trainer's rescoring matched generation as well. At 1,024 output tokens, throughput is 0.84–0.92 times Stock's. When SGLang-Det generates seeded rollouts, a single greedy or top-p request mixed into the batch changes the output in all 20 test cells. This shows the deterministic mode does not protect against mixed sampling.

When can request A's answer still change?

Detent's guarantee holds only within one deployment, one model and one list. If A gets a different list, its answer can differ, as the concurrency-8 example already showed. Changing multi-GPU parallelism (say from 4 GPUs to 8) breaks it too, because the summation grouping changes; consistency across parallelism needs kernels specially designed for it, like TBIK's.

Two things are still left to the user. Batch invariance of the kernels themselves is guaranteed by the user-supplied kernels; Detent handles only the choosing step. When adding a new kernel or engine, the user has to write its selection logic in SelIR by hand.

The cost when a deployment changes can be estimated: checking FA3's 7,168 candidate sets takes 130–180 seconds on 32 cores, and tuning each workload under a 5-minute budget per family takes 605–1,148 seconds (Qwen3-30B-A3B). The paper also solved 8 new deployments from their deployment specs alone (model architecture, parallelism, data type), covering parallelism 1 to 8, head dimensions 64 to 256, and Phi-3.5-mini and gemma-7b-it, and all tested batch-invariant. After tuning, latency is 0.97–1.09 times Stock's on ShareGPT and 1.01–1.13 times on closed-loop workloads. This shows the solve-and-tune process does not depend on the two main models above; for a new deployment you just rerun it.

Finally, back to request A: it gets the same bits in the morning and the evening because, before deployment, the selection points total_kv can influence (such as the split count) were replaced with constants from the list, and A does not have to take the slowest path for it. The cost becomes two things: one extra round of solving and tuning before deployment, and a new list whenever the traffic type changes.

How did this read for you?

One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.

Discussion

Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.

Loading the discussion…