Let the Gate Decide: Per-Lookup Precision for Quantized Conditional Memory in LLMs

When you quantize a language model's n-gram memory tables to low bits, the usual move is to keep the most frequently accessed rows precise. This paper finds that the rule backfires here: giving extra precision to the 41% of lookups with the most frequent rows does worse than picking 41% at random. The reason is the gate. In conditional memory layers such as DeepSeek's Engram, each token uses the preceding n-gram to fetch a few vectors from hashed tables, and a scalar gate that depends on the hidden state scales them before they enter the residual stream. Common function words get small gates, rare entity names get large ones, and quantization error is scaled by the gate. The authors build GateRQ (gate-driven residual quantization): each row is stored as a few levels of 1-bit residual codes, every lookup first reads level 1 to compute its gate, and only the lookups with the largest gates read deeper levels. Without retraining, it cuts DeepSeek-V4.1-Flash's table memory from 23.6 GiB to 3.0 GiB per GPU and raises long-context throughput by 11% to 31% over FP8 tables in GPU memory; at 2.10 bits read per lookup it is only 0.2 PopQA points below FP8.

Both inference engineers and people compressing recommender embedding tables should read it. For inference, the reason is direct: the conditional memory tables are 40% of the DeepSeek-V4.1-Flash checkpoint, 203 GB in FP8, and every deployment team will have to shrink them. For recommendation, the lesson is more interesting: the old "give hot rows more bits" rule assumes every access contributes equally, and that assumption breaks as soon as a context-dependent scale sits between the lookup and the output. The core idea in one sentence: precision becomes a property of each lookup rather than of the stored row, decided by a gate computed cheaply from a 1-bit code.

What problem is it solving?

First, what a conditional memory layer does. Engram attaches a module to two layers. At each position it takes the 2-gram, 3-gram and 4-gram ending at the current token, and each n-gram order uses 8 different hash functions to look up 8 tables, so 24 rows come back. Multiple hashes reduce collisions. The concatenated rows go through one projection to form a key, which is compared with the current hidden state to produce a gate between 0 and 1, and through another projection to form a value. The gate first scales the value: DeepSeek adds the result directly to the residual stream, while Qwen and original Engram add the output of a short convolutional branch to the gated value before injecting it. In both models the residual stream is split into 4 parallel branches; each branch computes its own gate from its own key projection and hidden state, while the value is shared, so each position has 4 gates. The tables do no matrix multiplication, so they are cheap to compute, yet they hold a lot of factual knowledge. Zeroing them costs DeepSeek-V4.1-Flash 36.8 points on PopQA (a QA set about long-tail entities), about seven tenths of its accuracy, which shows the tables are not an optional extra.

The problem is size. On an 8-GPU machine, keeping the FP8 tables in GPU memory takes 23.6 GiB per GPU and leaves a KV cache nine times smaller than when the tables are offloaded; offloading instead pins a large amount of host memory.

Quantization is the natural fix, but going straight to 1 bit costs 3.87 PopQA points. The loss is concentrated on long-tail knowledge: the rarest quartile of entities loses 6.24 points, the most popular only 1.51. So quantization error is not spread evenly.

A concrete question from the paper shows why. Asked "Who is the author of The Pianoplayers?", the FP8 tables answer correctly and the 1-bit tables do not. The table shows two positions on the 2-gram heads of layer 2:

Following the recommender habit of protecting frequent rows, the row for The would be well protected, yet its gate is only 0.035, so its error is shrunk before it reaches the residual stream. The rows that decide the answer belong to Pianoplayers, and they appear too rarely to ever be chosen by frequency.

Why does frequency-based allocation fail?

The authors first split a layer's quantization loss over individual lookups. A lookup's error reaches the output through two paths: it changes the key and therefore the gate, and it changes the value. The gate path turns out to be negligible. Gates computed from the 1-bit table and from FP8 correlate at 0.989 to 1.000 position by position; quantizing only the gate inputs leaves PopQA essentially unchanged, while quantizing only the values costs as much as quantizing both. Almost all the error comes from the value.

With the gate error dropped, the damage of one lookup is roughly its importance times the row's relative quantization error, where importance is:
$$I_{t,j}=G_t{\tau }_j\Vert {\mathbf{e}}_{t,j}{\Vert }_2^2$$
Here $G_t$ is what the authors call the gate energy at position $t$, the sum of the squared gates of the 4 branches; ${\tau }_j$ is how much head $j$'s value projection amplifies error, one constant per head; and $\Vert {\mathbf{e}}_{t,j}{\Vert }_2^2$ is the energy of the retrieved row. The intuition: the larger the gate, the more sensitive the head and the louder the row, the more a quantization error hurts. Only the gate energy changes with context.

A quick calculation with the example above (one branch only, illustrative): the squared gate is about 0.0012 at The and about 0.041 at Pianoplayers, more than thirty times apart. The same quantization error does an order of magnitude more damage on the latter.

This figure is the direct evidence. Gate and access count are negatively correlated in every layer, every n-gram order, and in English, Chinese and code text. Frequent rows lose on two counts: small gates and small row norms.

Why not rank rows by their average gate instead? The authors find this only partly helps, for two reasons. First, many of the most important lookups read rare rows that appear too seldom to estimate an average gate; rows never seen in the corpus alone carry 23% of 2-gram importance. Second, the same row's gate swings with context: in layer 15, the row read for "a good" has a gate energy of 45.6×10⁻³ in "key to a good holiday" and 0.8×10⁻³ in "there is a good and clear reason". Importance is also highly concentrated: on PopQA the top 1% of lookups in layer 15 carry 54% of the importance. Precision therefore has to be decided per lookup, from the gate at hand. Hoot says▾

Hoot saysSame row, different context: 57 times the importanceClose ✕

Precision must follow each lookup, not each row: for one row, head sensitivity and row energy are fixed and only gate energy changes with context, so the importance ratio of two lookups of that row equals the gate-energy ratio, 45.6 ÷ 0.8 ≈ 57.

key to a good holiday

gate energy

45.6×10⁻³

head sensitivity, row energy

same row, unchanged

row-level allocation

same precision as the right

per-lookup allocation

reads deeper levels

Importance is about 57 times the right column's.

there is a good and clear reason

gate energy

0.8×10⁻³

head sensitivity, row energy

same row, unchanged

row-level allocation

same precision as the left

per-lookup allocation

stops at the first level

The same quantization error does far less damage.

Both columns are lookups of the same layer-15 row for “a good”, with gate energies from the paper; row-level allocation gives both the same precision, while per-lookup allocation lets only the first read deeper levels. Source: paper §3 importance formula and the “a good” example (this section)

How GateRQ works: read 1 bit, then decide how deep to go

This seems circular: you need the gate to decide precision, but the gate is computed from the retrieved row. The previous section breaks the circle: the gate is insensitive to quantization and can be computed accurately from a 1-bit code.

The pipeline has four steps:

The key division of labour: level 1 computes the gate and the ranking; deeper levels only refine the value. The gate is always the one computed from level 1, and the same gate both ranks the lookup and scales its value.

Why residual codes rather than storing a 1-bit copy and a 3-bit copy? Because residual codes let you add precision after a partial read. Level 1 encodes the row with 1 bit; level 2 encodes the error left by level 1 with 1 bit; level 3 encodes what is left after that. Reading k levels means summing the first k. A two-dimensional example with plain sign quantization (illustrative): the row is (0.9, −0.3); level 1 uses its RMS, about 0.67, as scale and reconstructs (0.67, −0.67); the remaining error is (0.23, 0.37), level 2 has scale about 0.31 and both signs positive, giving (0.31, 0.31); the sum is (0.98, −0.36), much closer to the original. The paper uses TCQ (trellis-coded quantization, a 1-bit coder more accurate than per-element signs) and also works with integer and product quantization.

How is the budget guaranteed? Each level costs about 1.06 bits on DeepSeek (a 1-bit code plus one FP16 scale per row). In each batch, the authors take quantiles of importance: the top 41.2% of lookups read level 2 and the rest stop at level 1. On average that is about 1.06 + 0.412 × 1.06 ≈ 1.50 bits. Because the fractions are fixed per batch, every batch reads the same average number of logical bits per looked-up element. The total logical read volume depends on the number of lookups. Packing and alignment also affect the physical bytes transferred. This "refine where the marginal return is largest" rule is classic bit allocation; GateRQ changes the unit from table rows to individual lookups.

Experiments: at the same 1.5 bits, who picks better?

The question is whether gate-based selection beats other selection rules when every rule reads exactly the same bits per lookup. The baselines use the same residual codes and the same level fractions and change only the ranking: Prior uses the same importance score with current gate energy replaced by the row's expected gate energy, retaining head gain and row energy; Random ranks at random, and Frequency ranks by access count. The two models are DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next (whose similar layer is called PLE), with no retraining.

(In the bits column the first number is DeepSeek and the second Qwen; the four tasks are PopQA, TriviaQA, CMMLU (a Chinese multitask knowledge benchmark) and open-domain NQ-open.)

At 1.50 bits, GateRQ beats frequency-based selection by 1.84 PopQA points on DeepSeek, and frequency is even slightly below random. This matches the analysis: frequent rows are exactly the rows with small gates. Across two models, three coders and two budgets, 12 settings in all, GateRQ wins all 36 paired comparisons significantly.

At 2.10 bits, GateRQ is only 0.2 points below FP8, statistically indistinguishable, while reading all three levels takes 3.19 bits. So most of that extra bit would have been spent on lookups that do not matter.

The authors also run an offline ablation. The offline metric is the weighted error above, importance times relative error. At the 1.50-bit setting 41.2% of lookups read level 2; if every lookup read level 2, the weighted error would fall by a fixed amount, and the 41.2% chosen by the gate capture 93% of that drop, against 70% for Prior, 41% for random and 26% for frequency. Measured on PopQA, there is a gap between reading level 1 only and reading two full levels, and the four rules recover 96%, 58%, 41% and 34% of it, in the same order as the offline proxy, which means the importance formula, though approximate, is good enough for ranking.

Outside PopQA the differences are much smaller: only about half of the paired comparisons are significant on TriviaQA and CMMLU, and only one on NQ-open. These tasks lean less on long-tail entities, so level 1 alone already loses little.

What does the saved memory buy?

Serving 32K-token inputs in vLLM, GateRQ cuts the tables from 23.6 GiB to 3.04 GiB per GPU, so the KV cache capacity reported by vLLM grows about 8.1 times. With 128 concurrent requests, throughput rises from 349.6 tokens/s with FP8 tables in GPU memory to 458.1 tokens/s, meaning the same machines serve about 30% more traffic.

It matters where this gain comes from. The authors state that the throughput gain comes from the freed KV cache, not from reading fewer bits: level 1 alone reaches 464.5 tokens/s, about the same as GateRQ, and FP8 tables offloaded to host memory reach 460.1. So GateRQ's real gain over FP8 offloading is host memory: pinned memory drops from 188.7 GiB to 25.8–48.6 GiB at about the same throughput. Its gain over level 1 alone is accuracy: about 26 GiB of extra host memory buys back 2.85 PopQA points.

The table side itself is about 1% of a prefill step; at 8,192 tokens it drops from 3.1 ms to 1.3–1.6 ms, with no visible end-to-end effect.

Limitations as I see them

First, the throughput numbers come from single runs. The authors say each serving cell is one run and that serving accuracy has not yet been cross-checked against the accuracy runs. Read the 11% to 31% as a direction, not a precise gain.

Second, the benefit shows up almost only on long-tail facts. On TriviaQA only 20 of 36 comparisons are significant, and on NQ-open only one. If traffic is mostly common QA or code, level 1 alone already captures all the memory savings, and the long-tail loss may not be felt at all.

Third, per-batch quantiles make a lookup's depth depend on the other requests in its batch. When a batch holds a single token, as is common in decoding, the lookups at that position share one gate and the ranking falls back to head gain and row energy. The serving experiments use 32K-token inputs, dominated by prefill; the paper does not measure how much of the gate's advantage remains on decode-heavy workloads.

How a recommendation or inference team could try it

Step one: check the premise. On your own model, record each lookup's gate and the row's access count and compute their correlation. The paper sees −0.02 to −0.63; if yours is positive (as in a small model trained on 82 million tokens that the authors cite), frequency-based allocation may still be fine.

Step two: ship the level-1-only version. It needs no ranking logic and captures all the GPU memory savings. Measure how much your business metrics drop; if little, stop there.

Step three: if the drop is large, add level 2 and gate ranking, starting at 1.50 bits. Judge success on two things only: whether long-tail queries recover, and whether host-to-GPU reads are fully hidden inside prefill.

Large recommender embedding tables can borrow the idea too: if the model has a per-access scale such as target-attention weights, check how it relates to row frequency before deciding which rows get precision.

Verdict

In one sentence: when compressing gated lookups, the question is how important this particular lookup is, and that answer can be computed first from 1 bit.