Contents · 10 sections
1What problem is it actually solving?2Pipeline overview: store two representations, score two ways at query time3Sparse token matching: each query token credits only a few records4Why whole-record similarity is still needed, and how fusion works5A small query-side correction6Experiments: how big is the gain, and where does it come from7Cost: storage and latency8Limitations as I see them9How a retrieval team could try it10VerdictSHARE
Teams working on conversational memory, long-document QA or RAG retrieval should read this paper, because it finds a signal that is already computed and then thrown away: before an embedding model outputs its final vector, it has already produced a contextual representation for every token. Storing those and matching at token level raises recall noticeably without training anything. The core idea in one sentence: the whole-record vector decides "is this on the same topic", token matching finds "which record actually contains that sentence", and the two scores are standardized separately and added. On Qwen3-Embedding-4B, average Recall@10 over eight benchmarks rises from 62.5 to 73.8, and to 75.7 after training a small query-side module. The price is index size: the full token index is about 68 times the single-vector index (my calculation from the paper's table), and still about 10 times after 4-bit compression.
What problem is it actually solving?
Start with an illustrative scenario. A chat assistant has stored a few hundred memories for one user, each a stretch of conversation. The user asks "which day did I say I'd go to the dentist?" Three candidates:
| Memory (illustrative) | Content | Relation to the question |
|---|---|---|
| A | A long talk about work stress, ending with "going to the dentist next Wednesday afternoon" | Actually contains the answer |
| B | All about oral care and choosing an electric toothbrush | Very similar topic, no date |
| C | Visiting family in hospital at the weekend | Has "hospital" and "weekend", unrelated |
Single-vector retrieval compresses each memory into one vector and takes a dot product with the question vector. B talks about teeth throughout, so its vector looks most like the question; most of A is about work, and the one line "dentist next Wednesday" gets averaged away. A single-vector score can say whether a record is similar overall, not whether it contains that one line, and memory retrieval is very often the second kind of question.
The paper's Figure 1 is a real example: with the whole-record vector alone, the relevant memory ranks 28th; with token matching added it ranks 1st. The appendix statistic says more. LoCoMo (a long-term conversational memory benchmark) has 1,976 questions. For 607 of them, the original top 10 contained no relevant memory and fused scoring recovers at least one; 84 go the other way, a net gain of 523. The recoveries spread across question types, but not every type gains on net: open-domain questions have 9 recoveries and 10 losses, roughly a wash.

The idea is close to ColBERT: each query token is compared with document tokens. The difference is that ColBERT trains a dedicated token-level encoder, whereas this paper reuses the frozen general-purpose embedding model you already have, changing neither its parameters nor its original whole-record vector.
Pipeline overview: store two representations, score two ways at query time

The pipeline has three steps:
| Stage | Input | What it does | Output |
|---|---|---|---|
| Index build (once) | Title and text of each memory | One forward pass of the frozen model | One whole-record vector plus the last-layer representation of each content token, all L2-normalized; token representations stored in FP16 with their memory ID |
| Token matching | Representation of each content token in the query | HNSW nearest-neighbour search over all tokens of the candidate memories, top 64 | One local score per memory |
| Fusion | Local score and whole-record dot product | Each score is standardized within the current candidate set, then added with weights 0.5 and 0.5 | Final ranking |
A few details decide whether the method can be used as is. First, only "content tokens" are stored: instruction prefixes, padding and special tokens are not, and the whole-record vector keeps the model's native pooling (last valid token for Qwen, mean for the Llama family). Second, memory encoding does not depend on the query, so one index build serves every query; when the query adapter described later is trained, this index does not need to be rebuilt. Third, each query's candidate memory set is given by the evaluation protocol — for instance the few hundred memories cut from one user's long conversation — and token search runs only inside that set, not over a site-wide corpus. The paper measures storage and latency on a LoCoMo memory bank of 568 candidates.
Next, token matching and fusion in detail.
Sparse token matching: each query token credits only a few records
Why not just compute MaxSim between each memory and the query? MaxSim is ColBERT's scoring: for each query token, find the most similar token among a memory's own tokens and average those maxima. It compares every token of every candidate. This paper goes the other way: each query token first runs one global nearest-neighbour search over all tokens of all candidates, keeps only the k most similar (k=64), and then looks up which memory each token belongs to.
The difficulty is memories that are not retrieved. The top 64 matches for some query token may sit in three or four memories; the other few hundred have no score on that token. If they get 0, a token ranked 65th differs from the 64th by a sliver of similarity, yet its memory's score drops by a whole step. The authors borrow the fix from XTR (a retrieval method that fetches only some tokens and then estimates the missing scores): the similarity of the 64th retrieved match, the least similar one, becomes a "floor score" (0 if negative), written . A memory not retrieved for this token gets the floor; a retrieved memory gets the larger of the floor and the highest similarity among its retrieved tokens. Finally, average over all query tokens:
is the number of content tokens in the query, is the 64 memory tokens retrieved by query token , and is a normalized token representation, so the dot product is cosine similarity. The floor is the same for every memory on a given query token, so only matches above the floor create differences between memories.
A hand calculation with the opening example (illustrative, with k shrunk to 3). The question has just two content tokens, "dentist" and "which day".
- "dentist" retrieves: "teeth" in A at 0.82, "dental" in A at 0.78, "hospital" in C at 0.55. The floor is the third one, 0.55. A gets 0.82; C's best match equals the floor, so it gets 0.55; B was not retrieved at all and also gets 0.55. So C's "hospital", although retrieved, earns nothing over the unretrieved B.
- "which day" retrieves: "every day" in B at 0.60, "next Wednesday" in A at 0.58, "weekend" in C at 0.50. Floor 0.50. A gets 0.58, B 0.60, C 0.50. A generic word like "which day" has similar matches in every memory, so the three differ by a few hundredths and barely affect the ranking.
- Averaged: A is 0.70, B 0.575, C 0.525. A comes first on the strength of its unique match for "dentist".
If the floor were 0 instead, B would score 0 on "dentist" and its local score would be only 0.30, far below C, even though B really is about teeth. The paper's control backs up the floor: across two Qwen models and two conversational datasets, replacing the floor with 0 cuts Recall@10 by 1.61 to 7.14 points. Conversely, replacing approximate HNSW search with exact search changes Recall@10 by only −0.06 to +1.62 points. This shows that filling in scores for unretrieved memories matters far more than removing approximate-search error.
Why does sparse matching beat MaxSim? The result is clear: with the same fusion and query module, switching to MaxSim costs 3.4 points of average Recall@10 on Qwen-4B. My reading: MaxSim gives every memory its own best match on every query token. So the longer a memory, the more that "own best" score gets inflated, even when it is only middling across all candidates. Global top-64 plus the floor lets only matches that make the global top 64 differentiate memories. Everything else is recorded as the floor, so middling scores accumulated through length stop counting. Conversely, if one token of an irrelevant memory happens to land in the global top 64, it still counts, and the floor does not stop that.
Why whole-record similarity is still needed, and how fusion works
Token matching only sees local evidence, so it misses memories that are "about exactly this, in completely different words". The whole-record dot product scores every candidate and fills that gap: a memory with not a single retrieved token can still reach the top 10 through whole-record similarity.
The two scores are on different scales and cannot be added directly. The authors standardize per query: within the current candidate set, compute the mean and standard deviation of each score, subtract the mean from each memory's score and divide by the standard deviation (with a floor of ). Then add with a weight:
is the standardized score and the weight on whole-record similarity, 0.5 by default. Because standardization is done query by query, the two scores stay on the same scale across models and candidate sets, which is why the 0.5 default works across all eight models.

The figure shows where the gain comes from: at α=1, Qwen-4B's Recall@10 on LoCoMo is just above 40, while α=0 is already close to 0.5. In other words, most of the improvement comes from the token-matching branch, and whole-record similarity adds roughly one more point on top (removing fusion and keeping only the local score drops Qwen-4B's average Recall@10 from 75.7 to 74.5).
A small query-side correction
If you have some question–relevant-memory labels, you can also train a very small query adapter. It is a rank-32 low-rank residual: for each query token representation compute , with and shared across positions, then rescale the result back to the original length, so it changes direction but not length. starts at 0, so at the start of training nothing changes. It has parameters — about 160 thousand for Qwen-4B with (my calculation).
The module changes only query token representations and only feeds the token-matching branch. The query's whole-record vector, the encoder and every representation in the memory index stay fixed, so the memory index needs no rebuild before or after training.
Training takes a detour: top-64 neighbour selection at inference is discrete and has no gradient. During training the authors use MaxSim instead (max over all tokens of each candidate), with a multi-positive softmax cross-entropy (temperature 0.07) that pushes relevant memories up; at inference the trained module is plugged back into sparse matching. Training data is just 1,024 questions plus 256 for validation, from EvoTrain-180K, for 25 epochs, with the checkpoint picked on validation.
The effect is inconsistent: +1.9 points of average Recall@10 on Qwen-4B, only +0.3 on Qwen-0.6B, a slight −0.1 on Llama-3B, and no change on EmbeddingGemma. The training-free version is the real substance of the paper.
Experiments: how big is the gain, and where does it come from
Evaluation follows the protocol of LMEB (a memory-retrieval benchmark): eight datasets, 5,873 questions — three conversational memory sets (LoCoMo, LongMemEval, REALTALK) and five document sets (ESG reports, two scientific-paper QA sets, coronavirus literature QA, and a long-document retrieval set). The metric is the mean of per-dataset Recall@10 and NDCG@10. Same-model baselines use exactly the same inputs.
| Method (Qwen3-Embedding-4B) | Avg Recall@10 | Avg NDCG@10 |
|---|---|---|
| Original whole-record vector | 62.5 | 48.0 |
| BM25 + original vector (same standardization, half each) | 69.7 | 56.6 |
| Search-Adaptor (trains an adapter on the whole-record vector) | 66.9 | 52.3 |
| MaxSim directly on frozen token representations | 67.8 | 54.3 |
| ColBERTv2 (standalone system) | 70.8 | 59.5 |
| This paper, training-free | 73.8 | 59.9 |
| This paper, with query adapter | 75.7 | 62.3 |
The table shows three things. First, the training-free version already beats the purpose-trained ColBERTv2. Second, cheap BM25 hybrid retrieval already captures most of the gain: from 62.5 to 69.7, about 64% of the training-free improvement (my calculation), with a BM25 index of only 0.23 MiB. Third, using the same token representations, plain MaxSim reaches only 67.8; sparse matching plus fusion adds 6 more points, so the scoring itself matters a lot.
The gain is largest on conversational memory. Qwen-4B goes from 41.6 to 67.8 on LoCoMo and from 59.2 to 95.9 on LongMemEval. Discount the latter: in the same table the smaller Qwen-0.6B already scores 84.8 on LongMemEval, so the 4B original vector is unusually weak on that set, and part of the jump is fixing that weakness.
Across the other models, all eight improve, but the size depends on the starting point: EmbeddingGemma-300M from 72.8 to 79.8, Llama-Embed-Nemotron-8B from 76.8 to 79.2, Llama-NV-Embed-Reasoning-3B from 77.9 to 79.6. This means the stronger the original vector, the less room token matching has to fill, though none is negative.
Can chunking replace token matching? The authors ran a Chunk-MaxP control: cut memories into 512-token chunks (64 overlap), encode each chunk separately, take the best chunk. It scores almost exactly like the original vector. Shrinking the window to 128, 64 and 32 tokens on Qwen-0.6B LoCoMo keeps Recall@10 between 44.99 and 46.87, against 46.87 for the original vector and 67.98 for the training-free method. So the gain does not come from "cutting finer": independently encoded small chunks lose context, while the token representations here are computed in the context of the whole memory.
Downstream QA improves too. The authors let Qwen3-30B-A3B answer from the top 10 memories and GPT-4o-mini judge correctness. On Qwen-4B, LoCoMo accuracy goes from 63.44% to 67.86% and LongMemEval from 67.80% to 72.60%. All eight models improve on LoCoMo; on LongMemEval seven improve, and only Qwen-8B dips from 74.0% to 73.8%. This shows the retrieval gains do carry through to final answers rather than just making ranking metrics look better.
Cost: storage and latency
| Method (LoCoMo, 568-candidate memory bank) | Recall@10 | Index size (MiB) | Per query (ms) |
|---|---|---|---|
| BM25 | 48.28 | 0.23 | 0.76 |
| ColBERTv2 | 61.82 | 1.54 | 27.27 |
| Qwen-0.6B original vector | 46.87 | 2.23 | 16.66 |
| This paper 0.6B, full index | 68.53 | 153.93 | 29.21 |
| This paper 0.6B, 4-bit scalar quantization | 68.37 | 26.17 | 21.68 |
| Qwen-4B original vector | 41.60 | 5.56 | 22.71 |
| This paper 4B, full index | 67.93 | 376.65 | 35.12 |
| This paper 4B, 4-bit scalar quantization | 68.51 | 57.27 | 33.15 |
Storage is the main cost, because the full index stores a 1024- or 2560-dimensional FP16 vector per token; on 0.6B it is about 69 times the single-vector index.
4-bit scalar quantization costs almost nothing in quality — averaged over two models and eight tasks, Recall@10 is actually 0.33 points higher — and cuts size by more than 80%. But it is still around ten times the single-vector index, and ColBERTv2, with its built-in residual compression, needs only 1.54 MiB for the same bank, more than ten times smaller than the quantized index here. So this method wins on quality, not storage.
It can be squeezed further: a random projection to 128 dimensions before quantization brings the 0.6B index down to 10.70 MiB. The cost is a bigger quality drop — on 4B, Recall@10 falls from 67.93 to 63.09 — which suggests the larger model's token representations do not survive such a coarse projection. On latency, the quantized version adds about 5 ms per query over the original vector on 0.6B and about 10 ms on 4B (my calculation from the table), both including query encoding, which is acceptable for memory retrieval.
Limitations as I see them
First, scale has only been validated at the level of "one user's memory bank". Each query's candidates are a few hundred memories given by the benchmark; a 568-memory bank already takes 154 to 377 MiB, and one LongMemEval bank reaches 1460 MiB on 4B. On a general corpus of millions of documents the token count would be orders of magnitude larger, and the paper has no experiment at that scale. It fits memory scenarios that are naturally split per user or per session.
Second, the average gain is inflated by one weak baseline. Qwen-4B's training-free version gains 11.3 points on average; LongMemEval alone goes from 59.2 to 95.4, which accounts for about 4.5 points of the eight-set average (my calculation), and that is precisely the 4B original vector's weak spot; on strong models the gain is only 2 to 3 points. BM25 hybrid retrieval already captures about 64% of the gain on 4B at almost no cost. So the headline number should not be taken as the expected gain on your own model.
Third, latency was measured on only 8 fixed queries, each repeated 5 times with the median taken, on a 568-candidate bank with single-threaded HNSW. There is no data on tail latency or concurrency in a real service.
How a retrieval team could try it
- First run BM25 + vector hybrid retrieval on your own memory or long-document eval set, with the same per-query standardization and equal weights. It is the cheapest control and shows how much "local lexical matching" is worth on your data.
- Then store the last-layer content-token representations of your existing embedding model, implement sparse matching and fusion with k=64, the floor score and α=0.5, and go straight to 4-bit quantization. See how much it adds over the hybrid; only that extra is the true value of the token representations.
- Measure per-memory storage and per-query latency at your real bank size. If the bank can be split per user or per session, the scale is close to the paper's and you can ship it directly; for one large global index, work out the token-index memory first.
- Try the query adapter last, once you have labels, and confirm on a validation set that it is a net gain on your model.
Verdict
Within the same embedding model, token representations and the whole-record vector are two complementary signals, and the former has simply been thrown away — that is the main takeaway.
| Dimension | Assessment |
|---|---|
| Problem importance | High. In conversational memory and long documents, "the answer is in one sentence" is common, and single-vector retrieval systematically loses there |
| Method novelty | Medium. Global token search and the floor score come from XTR, and reusing token representations of a frozen model has prior work; the contribution is the combination and the thorough controls |
| Experimental evidence | Medium-high. Eight models, eight datasets, controls for chunking, BM25 hybrid, exact search and the floor score, plus downstream QA; missing large-corpus and online data |
| Engineering cost | Medium-high. No training, but the index is more than ten times the single-vector index, and there is an extra HNSW to maintain |
| Worth trying? | Yes, provided the bank can be split small. Compare against BM25 hybrid first and confirm the extra gain justifies the storage |
| Where it fits | Long-term memory for chat assistants, per-user document QA, long-conversation RAG |
How did this read for you?
Hoot made a note. Thanks for the feedback.
One tap, no sign-in; tap again to undo. Only anonymous counts are kept, never who you are.
Discussion
Add experiment details, reproduction notes or a different take. Requires a GitHub sign-in.
Loading the discussion…