Cache Engineering for Agents

An agent working on a hard task often goes back and forth with tools such as search, web pages and code execution for hundreds of turns. On every turn the model re-reads the KV cache of everything that came before. The paper measures that reading this old cache accounts for 82.4% of total inference cost, far more than generating new tokens. The usual way to save money is to delete or summarize old messages, but any edit to earlier text invalidates all the cache after it, so everything from the edit point onward has to be recomputed. A Tsinghua team proposes working at a different layer: leave the text history untouched and drop only selected entries from the cache, while the remaining entries are reused as they are. They built an inference serving system for this, called AKV. Across three models and four agent benchmarks, a simple policy of "keep the cache of only the 12 most recent tool outputs" beats the common "summarize when the length limit is reached" approach, with a 1.7-point higher average score and 27.7% lower average cost.

1. The problem: recomputation is expensive, and cutting everything at once loses information

The longer an agent's conversation gets, the more trouble the cache causes. The first problem is cost: the cache keeps growing and is read on every turn, so storing and reading it is expensive. The second is accuracy: a model can only handle so much context reliably, and once the content taking part in computation goes past that range, quality drops noticeably.

The standard fix today is to edit the text, also called "context engineering": delete old messages or replace them with summaries. But an LLM's cache is computed in order, and each token's cache depends on everything before it. The earlier the edit, the longer the stretch that must be recomputed. So common agent frameworks wait until the length hits some limit and then summarize most of the history in one go, keeping only the last few turns, which keeps the number of recomputations low. Cutting everything at once like this easily throws away useful information from the middle turns too.

The authors start with a comparison. They run GPT-OSS-120B on the first 100 questions of BrowseComp-Plus (a QA benchmark that needs many rounds of search to answer), 4 attempts per question with at most 100 turns. Keeping only the 12 most recent tool outputs by deleting text every turn costs 1.85 times as much as doing nothing, because of repeated recomputation. Summarizing once at 48K tokens cuts cost by only about 7% and drops accuracy by 5 points. Dropping old tool outputs directly from the cache improves accuracy and cost at the same time.

2. Method: three operations on the cache

The authors first separate two lengths. One is how long the conversation is as text. The other is how much cache actually takes part in later computation, which the paper calls the "active cache". Text editing changes the first; cache editing changes only the second.

In this framing, the paper defines three operations:

• Append: the normal case. Read in new tokens, attach their cache at the end, and increment the position index.
• Drop: exclude specified cache entries from all later computation. The remaining entries stay exactly as they are, the text history is unchanged, and new tokens keep counting positions upward as usual.
• Reposition: renumber the retained entries 0, 1, 2… in their original order, and number the next new token after them.

Why is the third operation needed? Dropping does not make the position indices smaller. The cache shrinks, but the index keeps climbing and will eventually exceed the largest position the model can handle. Repositioning pulls the indices back down. For models that use RoPE this step is cheap: to move an entry from position p to position q, rotate its key by the angle for q−p; the value stays the same and nothing is recomputed.

What gets dropped? The authors looked at the model's attention distribution and found two patterns: older content receives less attention, and at similar distances tool outputs receive less attention than the model's own reasoning. So the main policy is rolling drop of old tool outputs: keep all reasoning content and the 12 most recent tool outputs, and drop the cache of earlier tool outputs step by step.

3. The difficulty: the same text can have different caches

In a normal inference server, the same text prefix means the same cache, so caches can be reused by matching prefixes. Cache editing breaks this: the same text, reached through a different order of operations, yields a different cache.

Take the example in the figure. All three paths end with only B and C kept. But in ①, C was computed while A was still visible, so it carries A's influence. In ②, A had already been dropped when C was computed. In ③, there was no A before B even when B itself was computed. So C's cache differs across the three paths. The paper therefore describes a cache state by "the full sequence of operations applied since an empty cache": for a given model, that sequence uniquely determines the current cache. The inference server has to remember both the text history and the operation history to reuse cache correctly.

4. AKV: turning this into an inference service

AKV's interface is almost identical to a standard chat API. The client sends the full message list as usual, with one extra optional field that says "after message N, drop these entries and reposition here". The client does not need to know which caches the server still holds; each turn it just appends messages and operations as usual.

Inside the server there are three mechanisms.

The first is reuse. Inference servers usually do prefix caching with a radix tree, so requests with the same prefix share a segment of cache. AKV adds "drop" and "reposition" nodes to the tree and matches on both text and operations during lookup. That is why ① and ② in the figure can share the A–B segment, while each keeps its own copy of C because A was dropped at different times.

The second is reclamation. A cache entry can be freed only when no in-flight computation still needs it. For example, once both ① and ② have dropped A, A's cache can be freed without waiting for the requests to finish. When GPU memory is tight, the server first reclaims cache that has been dropped and only then cache from finished requests, because the latter is more likely to be needed next turn.

The third is recovery. Once cache has been evicted from GPU memory, it has to be recomputed the next time it is needed. But a normal recompute from scratch can only produce the cache of "nothing ever dropped", and replaying each operation step by step would take many passes. AKV instead splits the content to be recomputed, at each layer, into segments at the drop and reposition points. Each segment attends only to itself and to the cache that earlier segments still retained at that time, and all segments are computed in one pass. This way a single computation restores the exact state after the drops and repositions.

5. Results

The baselines are two "summarize at the limit" approaches. The "standard" approach uses each model's own higher limit (96K tokens for GPT-OSS-120B and MiniMax-M2.7, 192K for MiroThinker-1.7-Mini). The "text editing" approach uses the same summarization but lowers the limit to 48K for all models. AKV uses rolling drop with "keep the 12 most recent tool outputs". There are four benchmarks: two multi-round search QA sets (BrowseComp-Plus, xbench-DeepSearch), the 103 text-only questions of GAIA, and SWE-bench Verified, which fixes bugs in real code repositories. Cost is priced on tokens actually read and generated. To avoid favoring their own method, the authors also charge every token touched by repositioning at the cache-read price, deliberately overestimating the cost of that step.

Average over the four benchmarks (score is pass rate %, cost is cents per task):

On all three models AKV has the highest average score, and its average cost is 27.7% lower than the standard approach and 21.9% lower than text editing. It does not win every cell, though: on MiniMax-M2.7, text editing scores higher on BrowseComp-Plus, and the standard approach is slightly higher on SWE-bench.

Why does accuracy improve? The authors break the results down by conversation length. Below 48K tokens the three approaches are about the same; between 48K and 96K, text editing is worst; above 96K, the standard approach is worst. These are exactly the points where each one triggers summarization, which shows that the lost points come from information thrown away by cutting everything at once. Rolling drop removes the oldest tool outputs a little at a time, so it avoids this problem.

On speed, the authors measure serving efficiency on SWE-bench with 8 A800 GPUs. At the highest concurrency each system can sustain, AKV's average latency is 1.36 times faster, the slowest 5% of requests are 1.51 times faster, and output throughput is 1.18 times that of the standard approach. The gap widens as concurrency goes up: the standard approach runs short of GPU memory and keeps evicting and recomputing cache, so at concurrency 16 the slowest 5% of requests wait 230 seconds, versus 31 seconds for AKV.

Several ablations are also useful. Dropping without repositioning scores 55.0 on BrowseComp-Plus; adding repositioning raises it to 62.0, because the agent can keep searching instead of hitting the position limit and being forced to stop. Dropping the cache of reasoning content, alone or together with tool outputs, is worse than dropping only tool outputs. The authors also tried letting the agent decide what to drop, and two models reached 61.0% and 64.0% on a subset of BrowseComp-Plus, which suggests the policy itself still has a lot of room to explore.

6. Limitations and practical value

The limitations the authors list: what to drop and when still needs more exploration; agents have not yet been trained specifically under this mechanism; and models with newer architectures such as linear attention are not supported. A few other points are worth noting. The main policy in the experiments is a fixed hand-written rule, and the best "how many to keep" may vary by task. Dropped content still leaves an indirect influence through cache computed afterwards. The authors see this as a strength, but it also means results depend on the order of operations, so debugging and reproduction require recording the full operation sequence.

For teams building agent platforms and inference services, the value of this paper is that it moves "context compression" from the text layer to the cache layer. Suppose your agent produces a lot of tool output, such as retrieval results, web pages or code execution output, and you currently control length by "summarize at the limit". First check on your own trajectories how much the model's attention to tool outputs differs from its attention to reasoning content. Then try "rolling drop of old tool outputs + repositioning", the most robust combination in the paper. The prerequisite for deployment is a serving layer that can tell operation histories apart: if the server reuses cache only by matching text prefixes, different drop times under the same text will be mixed up incorrectly. AKV is built on SGLang, the code is public, and a client needs to send only one extra field.