When Agentic Retrieval Fails Silently: Framework and Production Evidence from a Creative AI Assistant

Adobe Express's template gallery search and its AI assistant run on the same retrieval system and serve the same users. Intuition says assistant users don't look at result lists, so they should reformulate queries less. In fact the chat entry point has a reformulation rate of 44.4%, against 33.1% for the gallery, 11 points higher. The reason is that "reformulation" means different things on the two sides. In the gallery, a user reformulates because the results were irrelevant and they try other words. In chat, most reformulations come after the user sees a generated result and wants to tweak it, such as "a bit bluer" or "use a bolder font". More surprising still, the assistant actually called retrieval on only 0.31% of prompts. The three Adobe authors call this the attribution gap: CTR, MRR and behavioral NDCG all assume the user saw a ranked list and acted on it, and in an AI assistant that assumption breaks in three directions at once. The paper's evaluation framework, LARA, uses three independent layers of measurement plus four log fields to give credit back to retrieval, without training any new model.

1. The problem: the agent reads first, the user comes later

The authors argue that the failure comes from three structural changes, none of which tuning can fix.

The first change is who looks at the results. In the gallery, a user searches "birthday invitation", scrolls through 40 templates and clicks one, and that click is a grade for retrieval. In the assistant, the user says they want a birthday invitation, a search sub-agent retrieves templates, and the agent picks one and turns it straight into a design; the user never sees the other 39. There may be no click at all, and what the user does see has already been filtered and reordered by the agent.

The second change is that action takes longer to arrive. The same asset may be retrieved in turn 1 and again in turn 3, and only exported in turn 5 or even in another session. The logs have no identifier linking that export to a particular retrieval, so timing alone cannot tell which retrieval deserves the credit.

The third change is that nobody can say what caused the outcome. When the result is bad, there is no way to tell whether retrieval or generation went wrong. When the result is good, the model may have quietly made up for a failed retrieval.

Without per-component instrumentation, teams can only watch session-level signals such as retries, abandonment and task completion. The authors point out that retries alone go wrong in both directions. They fire when generation fails for reasons that have nothing to do with retrieval, and they stay silent when the model covers for retrieval. The second failure is harder to catch: retrieval went wrong but the model filled in the answer itself, the user did not retry, and in session-level signals it looks exactly like real success.

2. Method: measure where a problem is first visible

LARA's principle is to measure each component's failures at the earliest point where they can be seen. So it has three layers; the higher the layer, the closer it is to product outcomes and the more it depends on the lineage fields described later.

Layer 1 asks only about retrieval itself: did the sub-agent return the right assets in the right order? The metrics are null rate, precision@1, precision@K and NDCG@K, computed at the moment retrieval fires from server-side logs with offline relevance labels (human or LLM judge), independent of what the user or agent does afterwards. CTR serves here only as a secondary directional signal. Only this layer can catch a retrieval failure that the model has masked.

Layer 2 asks whether what came back was actually used, and whether there are signs that retrieval was wrong. This layer counts positive evidence and failure evidence separately and reports them by "retrieval role". The retrieval role is written to the log at retrieval time and says what the retrieval was for: finding something directly, finding something to look at, finding something to act on, serving as a reference input for generation, being one step in a larger task, or depending on context. The layer has four metrics:

• Genuine positive rate (GPR): the share of retrievals where, after the results were shown to the user, the user actively used a returned asset, for example by asking a follow-up about it, or by selecting, editing or exporting it.
• Agent consumption rate (ACR): the share of retrievals where the agent used a returned asset within the same request, before the user received anything, for example when a user asks in one sentence for a logo with the background removed. This also counts as positive evidence, but it is tracked separately, because it reflects only the agent's judgment; the user's later action is the independent confirmation.
• Observable retrieval failure rate (ORF): the share of retrievals followed by one of three retrieval-specific signals. The first is the user saying the retrieved object or path is wrong. The second is searching again for the same unresolved need. The third is the agent using something else when the flow should have used the returned asset. When the user keeps the asset and only modifies the edit or the generated result, that is a "downstream correction" and does not count as a retrieval failure. 1−ORF should be reported alongside GPR and ACR, but it must not be treated as satisfaction.
• Unobserved outcome rate (UOR): the share of retrievals with neither evidence of use nor a failure signal. It is its own category, counted as neither abandonment nor success.

Layer 3 asks whether the user reached their goal. In a creative tool the clearest signals are export, share and download. Task completion is defined per retrieval role, and whether the goal was reached needs an LLM judge or human labels. Only when these success events can be linked back, through the lineage fields, to the retrieval that supplied the asset does this layer become a signal about retrieval; otherwise it only reflects overall product health.

3. Read two layers together, then add four log fields

Reading Layer 1 and Layer 2 side by side tells you which component went wrong:

The last row is the most dangerous. Without Layer 1 this failure is completely invisible; with Layer 1 it shows up the moment retrieval fires, with no need to wait for user complaints to pile up.

For Layers 2 and 3 to work, the logs must be able to link actions back to retrievals. Today Adobe's production logs have no field shared by retrieval events and action events, and the authors propose four fields to close the gap. The first is a unique retrieval_id on every retrieval. The second is returned_asset_ids, the full ranked list at the moment retrieval fires, recorded before the generation layer filters or reorders it; this field alone makes Layer 1 computable, so it is the single change with the best payoff. The third is a consumed_asset_id on every action event, whether the action comes from the user or from an agent tool call. The fourth is a source_retrieval_id on the same event, pointing to the retrieval that supplied the asset; for an asset already placed in the document, the field is inherited from the placement event.

All four are additions to logging, not predictive models. source_retrieval_id sits on the action event and holds the retrieval that actually supplied the asset; once the asset is placed in the document, later actions inherit the value recorded at placement. Because it points to a retrieval rather than a session, attribution stays clear even when the same asset is retrieved again over several turns or exported in a new session. Together the four fields can also recover the rank of the asset that was actually used within the retrieval results, independent of the order finally shown to the user. That lets RAG evaluation frameworks such as RAGAs and ARES use direct attribution instead of guessing from timing.

One example ties the four fields together. A user says in chat, "Find a birthday invitation template and change the title to Xiaoming's 6th birthday." The retrieval event records retrieval_id=r1 and 20 candidates. The agent picks the fourth one and changes the title; this is an agent event carrying consumed_asset_id and source_retrieval_id=r1, and it counts toward ACR. Two days later the user exports from the editor, and the export event inherits r1, counting as a "retrieval-attributed export" in Layer 3. Without these fields, the export would land in another session and the original chat session would be counted as abandoned.

4. Production data from Adobe Express

The authors analyzed production sessions from August 2026. LARA's lineage fields are not live yet, so every "was it used" number below is inferred from timing. Session, user and query counts are withheld under Adobe's disclosure policy; the paper says only that gallery volume is roughly 8–9 times the assistant's.

The data confirms the points from Section 1 in three ways. The first is that the agent reads first. The assistant's CTR is close to zero when every conversation is counted. But 99.7% of interactions never called retrieval and never showed a single result, so they do not belong in a CTR denominator at all; counting them drives CTR toward zero. When retrieval does fire, it behaves normally: the null rate is 1.51%, and users click about 1.05 times per returned result grid. Testing directly whether the agent used the results would need ACR, which cannot be computed yet.

The second is delayed action. Assistant sessions overall (chat and in-editor entry points combined) average 2.04 prompts, and chat-entry sessions alone average 2.65, with the same asset reappearing in later turns. An export started from chat is logged under a different session_guid once the user leaves the chat interface. Session-level attribution then counts those users as abandoning, so the chat abandonment rate (59.5%) is overstated and engagement is understated.

The third is that retrieval and generation are mixed together. The reformulation inversion from the opening is the example: one behavioral signal mixes retrieval failures with tweaks to generated results, so the aggregate number says nothing. LARA splits it in two: "search reformulation", counted only on retrieval turns, and "downstream correction", counted on the other turns.

With current logs, the team can compute only the null rate and clicks per grid today. Precision@K and NDCG@K also need returned_asset_ids and a labeled sample. GPR, ACR, ORF and UOR need the other three fields plus the retrieval role, and for now can only be obtained by labeling traces by hand. In other words, the team can confirm that retrieval works when it fires, but cannot yet confirm whether the user or the agent actually used what came back.

5. A surprise: retrieval is almost never called

Gallery search is retrieval by definition, while the assistant called retrieval on only 0.31% of prompts; about 82% of assistant activity is generative editing with no corresponding search scenario at all. The authors think the bigger risk may be missed retrieval: the planner routes "help me find a birthday invitation template" to generative editing, no retrieval event is created, and Layer 1 has nothing to score. In Table 3's "prompts with retrieval" row, 0.31% is the share of prompts across the whole assistant that actually triggered retrieval; the chat-entry column shows 10.7%, which the paper marks as only directional without saying how it was counted. If the 10.7% reflects the share of prompts where users wanted retrieval, the gap between it and 0.31% is a first rough estimate of missed retrieval. The two cover different scopes (chat entry only versus the whole assistant), so it can only be rough.

The authors call planner routing Layer 0 and treat it as an extension of LARA rather than a fourth layer. It has two metrics. The misroute rate is the share of retrieval calls whose prompt was not actually in scope for retrieval; those calls should be removed from Layer 1. The missed-retrieval rate is the share of prompts with retrieval intent that did not trigger retrieval, which needs ground-truth intent labels. In a sample of 273 prompts, about 5–7% contained two or more distinct retrieval intents, so routing also has to be able to split multi-intent requests.

6. Limitations and practical value

The limitations the authors list are concrete. All the evidence comes from one deployment, and the 0.31% figure and the reformulation inversion may be specific to a creative assistant dominated by generative editing. None of the lineage fields is live, so every Layer 2 number is inferred from timing, and the retrieval roles were inferred after the fact rather than recorded at retrieval time. The chat abandonment rate is overstated by cross-session exports. The data was not validated for schema or value ranges, and synthetic traffic was removed by hand. Latency and tokens per successful task were not instrumented by stage either. LARA covers only the retrieval component; how the agent understands intent, generates queries and splits multi-intent requests needs separate instrumentation. This is a short industry paper, and the framework has not been validated across multiple deployments or in online experiments.

The authors' three lessons for practitioners are useful. First, do not use the assistant's overall CTR or reformulation rate as retrieval metrics: retrieval metrics should be counted only on turns where retrieval fired, and reformulations should be split into search reformulations and downstream corrections. Second, start by logging the full ranked list when retrieval fires; only this field makes Layer 1 computable. Third, define "the user actively used it" once and publish the definition: in Express chat, the effective action rate (the share of responses after which the user does something meaningful with the result) ranges from 7.6% to 40.5% depending on which behaviors are counted. Countable behaviors include continuing with the result in a follow-up prompt, selecting, editing and exporting; the paper gives only the two endpoints, without listing which behaviors each includes or stating the denominator. Without a shared definition the numbers cannot be compared. For teams embedding search into an agent, the paper's biggest reminder is that when retrieval is called on only 0.31% of prompts, the question is not only whether it ranks well, but whether it gets called at all.