From Logs to Faults: Reference-Free Fault Localization for Retrieval-Augmented Generation Pipelines

RAGLog reads only the API logs each run of a RAG pipeline leaves behind, with no reference answers and no code changes, and points out which stage of the run went wrong. This matters because a pipeline can go wrong in the middle and still give the right final answer. Across natural runs of five open-source pipelines, 37% had at least one faulty stage, and more than a third of those still answered correctly, so checking the answer alone misses them. It checks each stage in turn: whether the rewritten query dropped names, numbers or dates from the question, whether any retrieved passage actually contains the answer, whether reranking pushed the passage with the answer out, and whether the answer is backed by the context. A classifier trained for each stage then decides whether that stage is faulty. On naturally occurring faults it finds the faulty stage with an F1 of 0.73, against 0.58 for Claude Code. For debugging RAG, the logs alone are enough for a first pass.

This is worth reading for any team running RAG, less for the F1 gain than for turning debugging into a fixed set of checks. Each run makes exactly 5 parallel LLM calls, about US$0.005 per question on natural runs, so the cost is predictable. Its output fits a debugging queue better than an alert, for reasons shown by the false-positive rates below. All F1 scores here are stage-level F1 pooled over every stage of every run.

Where a run can go wrong

The pipelines in the paper have five stages: query rewriting, turning the query into an embedding, vector retrieval, reranking and answer generation. Rewriting and reranking are optional. Any stage can fail, but a failure does not always show up in the final answer.

An illustrative example. A user asks for a company's operating cash flow in 2023:

The paper's numbers show the same. Among injected faults that were confirmed, retriever and reranker faults almost always led to a wrong answer (49 of 50 and 26 of 29 cases), while many rewriter and embedder faults left the answer correct. An evaluation that only looks at answers catches most retriever and reranker faults but misses many rewriter and embedder faults.

How RAGLog works

The first step is putting the logs into one format. Pipelines call all kinds of APIs, so RAGLog can record the pipeline's outbound HTTP traffic with its own logging proxy, or read existing OpenTelemetry traces. It first uses rules to decide which stage each call belongs to, then groups calls into runs by question, each run ending with the generator call that produces the answer. Each run becomes one record in a common format holding the inputs and outputs of every stage; the field mapping is generated only once per pipeline. Some pipelines query an in-process vector database and leave no retriever call in the logs; RAGLog then rebuilds the retrieved passages from the reranker's input. On the five evaluated pipelines, the rules classified all 80,778 calls correctly without an LLM; only calls to unfamiliar APIs are sent to an LLM.

The second step checks each stage, with three kinds of checks. Rule checks are the cheapest, for example counting how many names, numbers and dates from the question disappeared after rewriting; for the embedding stage, they compare the text sent for embedding with the original question and with the rewritten queries, looking for lost names, words one or two edits off, or added unrelated words. Vector-similarity checks use a small local embedding model to compute similarities, for example how close the passage most similar to the question is and where it ranks, and whether the reranked order agrees with the similarity order. LLM checks are the most expensive, for example asking a model for a 0 to 100 confidence for each typical fault of a stage, including retrieval faults such as no passage answering the question or passages covering only part of it. High similarity does not mean a passage contains the answer; this step is what judges that. LLM checks also verify that the claims in the answer are supported by the context. An evidence flow check goes further. The LLM first lists, from the question alone, the pieces of information needed to answer it. For each piece, it judges whether it is in the passages sent to the generator, only in passages the reranker dropped, or nowhere. It then judges whether the answer includes it, omits it or states something different. These judgments are first turned into per-stage numbers, such as the share of needed information missing from all retrieved passages, and passed to the classifiers with the other check results. After the classifiers decide, one fixed rule applies: if retrieval is flagged but the evidence flow check finds the needed information only in passages the reranker dropped, the retrieval flag is cleared and the fault goes to the reranker. These 5 LLM checks run in parallel. Hoot says▾

Hoot saysEvidence flow check: where the information got stuck (illustrative)Close ✕

Follow each needed piece of information and see where it stops. If retrieval found it but the reranker dropped it, the fault goes to the reranker.

‹1/2›

Illustrative example. The evidence flow check first finds where the information stops. Its results go to the classifiers first. Only clearing the retrieval flag and blaming the reranker is a fixed rule. Source: paper §3.4.1 evidence flow check and §3.4.2 ensemble decision; numbers are illustrative

The third step feeds the measurements of the whole run to a classifier for each stage. It reads the whole run rather than one stage because a fault often leaves traces in other stages; if the rewrite drops the year, the passages retrieved later will not match the year either. In the ablation, though, using only the stage's own measurements lowers F1 by just 0.01 to 0.02, which is not significant. Each stage gets a fault probability from three models (logistic regression, gradient-boosted trees and extremely randomized trees), averaged with weights of 1/3, 1/2 and 1/6 and compared with that stage's threshold. The classifiers are trained once, offline, so the checks only produce numbers, and which combinations count as a fault is decided by the trained classifiers.

How it was evaluated

The authors ran three QA datasets through five open-source pipelines and built a dataset called RAGLog Faults: 3,134 runs, each stage labeled faulty or not. The labels come from two LLM judges from different providers, which see the reference answer and the evidence passages; a third model settles their disagreements. The labels are used only to train the classifiers offline and to evaluate; RAGLog never sees them when it runs, only the logs. The test set has 1,500 natural runs, meaning nothing was changed and faults happened on their own, plus a set of runs with injected faults, such as deleting the passage with the answer from the retrieved set or moving it just below the rerank cutoff.

The baselines are two coding agents, Claude Code and Codex, given the same logs and fault definitions and asked to find the problems themselves. This mimics how a developer without a dedicated tool would debug.

What the results show

Injected faults may be easier to spot in the logs, and all three score close F1; the gap is on natural faults. Most of what the two agents report is correct, with precision of 0.80 and 0.76, but they find fewer than half of the faulty stages. RAGLog finds 78%, at the price of a false-positive rate of 4.6%, against about 1.4% for the agents. For a debugging tool this trade is worth it: reading a few false alarms is cheaper than missing faults.

The clearest evidence comes from runs that answered correctly but had a fault in the middle. Of 202 such runs, RAGLog identified at least one faulty stage correctly in 144, or 71%; Claude Code managed 29% and Codex 16%. On runs with wrong answers the gap is much smaller (85% against 63% and 58%). The paper takes this to suggest that the agents may rely more on whether the answer visibly fails, while RAGLog checks every stage regardless of the answer.

RAGLog and Claude Code use the same underlying model here, so the gap cannot be put down to the model alone. The ablation points to the check design as well: removing the LLM checks lowers F1 on natural faults by 0.275, the largest effect of the three kinds of checks.

Limitations

• Embedder faults are rare, only 10 in the natural runs, and the default RAGLog found none of them; neither did the agents. Injected embedder faults it does catch (F1 0.94), so what is missing is samples and detection of this kind of fault in real runs.
• At the run level RAGLog's false-positive rate is 13.6%, against 5.4% and 4.0% for the agents. As an alerting system it would be noisy; it fits better as a filter during debugging.
• The test labels come from LLM judges and were not checked against human annotation. The two judges agree well overall (Cohen's κ 0.72), but only 0.27 on the embedder and 0.47 on the rewriter, so scores for those two stages deserve a discount.
• All results come from open-source pipelines, benchmark questions and harmonized settings; the authors state plainly that production deployment is not yet validated.
• Only 22% of generator injections actually produced a fault, because the pipelines absorbed many disturbances at later stages. The injected test set therefore covers later stages thinly.

Trying it on your own system

1. Pick one RAG pipeline, route its outbound calls through the logging proxy in a staging environment (or feed existing OpenTelemetry traces), and confirm that inputs and outputs line up for rewrite, embedding, retrieval, rerank and generation.
2. Run the default configuration for a week, look at which stages get flagged, and start with the stage that fails most often. To save money, use the paper's economy mode, which drops the most expensive check, the statement-by-statement check of the answer: cost falls by about 20% and F1 changes by 0.01 at most. Do not drop all LLM checks; in the ablation that hurt F1 the most.
3. The classifiers work on new pipelines without retraining, but on pipelines unseen in training F1 drops from 0.73 to 0.67. If you have hand-labeled runs, RAGLog can suggest per-stage thresholds from them, keeping false positives at a level you accept.
4. Finally compare cost with the extra faults found, decide between checking every run and sampling, and send results to a debugging queue rather than straight to alerts.

Verdict