Why RAG systems fail in production (and how I debug them)

A customer asks if they can return an item after 45 days. The assistant finds a policy document, reads it correctly, and confidently tells the customer they’re covered. The document is last year’s policy. The current one caps returns at 30 days. Nothing about that interaction was a bug in the traditional sense: the model didn’t hallucinate, the search index didn’t crash, the answer was fluent and well-cited. It was just wrong, in the way that costs you a refund dispute and a support ticket.

That’s the pattern I keep running into with retrieval-augmented generation (RAG) systems once they leave the demo and hit real traffic. RAG fails in production not because retrieval is technically broken, but because the evidence it hands the model is often inapplicable, incomplete, or subtly outdated, and the model has no reliable way to notice. Fixing that requires treating retrieval, evidence quality, and answer generation as three separate things to test, not one black box to tune.

I build AI assistants at unicrew, on top of internal documentation, product catalogs, and support knowledge bases. The failure modes below aren’t theoretical. Most of them showed up in some form in a system I was responsible for, and where somebody has published a study or a production report measuring one at a larger scale than my own experience, I’ve linked it.

One quick definition, since I’ll use this term constantly: RAG means the system searches a set of documents, pulls out the passages it thinks are relevant, and hands them to a language model with instructions to answer using only that material. It’s the standard way to get an LLM to answer questions about your company’s specific, private information instead of whatever it memorized during training. If you’re at the stage of deciding how to build one rather than why yours is misbehaving, my colleague’s guide to RAG implementation covers the architecture side: chunking, retrieval tuning, and evaluation setup.

Relevant is not the same as right

Search doesn’t know about your business rules. It knows about word overlap and semantic similarity. A document about your return policy is relevant to a return question even if it describes last year’s terms, a different country’s terms, or a different product tier. The retriever did its job. The answer is still wrong.

I see this most often with anything that changes on a schedule: pricing tiers, plan-specific rules, software version differences, regional policy. A support assistant fed both v3 and v4 setup instructions will happily blend them, because from a text-similarity standpoint they’re nearly the same document. A billing assistant that doesn’t tag documents by plan tier will apply consumer-plan rules to a business account, because the language is close enough to match.

The practical fix is boring but it works: attach metadata to every document, date, version, region, customer segment, and filter or re-rank on it before the passage ever reaches the model. Decide explicitly which source wins when two documents technically apply (usually: most recent, most specific to the customer’s segment).

The versions of this I run into most often:

• Date mismatch: last year’s policy retrieved alongside (or instead of) the current one.
• Version mismatch: setup instructions for software v3 handed to a customer running v4.
• Plan mismatch: personal-plan terms applied to a business-tier account, or the reverse.
• Region mismatch: a policy written for one country surfacing for a customer in another.

Separate the first of those from a bug that looks identical from the outside and is fixed somewhere else entirely. “Both versions are in the index and the wrong one ranked first” is a ranking and metadata problem. “The new version is not in the index yet” is an ingestion problem: someone published a correction this morning and the index rebuilds overnight. The answer is stale either way, and the customer can’t tell the difference, but tagging documents harder will not fix the second one. Before you touch the retriever, check that the document you expect the system to quote is actually in the index, in its current form.

None of these are exotic edge cases. They’re the default behavior of any retriever that only matches on topic and wording, which is most retrievers out of the box.

This isn’t a hypothetical risk. RAGuard, a NeurIPS 2025 benchmark that builds its retrieval corpus out of real Reddit discussions rather than synthetic noise, found that when retrieval hands a model evidence that looks relevant but is actually misleading, every LLM-based RAG system tested did worse than the same model answering with no retrieval at all, while human annotators given the same evidence consistently did better than the models. Its domain is political fact-checking, which is about as adversarial as evidence gets, so read it as a stress test rather than as your Tuesday. The gap it measures is still the thing to worry about: the machine got confidently misled where the human noticed something was off.

Sometimes the answer was never in one piece to begin with

RAG doesn’t index documents, it indexes fragments of them. Something has to decide where one fragment ends and the next begins, and that decision is usually made by character count on ingestion day, by a person who never saw the questions users would eventually ask. When the split lands badly, the answer stops being retrievable as a unit, and nothing further down the pipeline can put it back together.

Every failure mode above assumes the answer exists somewhere as a passage a retriever could return in one piece. Often it doesn’t. A condition and its exception end up in different fragments, so the retriever returns the rule without the carve-out. A table header lands in one fragment and its rows in the next, so the numbers arrive with nothing saying what they measure. A procedure’s step 4 sits at the bottom of one fragment and step 5 at the top of the next, and whichever one comes back reads like the whole procedure.

What makes this expensive to debug is that it looks exactly like every other failure in this article. The retriever returns something topically correct. The model answers from it faithfully. The answer is half a rule. You can spend a week re-ranking and re-prompting a system whose real problem is that the sentence you needed was never sitting next to the sentence that qualifies it.

The check takes a minute, and it is step 4 of the checklist further down: find the passage that actually answers the question in the source document, and confirm it survived ingestion as one piece. If it did not, stop tuning and go fix the ingestion. Fixing it properly means chunking on the document’s own structure rather than on a character count, and overlapping fragments so a boundary can’t cleanly sever a rule from its exception. That is build-side work, and the chunking and retrieval-tuning guide goes into how to do it; what matters here is that you diagnose it as its own bug rather than mistaking it for a ranking problem.

More evidence can make a good model answer worse

There’s an intuition I run into a lot from product teams: if the assistant is wrong, the fix is a bigger model or a longer context window. That’s backwards more often than people expect. A capable model handed misleading context doesn’t usually override it with its own better judgment. It tends to trust the context, because that’s exactly what we instructed it to do.

Picture an assistant that pulls an old internal FAQ stating a service includes free delivery, when the live policy has since added a delivery charge. The model isn’t confused. It’s doing precisely what a RAG system is designed to do: answer using the retrieved material. The retrieved material is just stale.

The RAGuard results back this up directly: across the benchmark’s tests of misleading retrieval, RAG systems consistently scored below their own zero-shot (no-retrieval) baseline. To be precise about what that does and doesn’t mean: this is a benchmark specifically constructed around misleading evidence, so it’s not evidence that RAG generally underperforms a plain model. It’s evidence that when the evidence is bad, adding it to the prompt doesn’t get filtered out, it gets believed.

It isn’t the only result pointing that way. RAGChecker’s evaluation of eight RAG systems names, among its headline findings, “the tendency of faithful open-source models to blind trust on context.” Faithfulness to the retrieved passage is the behavior we ask for, and it is also the behavior that converts a bad passage into a confident wrong answer. Before reaching for a bigger model, I’d rather spend an afternoon checking what the retriever actually handed it.

A working citation is not proof of a correct answer

Citations create a specific kind of false confidence. A link that resolves, to a real, existing document, reads as trustworthy shorthand for “this is checked.” But a citation only tells you the source exists. It says nothing about whether the source is current, complete, or actually applicable to this user’s situation.

Back to the return policy example: the assistant links to last year’s policy page. The link works fine. The policy is simply out of date. A customer (or a reviewer skimming for red flags) sees a citation and reasonably assumes it settles the matter.

I treat “does the citation resolve” and “does the citation support this specific claim, for this specific customer, right now” as two separate checks, because they are. The second one is the one that actually protects you, and it’s the one most eval setups skip because it’s harder to automate.

The question gets rewritten before it’s ever searched

This is the failure mode I see teams miss most often, because it happens in a step nobody looks at. Most production assistants don’t search using the user’s raw message. They first rewrite a conversational question into a standalone search query, especially in multi-turn conversations, so the retriever has something self-contained to work with. That rewrite step can quietly drop the details that mattered.

Say a user opens with “I have a business plan in Poland,” then two messages later asks, “can I cancel without a fee?” A well-behaved retriever needs “business plan, Poland, cancellation fee.” If the rewriting step compresses that down to just “subscription cancellation fees,” the search itself can run perfectly and still retrieve the wrong country’s terms, because the query it received was never the full question.

This is one of the more underrated production risks because it’s invisible unless you specifically go look for it. I now treat it as a standard item in review: pull the exact string that was sent to the search index, side by side with the full conversation, and check whether anything, a country, a plan tier, a product name mentioned three turns back, quietly fell out.

The scale of this in a real system: NVIDIA’s internal knowledge assistant, which serves more than 30,000 employees, was analyzed by its own engineering team over a three-month post-launch window, published in 2025. Out of 495 negative feedback samples collected in that period, NVIDIA’s team attributed 3.2% specifically to query-rephrasing errors like this. That’s a measured share of negative samples, not a failure rate across all requests, but it’s a real, recurring cause in a system running at genuine enterprise scale.

Sometimes the assistant searches the wrong knowledge base entirely

Before a system can pick the right document, it often has to pick the right place to look. Many production assistants route a question to one of several knowledge bases first (HR docs, product docs, billing docs, engineering wikis) and only then search within that source. Get the routing wrong and no amount of retrieval quality inside that source will save you, because the answer was never in the collection you searched.

An employee asks how to file a travel expense claim. If the routing step sends that question to customer billing documentation instead of the internal HR/expense policy, the retriever can be flawless and still come back empty-handed or, worse, confidently pull something adjacent-sounding that doesn’t actually apply.

In that same NVIDIA analysis of 495 negative samples, routing errors accounted for 5.25%, making it the single largest identified failure category in their post-launch review, ahead of query-rephrasing errors. Again: a share of the negative samples that were analyzed, not the overall error rate of the assistant. What I take from it operationally is simpler than the number itself: log which source got selected for every query, and specifically test questions where two knowledge bases use overlapping vocabulary for genuinely different topics. That overlap is where routing quietly breaks.

An incomplete answer can look like a good one

Not every wrong answer is factually incorrect. Some are just incomplete, and incomplete answers are sneaky because they can pass every automated accuracy check while still leaving the user stuck.

A shopper asks whether a lamp works outdoors and needs a power outlet nearby. The assistant confirms it’s rated for outdoor use and stops there, leaving out the part about needing access to power. Every sentence in that answer is true. The customer still can’t decide whether the lamp works for their patio.

This is the single largest category of real user dissatisfaction I’ve seen documented at scale. Trendyol’s 2026 production analysis of roughly 150,000 real e-commerce assistant interactions broke down what dissatisfied users actually complained about: 62.3% “insufficient or incomplete,” 15.7% “unclear,” 14.2% “misleading or incorrect,” and 7.8% “irrelevant.” Read those four together rather than just the first. Factual error is the failure everyone builds guardrails against, and it sits third. Incompleteness beats it more than four to one, and even vagueness beats it.

The same study surfaced something I think is even more important for how teams should evaluate these systems day to day: human satisfaction tracked closely with the relevance of the retrieved context, running 79.6% for high-relevance retrievals versus 61.1% for low-relevance ones, a gap of 18.5 points. But automated LLM-as-a-judge scoring diverged from what actual users reported, and the paper’s own conclusion is that judge models broadly reflect satisfaction trends while missing important nuances of dissatisfaction. If your evaluation pipeline is purely an automated judge model, it may be optimizing for exactly the wrong failure mode. My practical response to this one: split multi-part questions into their required facts during test design, and explicitly check whether the retrieved material covers each part, not just whether it covers the topic.

More documents help, until they don’t

The obvious fix for an incomplete answer is to retrieve more passages. That does help, up to a point, and then it starts working against you. More documents can supply the missing fact you needed. They can also introduce an irrelevant promotion, a regional exception that doesn’t apply, or a conflicting older version of the same policy, and now the model has to arbitrate between sources instead of just reading one clean answer.

Picture a delivery-timing question where the retrieved set includes the current policy, an expired promotional offer, and an exception clause for a different region. The model has to decide which conditions actually govern this customer’s situation, and it doesn’t always decide correctly.

RAGChecker, a 2024 diagnostic framework built to evaluate RAG systems component by component rather than as a single black-box score, lists exactly this among the insights from evaluating eight RAG systems: a trade-off between retrieval improvement and noise introduction. Retrieving more context improves coverage of the facts a question needs, and it also raises what the framework measures as noise sensitivity, meaning the same change can help some answers and hurt others simultaneously. The practical implication is that “how many passages should we retrieve” isn’t a number you tune once. It’s a trade-off you measure on both axes, coverage and noise sensitivity, not just the one that’s easier to compute.

Teaching the system to say “I don’t know”

A related document showing up in search results doesn’t guarantee your knowledge base actually contains the answer. Sometimes it just contains something adjacent, and a system that’s only ever been trained or tested to produce an answer will produce one anyway.

Say your documents cover standard delivery timeframes in detail but say nothing about delivery to a specific island. A well-behaved assistant should recognize that gap rather than defaulting to the mainland timeframe and hoping it’s close enough. That’s a much harder behavior to get right than it sounds, because most RAG systems are implicitly optimized to always produce a confident-sounding answer.

UAEval4RAG, a framework from Salesforce Research presented at ACL 2025, was built specifically to test this. It defines a taxonomy of six categories of query that shouldn’t be answered at all, including underspecified requests, false premises, and questions genuinely outside the knowledge base, then synthesizes those queries for whatever knowledge base you point it at. The authors swept embedding models, retrievers, rerankers, rewriters, three LLMs and three prompting techniques across four benchmarks. Two findings came out of it, and they fit together better than they first sound.

The first is that no configuration won everywhere. Because knowledge sits differently in every knowledge base, nothing they tried was best at both answering and declining across all four datasets. So there is no setting to copy from this paper into your system.

The second is that the levers still work, hard, within any one knowledge base. The UAEval4RAG authors, at Salesforce Research, call LLM choice critical, and prompt design “equally important”: their best prompt improved performance on unanswerable queries by roughly 80%, at minimal cost to correctness on the answerable ones. Refusal, in other words, is largely a behavior you can prompt your way into. It is just not one you can tune once, publish, and assume survives the next model swap. My takeaway: if your test suite is entirely made of questions with correct answers in the knowledge base, you’ve never actually tested whether your assistant knows how to say no.

Retrieval bugs and generation bugs need different fixes

Once you’ve seen enough of these failures, a pattern emerges: some are retrieval problems (the right evidence never made it to the model) and some are generation problems (the model had the right evidence and still misused it). Treating them as the same bug wastes time. I’ve watched teams spend a sprint rewriting prompts when the actual problem was that the correct document never got past the search filter, and I’ve watched the opposite: teams rebuilding their retrieval pipeline when the model had the right passage in front of it the whole time and just misread the number in it.

RAGChecker’s whole design is built around this separation, offering distinct diagnostic metrics for the retrieval stage and the generation stage instead of one combined accuracy number. That’s the right instinct to borrow even if you’re not running their exact framework: before changing anything, figure out which side of the pipeline actually failed.

How I debug a wrong RAG answer, end to end

When an answer comes back wrong, I don’t guess. I walk the same path every time, and the order matters: it follows the evidence forward, from what the user asked to what the model finally did with what it got, so each step hands the next one a smaller problem. Skipping ahead is how you end up rewriting a prompt to fix an indexing bug.

1. Read the original question and the relevant earlier turns in the conversation, not just the final message.
2. Pull the exact string that was sent to the search index (the rewritten query, not the user’s words) and compare the two directly.
3. Check which knowledge source or index was selected, and whether that was the right one.
4. Open the source documents yourself and find the passage that should have answered this. Confirm it is in the index in its current form, and that it survived chunking as one coherent unit. If it isn’t there, or it’s split across two fragments, stop: nothing downstream can recover it.
5. Look at every passage the search step returned, before any ranking or trimming.
6. Look at what actually reached the model after ranking, re-ranking, and context-window trimming, because it’s often a smaller set than what search returned.
7. Check the dates, versions, regions, and completeness of what made it through.
8. Compare the final answer against that evidence, claim by claim, not just topic by topic.

Step 1 needs only the conversation. Step 4 needs your source documents, plus some way to see what is actually in the index, which is usually a query against the index rather than a log. Everything else needs the pipeline to have recorded what it did: the query it rewrote, the source it picked, what search returned, and what survived trimming. If none of that is recorded today, fix that before you touch anything else. It is most of the evidence path, and without it you are debugging the one step you can see.

Then I rerun the same question with manually verified, correct evidence swapped in. If that fixes the answer, the bug lives in retrieval or context selection. If the answer is still wrong with perfect evidence in front of it, the bug lives in how the model is using what it’s given, and no amount of re-indexing will fix that.

One optional extra step: run the same question with retrieval turned off entirely and see if the model still answers correctly from its own training. Read this result cautiously. A right answer with no retrieval could mean the model got lucky, or it could mean the question wasn’t actually private/proprietary information in the first place. I’d rather rerun an unstable case a few times than draw a conclusion from one lucky pass.

Run this checklist against the return-policy example from the start of this article and it takes about two minutes to find the bug: the conversation shows a straightforward 45-day return question, the search step retrieved two policy documents (one dated last year, one current), and the ranking step let the older, more verbose document rank first. Nothing in the model’s reasoning was broken. Steps 5 through 7 of the checklist find this instantly, because they’re the steps that actually look at what the retriever handed over, not just at what the model said.

This isn’t a quoted procedure from any one paper, it’s the operating checklist I’ve settled into after doing this enough times to trust it.

The regression tests I put in front of every RAG system

Debugging one bad answer fixes one bad answer. What actually prevents regressions is a standing test suite that includes questions designed to expose exactly the failure modes above, run every time the retriever, the prompt, or the underlying model changes. This is the same discipline our test automation and QA practice applies to any other part of a production system: define the expected behavior once, then keep checking it stays true.

Every failure mode in this article becomes a row. This is the starting set I build for any knowledge-base assistant, before adding whatever is specific to the client’s domain:

Test case	Failure mode it catches	What a pass looks like
A question whose answer changed, with last year’s document still in the index	Date mismatch	Answers from the current document, and names the version it used
A question answered by a document edited since the last index rebuild	Ingestion lag	The current wording, not the version the index still holds
A question whose answer is a rule plus its exception, written across a chunk boundary	Chunking	Returns the rule and the carve-out together, not the rule alone
A feature that differs between two product versions, asked by a customer on the newer one	Version mismatch	Applies the customer’s version, rather than blending the two
The same policy question asked as customers on different plan tiers	Plan mismatch	Two different answers, each correct for its tier
The same policy question asked from two countries	Region mismatch	Regional terms, not the headquarters default
Constraints stated in turn one (plan, country), then a bare follow-up three turns later	Query rewriting dropping constraints	The rewritten query still carries plan and country
A question whose vocabulary appears in two knowledge bases with different meanings	Routing error	Searches the source that actually holds the answer
A two-part question where only one part is documented	Incomplete answer	Answers the documented part and flags the gap in the other
A question with no answer anywhere in the knowledge base	Inability to refuse	Says it doesn’t know, rather than reaching for an adjacent document
A question with a false premise built into it	Accepting the premise	Corrects the premise instead of answering around it
Any failed case, re-run with correct evidence pasted in by hand	Retrieval versus generation	Tells you which half of the pipeline to fix

For every row, write down the expected answer (or the expected acknowledgment that information is missing) before you run the test, and keep every case that has ever failed in the suite permanently. RAG regressions have a habit of quietly coming back after an unrelated prompt tweak.

The takeaway

Most of the RAG failures I run into aren’t dramatic. Nothing crashes. The model doesn’t spout nonsense. It reads a plausible document and gives a plausible-sounding answer that happens to be wrong, outdated, or half the story. That’s exactly what makes it dangerous in production: it looks like success on a dashboard and feels like a broken promise to the customer who acted on it.

The fix isn’t a bigger model or a longer context window. It’s inspecting the actual evidence path, what was searched, what was found, what survived trimming, and what the model did with it, before deciding whether the bug lives in retrieval or in generation.

If you’re earlier in the process and still deciding whether RAG is even the right architecture for what you’re building, it’s worth reading up on how it compares to agentic AI approaches before committing.

This article covers failure patterns I’ve encountered building production RAG systems, cross-referenced against publicly available research and production reports from RAGuard (NeurIPS 2025), NVIDIA, Trendyol, RAGChecker (Amazon Science), and UAEval4RAG (Salesforce Research, ACL 2025). Specific examples used to illustrate each concept (the return policy, the lamp, the travel expense claim) are illustrations I constructed to make the pattern concrete; they are not incidents reported by unicrew clients or by the cited researchers.

If you’d rather not do this part yourself

Everything above is work you can do in-house, and if you have someone who can run that checklist against twenty bad answers, you should. The reason teams call us is usually that nobody has the twenty bad answers in one place yet, or that the person who would do it is the same person shipping the roadmap.

What we do first is not a rebuild. Bring a set of answers your users complained about, ideally with the conversations they came from, and we walk each one down the evidence path in this article: what was asked, what was searched, which source was chosen, what came back, what survived trimming, and where the answer parted company with the evidence. What comes out is a list of your failure modes ranked by how often they fire, a verdict on which side of the pipeline each one lives on, and the regression suite from the section above, populated with your cases, so the next change you make can’t quietly undo the fix. That last part matters more than the diagnosis: a fix you can’t re-test is a fix with a shelf life.

From there the work splits by what the list actually says. If the failures are architectural, that’s our AI and ML development team. If they’re evaluation and regression problems, it’s our AI QA and evaluation practice, which builds the datasets, graders and CI gates that keep this from recurring. Most engagements start within two to four weeks, and we work on time and materials, fixed price, or as an extension of your team, whichever fits how you’d rather run it.

If that sounds useful, send us a few of the bad answers and we’ll look at them together on a call. You’ll get a straight read on which failure modes you’re dealing with whether or not you hire us.