Most RAG failures are retrieval failures
When a RAG system answers badly, teams reach for a bigger model. In my experience the model is rarely the problem. If the right passage never made it into the context window, no model can save the answer, and a stronger one will simply produce a more articulate wrong answer. Before touching generation, measure retrieval on its own: for a set of real questions, is the correct source document in the top five results?
That single metric reframes the whole project. It is measurable without a model in the loop, it improves through unglamorous work on chunking and indexing, and it caps everything downstream. I have seen answer quality jump substantially from retrieval work alone, with the generation prompt left completely untouched.
Chunking is the highest-leverage decision
Fixed-size chunking is the default and it is usually wrong. Splitting every 500 tokens cuts sentences in half, separates a heading from the table it describes, and orphans the clause that gave a paragraph its meaning. Chunking along the document's own structure — headings, sections, list boundaries — preserves the semantic unit, and semantic units are what retrieval is trying to find.
I also attach context to every chunk rather than storing it bare. Each one carries its document title, its heading path, and a short summary of the section it came from. This costs a little storage and pays for itself immediately, because a chunk that reads 'it dropped by 40 percent' is meaningless in isolation but perfectly retrievable when it carries the heading that says what 'it' is.
Hybrid search beats pure vector search
Embeddings capture meaning and miss specifics. Ask about error code E4021 or a product SKU and semantic similarity will happily return passages about error handling in general while ignoring the one document containing that exact string. Keyword search has the opposite failure: it nails the identifier and misses the paraphrase.
Running both and fusing the rankings fixes the blind spots in each. This matters most for exactly the queries enterprise users actually type, which are full of part numbers, policy references, and internal jargon. If you only implement one thing from this article beyond chunking, make it hybrid retrieval.
Re-ranking earns its latency
Retrieval optimises for recall: cast a wide net, get the right passage somewhere in the top twenty. Generation needs precision: put the best three or four in the context. A cross-encoder re-ranker bridges that gap by scoring each candidate against the query properly rather than through pre-computed vector distance, and it consistently promotes the genuinely relevant passage over the merely topical one.
It adds latency, and it is worth it. The alternative is stuffing twenty mediocre chunks into the context window, which costs more tokens, increases the chance the model latches onto an irrelevant passage, and degrades answers. Retrieve broadly, re-rank hard, send few.
Citations are an architectural requirement
In any enterprise setting the answer is not the deliverable — the answer plus its source is. Every chunk keeps a stable reference back to its document and location, and the generation prompt requires the model to cite which chunks it used. Users who can check the source trust the system; users who cannot, quite reasonably, do not.
Citations are also your debugging tool. When an answer is wrong you can immediately see whether retrieval surfaced the wrong passage or the model misread the right one. Those are completely different bugs with completely different fixes, and without citations you cannot tell them apart.
Freshness, permissions and the boring parts
Documents change. A RAG index built once and never updated becomes confidently wrong, which is worse than being unavailable. I track a content hash per source document and re-embed only what changed, which keeps re-indexing cheap enough to run continuously instead of as a scary quarterly event.
Permissions are the requirement teams discover too late. If the corpus spans departments, retrieval must filter by the requesting user's access before ranking, not after. Filtering after ranking leaks information through the shape of the results, and retrofitting per-user filtering into a vector index that was designed without it is a rebuild, not a patch.
Key takeaways
- Measure retrieval separately: is the right document in the top five? Answer quality cannot exceed that ceiling
- Chunk along document structure, not fixed token counts, and attach title and heading path to every chunk
- Use hybrid keyword-plus-vector retrieval — pure embeddings miss error codes, SKUs and internal jargon
- Retrieve broadly, re-rank with a cross-encoder, then send only three or four passages to the model
- Require citations: they build user trust and tell you whether a bad answer was a retrieval or generation failure
- Re-embed on content-hash change, and enforce per-user permission filters before ranking, not after
Conclusion
A production RAG system is 80 percent search engineering and 20 percent prompting. Spend your time on chunking, hybrid retrieval, re-ranking and permissions, and the generation layer becomes almost boring — which, for a system people rely on at work, is exactly what you want.
Enjoyed this article?
Vivek Kumar Singh
Technical Expert · Full Stack Cloud Engineer · Tokyo, Japan
