AI
What actually breaks when RAG systems reach production
11 July 2026 · 11 min read · By Jitu Solanki
The Demo Worked Fine
The demo usually goes like this: A small corpus of PDFs is loaded into a vector database. A few hundred text chunks are embedded. A developer asks, "What is our refund policy?" The system retrieves two relevant paragraphs, feeds them to an LLM, and returns a clean, accurate answer.
It feels like the problem is solved. Then the system reaches production.
Suddenly, users ask things the demo never covered. Documents change daily. Some users should not see certain files. Latency matters. Costs start climbing. The same LLM that looked brilliant in the demo starts producing answers that are subtly wrong, outdated, or confusing.
Most of these failures are not caused by the LLM. They come from the system around it: ingestion, chunking, retrieval, metadata, permissions, evaluation, observability, and operations.
Why Prototypes Are Deceptive
A prototype demonstrates that RAG can work, not that it will remain reliable.
Retrieval Quality Breaks First
The most common failure is not the language model. If the wrong context is retrieved, even a powerful LLM will generate a wrong or incomplete answer.
- Chunks too large: relevant content is buried.
- Chunks too small: meaning is lost without surrounding context.
- Bad metadata: retrieval cannot filter by date or owner.
- Similar but irrelevant: vector search returns near neighbors that are not actually useful.
Retrieval quality is the foundation. If it breaks, nothing downstream can fully recover.
Chunking Becomes a Real Engineering Problem
In a demo, fixed-size chunking often works. In production, it becomes a main failure mode. Naive chunking cuts through section boundaries, splits table rows, or separates definitions from rules.
Your Documents Are Messier Than You Thought
Production collections have duplicate titles, outdated versions, broken PDFs, OCR errors, and missing metadata. A retrieval system can only search what it has indexed. If the index contains duplicate or bad content, the LLM will build answers on it.
Stale Data and Re-Indexing Problems
Old chunks remain in the database while new versions are indexed alongside them. Users receive answers generated from outdated policies.
Solution: Use stable document IDs, explicit deletion strategies, and periodic full re-indexing when embedding models change.
Access Control and Data Leakage
In a prototype, everyone searches everything. In production, Tenant A might see Tenant B’s support articles. Authorization must be enforced at retrieval time (e.g., via metadata filters in the vector DB), not just in the frontend.
Vector Search Alone Is Often Not Enough
Pure vector search works poorly for exact terms like product codes (AF-101-XXL), error messages (ECONNREFUSED), or legal citations. You need Hybrid Retrieval—merging BM25 lexical search with vector search.
Reranking Changes the Game
Vector similarity gives a rough ranking. A reranker compares the user query against each retrieved chunk more carefully and reorders them, pushing the exact answer to the top and reducing irrelevant context.
The LLM Starts Hallucinating
RAG reduces hallucination; it does not eliminate it. If retrieved context is incomplete, conflicts, or is diluted by noise, the LLM will hallucinate confidently. Strong prompting and good retrieval are required.
Context Windows Become Expensive
When retrieval is weak, teams add more chunks to the prompt. This increases token costs, latency, and dilutes the model's attention. Better retrieval is always a better investment than a larger context window.
Latency Suddenly Matters
Users experience end-to-end latency: embedding generation + vector DB query + hybrid search + reranking + prompt assembly + LLM generation. You must cache queries, parallelize retrieval, and stream LLM tokens.
Costs Become the Hidden Failure
A RAG request looks cheap until multiplied by thousands of users. Cost drivers include embedding generation, vector DB compute, reranking, and multi-turn conversations accumulating history.
Evaluation Is Much Harder
Manual testing with 20 questions fails in production. You must track Retrieval Metrics (Precision@k, Recall@k) and Answer Metrics (Faithfulness, Groundedness) against real user traffic.
Observability Is Missing
When an answer is wrong, you need to know why. Did the right document get retrieved? Did the reranker fail? Was the LLM context diluted? Without detailed traces and logs, debugging is pure guesswork.
Query Understanding Becomes Important
Users don't ask questions the way documents are written. A small query understanding layer (query rewriting, synonym expansion, intent detection) can dramatically improve retrieval quality.
Multi-Turn Conversations Create New Problems
Follow-up questions are often ambiguous ("What about for digital products?"). The system retrieves based only on the latest message and fails. You must rewrite follow-up questions using conversation history.
Production Ingestion Is a Pipeline
A simple script to load files and embed them is not enough. Production needs change tracking, OCR, retry handling, dead-letter queues, and alerting for silent failures. It’s a data pipeline, not a script.
Failure Handling Gets Real
External services fail. You need retries with exponential backoff, circuit breakers, graceful degradation, and timeouts on every external call.
Security Goes Beyond Access Control
Prompt injection from retrieved documents can hijack the LLM. If an uploaded document instructs the model to ignore previous rules, your system must isolate prompts and handle retrieved text carefully.
Architecture Changes as Usage Grows
A prototype is a single script. Growing production requires separate ingestion and query services, multiple indexes by tenant, and async processing queues. Isolate failures and scale components independently.
RAG Is a System, Not a Feature
The hard part of RAG in production is rarely connecting an LLM to a vector database.
The hard part is making retrieval reliable, data fresh, permissions correct, performance predictable, costs manageable, and failures observable.
Production Readiness Checklist
Conclusion
A RAG demo works because the conditions are artificial: small data, clean files, simple queries, no permissions, no traffic, no cost pressure.
Production breaks that illusion.
If you are building a production RAG system, treat it as a full system. The LLM is only one component. The surrounding infrastructure decides whether the system works at scale. And when the answer is wrong, the first place to look is rarely the model. It is usually the context you gave it.
Have a project in mind?
Tell us what you are building. You will get a technical point of view, an indicative budget and a delivery plan within two working days.