RAG in Production: What Nobody Tells You

The demo always works. You pick a handful of well-formatted documents, ask exactly the question your data was built to answer, and the LLM gives a perfect, confident response. Then you go to production — real users, messy data, unexpected queries — and the wheels come off. RAG is not an AI problem. It is a systems engineering problem.

This is what we see repeatedly when helping teams ship AI features: the prototype took a week, the production-grade system took three months, and most of the work had nothing to do with the LLM.

This article covers the decisions that actually determine whether your RAG system works in production — chunking, retrieval quality, hallucination control, latency, cost, and observability.

1. Chunking Is Where Most RAG Systems Break

Chunking — splitting documents into retrievable pieces — sounds trivial. It is not. The majority of retrieval failures we debug trace back to bad chunking decisions made early in the project.

Fixed-size chunks lose context

Splitting every 512 tokens regardless of document structure is the default and often the worst choice. A chunk that starts mid-sentence, cuts a table in half, or splits a code example from its explanation will retrieve correctly but answer incorrectly. The LLM gets semantically valid text that is contextually useless.

What to do instead

• Respect document structure: split on headings, paragraphs, and sections — not character counts
• Add overlap: a 10–20% token overlap between adjacent chunks preserves context across boundaries
• Store metadata with every chunk: source document, section title, page number, creation date — this enables filtering before retrieval
• Match chunk size to your embedding model: most models are trained on sequences of 256–512 tokens; longer chunks degrade embedding quality
• Consider hierarchical chunking: store a small chunk for retrieval but pass the parent section to the LLM — you retrieve precisely, the model sees enough context

2. Your Embedding Model Is a Long-Term Commitment

Switching embedding models means re-embedding your entire document corpus. If you have a million documents, that is an expensive, time-consuming operation. Choose deliberately from the start.

• General-purpose vs domain-specific: OpenAI's text-embedding-3-large works well broadly. For legal, medical, or technical domains, a domain-tuned model will outperform it on retrieval quality.
• Dimension count matters operationally: higher dimensions mean better semantic precision but larger vector indexes, slower similarity search, and higher storage costs. You often do not need 3072 dimensions for a customer support bot.
• Benchmark on your actual data: embedding model leaderboards are useful but not your benchmark. Test on representative queries from your domain before committing.

3. Retrieval Quality Matters More Than the LLM

Garbage in, garbage out. No LLM can synthesize a correct answer from irrelevant retrieved chunks. Retrieval quality is the highest-leverage investment in a RAG system.

Top-k alone is not enough

Retrieving the five most similar chunks by cosine distance gives you five chunks that are semantically close to the query — not necessarily the five most useful for answering it. Two improvements matter most:

• Add a reranker: retrieve a larger candidate set (top-20 or top-50), then use a cross-encoder reranker like Cohere Rerank or a local model to score relevance more precisely. Pass only the top-3 to the LLM. This consistently improves answer quality with minimal latency overhead.
• Hybrid search: combine dense vector search with sparse keyword search (BM25). Dense search finds semantically similar content; sparse search finds exact terminology. For technical or domain-specific queries, hybrid search reliably outperforms either alone.

Also filter by metadata before vector search where possible. If a user is querying documents from a specific product line or date range, apply that filter at the database layer. It is faster and more precise than relying on the embedding alone.

4. Hallucination Does Not Go Away — You Manage It

RAG reduces hallucination relative to a prompt-only LLM, but it does not eliminate it. The model can still confabulate when retrieved chunks are ambiguous, incomplete, or slightly off-topic.

• Instruct the model explicitly: if the context does not contain the answer, say so. Do not invent. This sounds obvious — most teams skip it.
• Set retrieval quality thresholds: if the highest similarity score in your retrieval results is below a threshold (e.g. 0.65 cosine similarity), surface a fallback response rather than sending low-confidence chunks to the LLM.
• Grounding checks: for high-stakes use cases, add a post-generation step that verifies key claims in the response against the retrieved source chunks. Tools like Trulens and RAGAS can automate this.
• Cite your sources: forcing the LLM to cite the chunk it drew from exposes hallucinations naturally — it cannot cite a source for a fact it invented.

Note: numbers like 0.65 cosine similarity and 256–512 token chunk sizes are illustrative starting points, not universal constants. The right values depend on your embedding model, domain, and query distribution. Treat them as a starting hypothesis and calibrate against your own retrieval quality metrics.

5. Latency Is a Product Problem

A RAG pipeline that takes 8 seconds to respond will not be used, regardless of answer quality. Latency budgets matter from the start.

• Stream responses: start rendering tokens as they arrive. Time-to-first-token matters more than total latency for perceived performance.
• Cache embeddings for repeated queries: if users frequently ask the same questions, caching query embeddings at the application layer eliminates redundant inference calls.
• Run retrieval and any preprocessing in parallel where possible — query embedding and metadata filtering can often happen concurrently.
• Choose your vector database for your query volume: pgvector is excellent for moderate scale; Weaviate, Qdrant, and Pinecone handle higher throughput with lower latency at the cost of operational complexity.
• Pre-compute embeddings for predictable queries: if your product has a known set of high-frequency questions, pre-embed them and cache the results.

6. Cost Compounds at Scale

RAG costs accumulate in three places that are easy to underestimate.

• Embedding costs at ingestion: re-embedding your corpus whenever documents change adds up. Prefer incremental indexing — embed only updated or new documents, not everything.
• LLM token costs with context: passing five chunks of 512 tokens plus a system prompt into a model that bills per token on input is expensive at volume. Trim context aggressively — pass only what the reranker scored as highly relevant.
• Vector database hosting: managed vector databases charge for storage, queries, and often writes separately. Know your query volume before choosing a tier.

The most effective cost lever is reducing LLM calls by improving retrieval precision. If your retrieval is good enough that the first result is almost always sufficient, you can experiment with smaller context windows and cheaper models for the generation step.

7. Observability Is Non-Negotiable

You cannot improve what you cannot see. RAG systems require domain-specific observability, not just infrastructure metrics.

• Log every step: the query, retrieved chunks and their scores, the reranked order, the prompt sent to the LLM, and the final response. You will need this to debug bad answers.
• Track retrieval hit rate: what percentage of queries return at least one chunk above your quality threshold? A dropping hit rate means your index is stale or your data coverage has gaps.
• Collect user feedback: a simple thumbs-up/down on responses gives you ground truth that automated metrics cannot. Cluster negative feedback — patterns emerge quickly.
• Monitor latency at each stage: embedding time, retrieval time, reranking time, and LLM time separately. Bottlenecks shift as your system evolves.

Tools like LangSmith, LangFuse, and Arize Phoenix provide out-of-the-box RAG tracing. Start with one early. Adding observability after the fact means re-instrumenting code you already shipped.

The Bottom Line

RAG is a retrieval system with an LLM attached to it. Most of the work — and most of the failures — live in the retrieval half, not the generation half. Getting the LLM right is the easy part.

The teams that ship reliable RAG in production are not the ones that picked the best model. They are the ones that treated chunking as an engineering problem, invested in retrieval quality before the LLM, built observability in from day one, and managed latency and cost as first-class product constraints.

The demo is not the product. The production system is the product. Build for that from the start.