AppMatic Tech builds production RAG pipelines with structure-aware chunking, hybrid BM25 and vector retrieval, cross-encoder reranking of the top 20 candidates, relevance-thresholded context, and a 20 to 30 query evaluation set, for healthcare documentation, SaaS knowledge bases, and construction compliance archives. AppMatic Tech is based in Ahmedabad, India.
Retrieval-Augmented Generation works straightforwardly in a prototype: embed a document corpus, retrieve the most similar chunks for a query, pass them to the LLM as context, get a reasonable answer. The prototype is convincing. The production system, faced with a real document corpus, real user queries, and real response time requirements, frequently produces worse results than the prototype suggested it would.
The reasons for this gap are specific and well-understood. AppMatic Tech has implemented RAG pipelines for healthcare documentation systems, SaaS knowledge bases, and construction safety compliance archives. The failure modes below appear consistently across production RAG deployments when the implementation decisions that prevent them are not made during the build.
Failure mode 1: How should documents be chunked for RAG?
The default chunking approach in most RAG prototypes splits documents into fixed-size chunks by character count. This is simple to implement and works adequately for documents with uniform structure. It fails for documents with section hierarchies, numbered procedures, table data, or cross-references, because it splits content at arbitrary points that do not respect the document's logical boundaries.
A clinical protocol split at 500-character intervals will frequently separate a medication dosage instruction from the condition it applies to. A safety procedure split mid-step will produce a chunk that begins in the middle of an action description without the context of what system or state it applies to. The LLM retrieves these chunks and either produces incomplete answers or generates plausible-sounding completions that fill in the missing context incorrectly.
AppMatic Tech's production RAG implementations use structure-aware chunking that respects document headings, numbered sections, and paragraph boundaries. The chunk size target is a guideline, not a hard cutoff. A section that is 200 characters under the target size is kept intact. A section that is 400 characters over the target size is split at a paragraph boundary rather than a character count.
For documents with tables, AppMatic Tech serialises table content separately from prose content and stores the table's column headers as metadata on each row chunk, so a row retrieved in isolation retains the context of what each value represents.
Failure mode 2: How do you stop RAG returning related but wrong content?
Semantic similarity retrieval returns chunks that are similar in meaning to the query. For well-formed queries against a focused corpus, this produces relevant results. For vague queries, for queries that use different terminology than the corpus, or for queries that are semantically similar to multiple unrelated topics in the corpus, semantic similarity alone returns results that are adjacent to the answer rather than the answer itself.
The production pattern that improves retrieval precision is a combination of semantic similarity and keyword filtering. AppMatic Tech's RAG pipelines use BM25 keyword search alongside vector similarity search, with a reranking model that combines both scores to produce a final ranking. This hybrid approach retrieves documents that match both the semantic intent of the query and the specific terminology the query uses, which reduces the frequency of topically adjacent but non-specific retrievals.
The reranking model is a cross-encoder that evaluates the relevance of each retrieved chunk to the query independently, rather than relying on the cosine similarity score from the initial vector search. Cross-encoder reranking is computationally more expensive than vector similarity retrieval, which is why it is applied to the top 20 candidates from the initial retrieval rather than the full corpus.
Failure mode 3: How do you evaluate RAG retrieval quality?
A RAG pipeline without an evaluation framework has no mechanism for detecting when retrieval quality degrades. Retrieval quality degrades when the corpus grows and new documents introduce terminology that the existing embedding model represents poorly, when user queries shift toward patterns the corpus does not cover, or when chunking decisions that worked for the initial corpus produce poor results for document types added later.
Without evaluation, these degradation patterns are discovered through user complaints rather than through monitoring.
AppMatic Tech builds a retrieval evaluation set during the implementation phase: a set of representative queries with known correct answers drawn from the corpus, used to measure retrieval precision and recall before launch and after any change to the pipeline. The evaluation set is maintained alongside the codebase and run automatically when the chunking strategy, embedding model, or retrieval parameters change.
The evaluation set does not need to be large to be useful. Twenty to thirty representative queries covering the range of topics in the corpus and the range of query styles the intended users produce is sufficient to detect regression in retrieval quality.
Failure mode 4: How many retrieved chunks should reach the LLM?
A common RAG implementation pattern retrieves the top K chunks and passes all of them to the LLM as context, regardless of whether all K chunks are relevant to the query. For a query with one clearly relevant answer, passing 10 chunks to the LLM introduces nine additional chunks of context that may contradict, dilute, or distract from the correct answer.
LLM performance on question answering degrades when the relevant information is surrounded by substantial irrelevant context. The model attends to content throughout the context window and produces answers that reflect the full context rather than only the relevant portion.
AppMatic Tech's production RAG pipelines use a minimum viable context approach: the reranking step filters the retrieved chunks to those above a relevance threshold before they are passed to the LLM, rather than always passing a fixed K chunks. If only two chunks pass the relevance threshold, two chunks are passed. The LLM receives clean, focused context rather than a full retrieval set.
Failure mode 5: When does RAG need a domain-specific embedding model?
General-purpose embedding models are trained on broad web text. For domain-specific corpora in healthcare, legal, financial, or technical domains, general-purpose embeddings may represent domain terminology with lower precision than domain-specific embedding models.
This does not mean domain-specific embedding models are always required. For corpora where the domain terminology is also well-represented in general web text, the performance difference between a general-purpose and a domain-specific embedding model is often small. For corpora with highly specialised technical content, clinical abbreviations, or regulatory language, the difference can be significant.
AppMatic Tech evaluates embedding model performance against the retrieval evaluation set during the implementation phase. If the evaluation results show that a domain-specific embedding model produces meaningfully higher retrieval precision than the general-purpose model, the switch is made during the build rather than as a post-launch remediation.
When is RAG the right architecture instead of MCP?
RAG is the right architecture for document retrieval from a corpus that changes infrequently: internal knowledge bases, product documentation, compliance archives, historical records. It is not the right architecture for live transactional data queries where the answer depends on the current state of a database.
For AI-native products that need both document retrieval and live data queries, AppMatic Tech combines a RAG pipeline with an MCP server. The RAG pipeline handles documentation search. The MCP server handles structured queries against live data. The LLM uses whichever is appropriate for each specific query, sometimes both in a single response.
For a detailed comparison of when to use each architecture, see What Is an MCP Server and When Does a Business Need One. For the broader production AI integration lessons that apply across both RAG and MCP implementations, see LLM Integration in Production: What Nobody Tells You Before You Build.
For a live-business-data AI example, see how AppMatic Tech built the ConstructionMCP platform, and read RAG vs MCP for live business data for when retrieval alone is not enough. Full engagement details are on the AI engineering services page and the MCP server development service page. Contact AppMatic Tech at contact@appmatictech.com.
Frequently Asked Questions
- Why do RAG systems work in demos but fail in production?
- AppMatic Tech sees five recurring causes: fixed-size chunking that splits content mid-thought, retrieval that returns related rather than relevant content, no evaluation framework so quality degrades unnoticed, too many chunks stuffed into the prompt, and a general-purpose embedding model that represents specialised terminology poorly.
- What is the best chunking strategy for RAG?
- Structure-aware chunking. AppMatic Tech splits documents at headings, numbered sections, and paragraph boundaries, treating chunk size as a guideline rather than a hard cutoff, so a dosage instruction stays with its condition. Tables are serialised separately, with column headers stored as metadata on each row chunk.
- What is hybrid search in a RAG pipeline?
- Hybrid search combines BM25 keyword matching with vector similarity, so results match both the query's meaning and its exact terminology. AppMatic Tech then reranks the top 20 candidates with a cross-encoder model that scores each chunk against the query directly, which is more precise than cosine similarity alone.
- How many retrieved chunks should be passed to the LLM?
- Only those above a relevance threshold after reranking, not a fixed top K. If two chunks pass, AppMatic Tech passes two. Extra, irrelevant chunks dilute or contradict the correct answer, because the model draws on everything in its context window.
- How do you know if a RAG pipeline is getting worse?
- With an evaluation set of 20 to 30 representative queries with known correct answers, built during implementation. AppMatic Tech runs it automatically whenever the chunking strategy, embedding model, or retrieval parameters change, so retrieval regressions surface in testing rather than through user complaints.