Three places your retrieval pipeline breaks, and what I measured when I fixed them.
Every RAG demo works. You load a PDF, ask it a question, and the answer comes back looking smart. Then real users show up with real documents, and the answers get vague, wrong, or confidently made up.
Most of the time the LLM is fine. The retrieval is naive. This week I released a one-hour RAG Master Class that walks through the fixes, along with the notebooks I used to test them. Here’s the short version.
The pipeline everyone starts with
split text → embed chunks → retrieve top-k → stuff into prompt → generate
It’s a reasonable first draft. It also breaks in three predictable places:
Chunking cuts through ideas. A chunk boundary lands mid-thought, and the retriever finds half an answer.
Mixed content embeds badly. Tables and dense sections don’t produce embeddings that match how people ask questions.
Precision and context fight each other. Small chunks match the query well but give the LLM too little to work with. Big chunks give context but retrieve poorly.
Fix 1: Choose your chunking with data
Chunking shapes everything downstream: answer quality, latency, token cost and hallucination rate. So I ran the same pipeline on the same corpus (synthetic medical case studies), changing only the chunking strategy, and scored each run with RAGAS.
The setup: 1,000-character chunks with 200 overlap, text-embedding-3-small, Chroma, gpt-4o-mini, and a 10-question generated test set.
The semantic run is covered in the video. Only its faithfulness score is shown here.
Semantic chunking is supposed to be the smart option, and on this dataset it scored worst by a wide margin. Recursive chunking splits on paragraph breaks first, then line breaks, and it kept each case study’s facts together. That was enough to win.
This is a small eval, and your data may behave differently. That’s the point: don’t pick a chunking strategy because a blog post said so. Measure it on your own documents. To narrow down what to test, ask three questions:
What does your content look like? Is it prose, structured records, code or legal clauses?
What does your embedding model handle well? Check its input limit and the chunk length it was trained on.
What will users ask? Short factual lookups and broad “explain this” questions need different chunk sizes.
Fix 2: Search the summary, answer from the source
Real documents mix prose with tables. Raw table text makes a poor embedding, so a user’s question often never matches the table that holds the answer.
Multi-representation indexing separates what you search from what you return:
Split the PDF into text and table elements. I used Unstructured’s
partition_pdf.Have an LLM write a short summary of each element.
Embed the summaries, and keep the original elements in a docstore linked by ID. LangChain’s
MultiVectorRetrieverhandles this.Match the query against the summaries, then give the LLM the full original content.
The search runs against something compact and easy to match, and the answer comes from the complete source. In the notebook I tested this on a clinical paper about immunotherapy in non-small-cell lung cancer, which is full of tables.
Fix 3: Small chunks to find it, big chunks to answer it
Parent document retrieval resolves the precision vs. context tradeoff. You index small child chunks, which match queries sharply. When one matches, you return the larger parent section it came from, so the LLM gets the surrounding context it needs.
Retrieve small. Generate big.
It’s one of the cheapest upgrades you can make to a naive pipeline, and it works especially well on long, structured documents like contracts, manuals and research papers.
Try it yourself
All the notebooks are now in one repo: fixed-size and recursive chunking with RAGAS evaluation, plus multi-representation indexing. Each one opens in Colab, and you only need to add your OpenAI key as a Colab secret.
https://agentexpert.io/get/advanced-rag
Watch the full course · 1 hour
RAG fundamentals, chunking, multi-representation indexing and parent document retrieval, with chapters so you can skip to the part you need.
One question for you: which part of your RAG pipeline breaks first in production? Is it chunking, tables, or answers that lose context? Hit reply and tell me. The most common answer will shape a future issue.
Keep adapting,
Zahir
The Adaptive Engineer