Library
Designing Large Language Model Applications · 9 of 12
Designing Large Language Model Applications
ai CRITICAL

Retrieval-Augmented Generation Pipeline

Designing Large Language Model Applications Suhas Pai
rag retrieval reranking hyde long-context

Key Principle

RAG is the book's primary prescription for grounding LLMs in reliable external knowledge. The pipeline — rewrite-retrieve-rerank-refine-insert-generate — is a systems engineering framework where only retrieve and generate are mandatory. The central claim: retrieval quality is the binding constraint — everything else is secondary. "If the retrieval process fails to extract suitable candidate text, the LLM's powerful capabilities will all be for nothing" (Chapter 12).

Why This Matters

RAG exists because scaling model size cannot solve the long-tail memorization problem economically. LLMs need many training samples to memorize a fact (sample-inefficiency), and the relationship is log-linear — competitive long-tail performance would require quadrillions of parameters (Kandpal et al.). BLOOM-176B achieves only 25% QA accuracy when relevant documents appear 10 times in pre-training data vs. 55% at 10,000 occurrences (Chapter 12).

This is a concrete instance of "engineering around limitations rather than waiting for perfect models."

Good Examples

  • Hybrid search (BM25 + embeddings) is standard practice and often sufficient. BM25 remains "unreasonably effective" — paired with query/document rewriting, it suffices for many applications (Chapter 12).
  • HyDE (Hypothetical Document Embeddings): Queries and documents live in different embedding subspaces. HyDE bridges this by having the LLM generate a hypothetical answer document — factually wrong but semantically aligned — and using it as the retrieval query (Chapter 12).
  • Chain-of-Note (CoN): The LLM generates notes about each retrieved document's relevance before answering, enabling "I don't know" responses when retrieval quality is low (Chapter 12).
  • RAG as few-shot selection (LLM-R): Retrieve examples dynamically based on query similarity, then fine-tune the retriever using LLM feedback on log-probability of correct output. This closes the loop between retrieval and generation quality (Chapter 12).

Counterpoints

  • When RAG hurts: For popular entities, the LLM's parametric memory may be more reliable than retrieved documents, because retrieval can introduce irrelevant or contradictory information (Mallen et al.). "Dynamic retrieval is mostly useful when you are using very large LLMs. For smaller models (7B or below), it is almost always beneficial to prefer using RAG" (Chapter 12).
  • HyDE failure mode: Smaller LLMs produce lower-quality hypothetical documents, amplifying topic drift risk (Chapter 12).
  • Long-context is not a replacement: "Forgoing retrieval completely in favor of using long-context models is akin to buying a laptop and storing all your files in RAM instead of disk" (Chapter 12). Long-context reduces the need for reranking/refining but does not eliminate retrieval.
  • Document ordering matters: LLMs recall beginning and end of context better than the middle (Liu et al.) — critical information placed mid-context is effectively invisible (Chapter 12).

Key Quotes

"Retrieval becomes the limiting factor of the pipeline. If the retrieval process fails to extract suitable candidate text, the LLM's powerful capabilities will all be for nothing." — Suhas Pai, Chapter 12

"Forgoing retrieval completely in favor of using long-context models is akin to buying a laptop and storing all your files in RAM instead of disk." — Suhas Pai, Chapter 12

Rules of Thumb

  • Invest in retrieval quality (chunking, embeddings, reranking) before optimizing generation
  • Start with hybrid search (BM25 + embeddings) — it's often sufficient
  • Use HyDE for vocabulary mismatch problems, but watch for topic drift with small LLMs
  • Place critical retrieved content at the beginning or end of context, never the middle
  • RAG consistently outperforms fine-tuning for knowledge-intensive tasks, but they're complementary
  • For smaller models (7B or below), almost always prefer RAG over parametric memory

Related References