Library
Designing Large Language Model Applications
ai

Designing Large Language Model Applications

Suhas Pai 2025 12 references

Suhas Pai's holistic framework for building production-grade LLM applications — covering the full stack from pre-training data through tokenization, architecture, evaluation, RAG, agents, and multi-model system design.

llm production-engineering rag agents system-design machine-learning

Overview

The Core Framework

  • The central problem is the prototype-to-production gap: demos are trivially easy; production systems require holistic understanding of every LLM ingredient
  • Data quality > model selection > prompt engineering — in order of impact
  • LLM limitations (hallucination, reasoning failures, bias) are design constraints, not bugs — engineer around them
  • Production deployment is a systems engineering problem: routers, cascades, guardrails, verifiers, orchestration
  • Default to explicit interaction paradigms over autonomous agents for anything mission-critical

Quick Lookup

Situation Do This Avoid This
Starting a new LLM project Clean data, check tokenization, build internal benchmarks Chasing leaderboard rankings or picking the biggest model
Model behaving oddly on specific inputs Check tokenization of those inputs first Debugging at prompt or architecture level only
Need structured output Use constrained decoding (Jsonformer, LMQL) Hoping the model complies via prompting alone
Building a knowledge-grounded app Implement RAG with hybrid search (BM25 + embeddings) Relying on parametric memory or dumping everything into long context
Need high reliability Generate n > 1 completions, use self-consistency voting Single generation with no verification
Optimizing cost at scale Use LLM cascades (smallest model first, escalate on low confidence) Defaulting to the largest model for all requests
Building an agent Use explicit (pre-programmed) tool orchestration Autonomous agents for mission-critical applications

The Key Insight

"Plenty of software frameworks have emerged that enable rapid prototype development of LLM applications. However, advancing from prototypes to production-grade applications is a road much less traveled, and is still a very challenging task." — Suhas Pai, Preface

References

1

LLM Agents, Tools, and Interaction Paradigms

Three interaction paradigms define the spectrum of LLM agency: passive (LLM receives context unknowingly), explicit (pre-programmed tool use), and…

10
agents tools interaction-paradigms +2
CRITICAL
2

The Prototype-to-Production Gap

The central problem in applied LLM engineering is the prototype-to-production gap: building a working LLM demo is trivially easy, but advancing it to…

Preface, 1
production systems-engineering llm-limitations
CRITICAL
3

Embeddings, Document Parsing, and Semantic Search

Embeddings are the infrastructure layer beneath RAG. The most underappreciated and most impactful step in the entire pipeline is document parsing —…

11
embeddings hard-negatives chunking +2
HIGH
4

Implementation Playbook

Building production LLM applications requires a systematic approach that prioritizes data quality, treats model limitations as design constraints,…

1, 5, 10, 12, 13
implementation production best-practices +1
HIGH
5

Inference Optimization Taxonomy

Inference optimization makes LLM deployment economically and physically viable. Techniques are organized by the bottleneck they attack: compute (K-V…

9
inference kv-cache distillation +2
HIGH
6

Selecting and Evaluating LLMs

Model selection is overrated relative to data quality, benchmark evaluation is broken, and the real differentiator of open source models is not cost…

5
evaluation benchmarks logits +2
HIGH
7

Multi-LLM System Architecture

Production LLM applications should not be treated as standalone LLM components. They require multi-model systems with routers, cascades, guardrails,…

13
cascades routers confidence +2
CRITICAL
8

Pre-Training Data: The Most Important Ingredient

Pre-training data is the most consequential and underappreciated LLM ingredient. Many downstream failure modes — hallucination, bias, privacy leaks,…

2
data-quality bias grounding-problem +1
CRITICAL
9

Retrieval-Augmented Generation Pipeline

RAG is the book's primary prescription for grounding LLMs in reliable external knowledge. The pipeline —…

12
rag retrieval reranking +2
CRITICAL
10

Rules of Thumb

Collected heuristics and decision rules from across the book for quick reference during LLM application development. Each rule is grounded in a…

1, 2, 3, 5, 9, 10, 11, 12, 13
heuristics decision-rules quick-reference
HIGH
11

Tokenization and Its Hidden Failure Modes

The tokenizer determines what the model can "see." Subword tokenization (BPE) balances vocabulary coverage with compression efficiency, but…

3
tokenization bpe failure-modes +1
HIGH
12

Transformer Architecture and Learning Objectives

The Transformer architecture has been remarkably stable since 2017 — most gains have come from data and training methodology, not architectural…

4
transformer self-attention moe +1
HIGH