Chapter 5 From Model to Application
The earlier chapters cover how to choose, adapt, and align a model. This chapter covers what happens after that: making the model cheap and fast enough to serve, and connecting it to data and tools so that it can do useful work reliably. It closes with the risks specific to generative applications.
Three questions shape this stage of the project lifecycle (Section 1.7):
- Serving constraints. How fast must completions be, what compute budget is available, and how much quality can be traded for speed or cost?
- External resources. Does the model need data it was not trained on, or actions in other systems?
- Consumption. Who calls the model (people through a UI, or other systems through an API), and what must be logged, checked, or secured on the way?
5.1 Optimizing models for inference
Inference cost scales with model size. Smaller or lower-precision models load faster, need less GPU memory, and generate tokens with lower latency. The challenge is to shrink the model without losing the quality that matters for the task. Three techniques work on the model itself.
5.1.1 Knowledge distillation
Distillation trains a small student model to imitate a large, frozen teacher (Hinton et al. 2015). The teacher’s full output distribution carries more information than a single correct label. It shows which wrong answers are nearly right. A temperature \(T > 1\) in the softmax, \(p_i = \exp(z_i / T) / \sum_j \exp(z_j / T)\), flattens both distributions so that this information is visible (Section 1.6 explains temperature). The student minimizes a weighted sum of two losses:
\[ \mathcal{L} = \alpha \, T^2 \, D_{\text{KL}}\!\big(p^{\,T}_{\text{teacher}} \,\big\|\, p^{\,T}_{\text{student}}\big) + (1 - \alpha)\, \text{CE}\big(y,\; p^{\,T=1}_{\text{student}}\big) \]
- The first term (the distillation loss) matches the teacher’s softened distribution, the soft labels. The \(T^2\) factor keeps its gradient scale comparable as \(T\) changes.
- The second term (the student loss) is ordinary cross-entropy against the ground-truth hard labels at \(T = 1\).
Distillation does not shrink the teacher. It produces a separate, smaller model that replaces the teacher at inference. DistilBERT is the classic encoder example: about 40% smaller and 60% faster than BERT, while keeping about 97% of its language-understanding performance (Sanh et al. 2019).
Distillation is also standard for generative models. Two forms are common:
- Token-level, as above, matching the teacher’s next-token distribution. Gemma 2’s smaller models were trained this way (Gemma Team 2024).
- Sequence-level, where the student is fine-tuned on complete outputs generated by the teacher (Kim and Rush 2016). The small DeepSeek-R1 models were made by fine-tuning open models on reasoning traces generated by DeepSeek-R1 (DeepSeek-AI 2025).
A practical pattern follows: launch with a large, capable model, collect real traffic and approved outputs, then distill or fine-tune a smaller model for that specific workload.
5.1.2 Post-training quantization
Quantization stores numbers in fewer bits (Section 2.4). Quantization-aware training simulates low precision during training. Post-training quantization (PTQ) converts a model that has already been trained, so it is the usual route at deployment:
- Weights only or weights and activations. Quantizing activations as well saves more compute but usually costs more accuracy. Large LLMs have rare activation outliers with very large magnitude, which break naive 8-bit schemes. LLM.int8() handles them by keeping those dimensions in 16-bit (Dettmers et al. 2022).
- Calibration. PTQ runs a small sample of data through the model to estimate the range of values to be quantized.
- Error-aware methods. GPTQ quantizes weights to 3–4 bits layer by layer. It adjusts the remaining weights to compensate for each rounding error, which keeps accuracy close to the 16-bit model (Frantar et al. 2023).
The result is a smaller memory footprint and often faster inference, usually with a small, measurable drop in quality. Measure that drop on your own evaluation set before deploying.
5.1.3 Pruning
Pruning removes weights that contribute little. The simplest criterion is weights near zero (magnitude pruning). Methods differ in when they prune (during training, during fine-tuning, or after training) and in what they remove:
- Unstructured pruning zeroes individual weights. SparseGPT pruned 175B-parameter models to 50% sparsity in one shot with little loss in quality (Frantar and Alistarh 2023). Wanda reached similar results with a simpler score: weight magnitude times the norm of its input activation (Sun et al. 2024).
- Structured pruning removes whole neurons, attention heads, or layers.
Zeros only save memory and time if the storage format and hardware can exploit them. Unstructured sparsity often gives little real speed-up. Structured or hardware-supported patterns (such as 2:4 sparsity on recent NVIDIA GPUs) are needed for actual gains.
5.1.4 Serving-side optimizations
These leave the weights unchanged and optimize how tokens are generated:
- KV caching stores the attention keys and values of earlier tokens, so each new token is not recomputed from scratch. Memory for the cache, not compute, often limits how many requests fit on a GPU.
- Batching and paged KV memory. Serving systems such as vLLM manage the cache in pages to pack many concurrent requests efficiently (Kwon et al. 2023).
- Speculative decoding. A small draft model proposes several tokens, and the large model verifies them in one pass. The output distribution is unchanged, and latency drops when the draft is usually right (Leviathan et al. 2023).
5.1.5 Effort across the lifecycle
| Stage | Typical effort | Needs training? | Main expertise |
|---|---|---|---|
| Pre-training | Weeks to months, very large compute | Yes | Architecture, data, distributed training |
| Prompt engineering | Hours to days | No | Task knowledge, evaluation |
| Fine-tuning / PEFT | Hours to days once the data exist | Yes | Data preparation, evaluation |
| Alignment (RLHF, DPO) | Fast if a reward model or preference data exist; slow if human feedback must be collected | Yes | Labeling design, RL or preference tuning |
| Optimization for deployment | Hours to days | Sometimes (distillation) | Quality-versus-cost trade-off |
Most applications start from an existing model and spend their effort on the bottom four rows.
5.2 Problems that training alone does not solve
Even a well-adapted model has structural limits:
- Knowledge cut-off. The weights reflect the training data at a point in time. A model trained before a change of government will name the old prime minister with full confidence.
- Arithmetic and exact computation. The model predicts likely tokens; it does not calculate. Long multiplication or a tax calculation can come out close but wrong.
- Hallucination. Models produce fluent, specific text even when the content is unsupported. Examples include invented citations, products, or facts (Ji et al. 2023).
- No access to private or real-time systems. The model cannot see your order database or call your shipping API unless the application connects them.
The fix is to augment the model at inference time. An orchestration layer (application code or a framework) passes user input to the model, calls external data sources and tools based on the model’s output, and returns results to the model and the user.
5.3 Retrieval-augmented generation (RAG)
Retrieval-augmented generation gives the model relevant external text at inference time. It does not try to store that knowledge in the weights. Retraining for every update would be slow and expensive. Retrieval keeps knowledge current, supports private data, and lets answers cite their sources.
5.3.1 The original formulation
Lewis et al. (2020) paired a retriever with a generator. The retriever was a dense passage retriever (DPR) (Karpukhin et al. 2020). A query encoder embeds the question, and the closest passages are found in a pre-built index of Wikipedia passage embeddings. The generator (BART) conditions on the question plus the retrieved passages. The query encoder and generator were fine-tuned jointly, with the document index held fixed. RAG improved results on open-domain question answering over models that relied on their parameters alone.
5.3.2 A typical RAG pipeline today
Most applications use the same idea without joint training: off-the-shelf embedding models, a vector index, and a general LLM.
Indexing (offline):
- Load and clean the sources: documents, wikis, tickets, web pages.
- Chunk them into passages small enough to retrieve precisely, often a few hundred tokens with some overlap. Keep metadata such as source, date, and access rights.
- Embed each chunk into a vector with an embedding model.
- Store vectors and metadata in a vector store. A vector database adds keys, filtering, and updates, so each chunk can be traced back to its source document.
Answering (online):
- Embed the query with the same embedding model.
- Retrieve the top-\(k\) most similar chunks, usually by cosine similarity, \(\cos(\mathbf{q}, \mathbf{d}) = \frac{\mathbf{q} \cdot \mathbf{d}}{\lVert \mathbf{q} \rVert \lVert \mathbf{d} \rVert}\).
- Augment the prompt with those chunks, clearly delimited, along with an instruction to answer only from them and to cite them.
- Generate the answer, with citations back to the chunk sources.
Retrieval is not limited to vectors. The “retriever” can be a keyword search engine, a SQL query against a database, or a web search API.
5.3.3 What makes RAG work or fail
- Retrieval quality caps answer quality. If the right chunk is not retrieved, the model cannot use it. Evaluate retrieval separately, for example with recall@\(k\) on labelled question–passage pairs.
- Chunking is a design decision. Chunks that are too large dilute relevance. Chunks that are too small lose context.
- Hybrid search and reranking usually help. Combine keyword search (exact names, codes, IDs) with dense search (paraphrases), then re-score the candidates with a more accurate cross-encoder or LLM reranker.
- Position matters in long contexts. Models use information at the beginning and end of a long context more reliably than information in the middle (Liu et al. 2024). Retrieving fewer, better chunks beats stuffing the context.
- Grounding reduces but does not remove hallucination. The model can still ignore, misread, or over-generalize from retrieved text. Evaluate faithfulness (is every claim supported by the cited chunks?) separately from retrieval.
- Access control. Retrieval must respect the user’s permissions. Otherwise RAG becomes a data-leak channel.
Current practice (October 2026). Context windows are now far larger than the few thousand tokens of 2022–23 models. Check current model documentation. Long context reduces the need for aggressive chunking but does not replace retrieval. Cost, latency, position effects, and access control still favour sending only relevant material. Chapter 17 treats context selection as an engineering discipline.
5.4 Helping models reason
5.4.1 Chain-of-thought prompting
Multi-step problems are a classic weakness of models that answer in one step. Chain-of-thought (CoT) prompting (Wei, Wang, et al. 2022) includes worked intermediate steps in the few-shot examples. The model then produces its own steps before the answer. The paper’s standard example:
Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls.
Each can has 3 tennis balls. How many tennis balls does he have now?
A: Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis
balls. 5 + 6 = 11. The answer is 11.
Q: The cafeteria had 23 apples. If they used 20 to make lunch and bought
6 more, how many apples do they have?
A:With a bare answer in the example (“The answer is 11.”), the model in the paper answered 27. With the worked steps, it reasoned 23 − 20 = 3, 3 + 6 = 9 and answered correctly.
What the evidence says:
- CoT gains appeared mainly in large models (about 100B parameters in the original experiments). Smaller models produced fluent but illogical chains (Wei, Wang, et al. 2022).
- Zero-shot CoT: just adding “Let’s think step by step” also helps (Kojima et al. 2022).
- Self-consistency: sample several reasoning paths and take the majority answer. This improves accuracy over a single chain (Wang et al. 2023).
- The written chain is not guaranteed to be the real reason. Models can produce plausible explanations that hide the actual influence on their answer (Turpin et al. 2023). Treat CoT as a performance technique, not as a faithful explanation.
Current practice (October 2026). Reasoning models are trained with RL to produce long internal chains of thought before answering (Chapter 4). With these models, hand-written CoT examples matter less. Reasoning effort is often a setting (a “thinking” budget) instead. The Prompt Engineering part covers prompting for reasoning in detail.
5.4.2 Program-aided language models (PAL)
CoT fixes the plan but not the arithmetic: a correct chain can still contain a wrong multiplication. PAL (Luyu Gao et al. 2023) has the model write its reasoning as a program. Each reasoning step is a code comment, followed by the code that computes it:
# Roger started with 5 tennis balls.
tennis_balls = 5
# 2 cans of 3 tennis balls each is
bought_balls = 2 * 3
# tennis balls. The answer is
answer = tennis_balls + bought_ballsThe orchestrator runs the generated code in an interpreter and returns the result. The model decides what to compute; the interpreter guarantees the computation. PAL outperformed CoT on several mathematical and symbolic reasoning benchmarks in the original paper. Today the same idea is offered as a built-in code-execution tool by model providers. Run generated code only in a sandbox.
5.4.3 ReAct: reasoning plus acting
Many applications need several steps that involve external systems: look up an order, check a policy, request a shipping label, confirm an email address. ReAct (Yao et al. 2023) interleaves three kinds of output in a loop:
- Thought: reasoning about what to do next.
- Action: a call to one of a fixed set of allowed actions. In the
paper’s Wikipedia question-answering tasks these were
search[entity],lookup[keyword], andfinish[answer]. - Observation: the result of the action, which the orchestrator appends to the context before the next thought.
The loop repeats until the model chooses finish. Few-shot examples teach the
format, and an instruction block defines the allowed actions. Restricting the
action space is essential: a model will otherwise “plan” steps the application
cannot execute. In the paper, ReAct beat acting-only prompting throughout.
On fact verification (FEVER) it also beat chain-of-thought. On multi-hop
question answering (HotpotQA), the best results came from combining ReAct with
chain-of-thought and self-consistency. The largest gains were on interactive
decision-making tasks (ALFWorld, WebShop).
For an LLM to drive actions reliably, its output must:
- name a valid action from the allowed set;
- be machine-parseable (a fixed syntax, JSON, a SQL statement, a script); and
- include the information needed to validate the action, such as the customer’s email address before a return label is sent.
Current practice (October 2026). ReAct’s text-parsing loop has largely been replaced by native tool calling. The application declares tools with JSON schemas, the model emits structured tool calls, and the application executes them and returns results. The Model Context Protocol standardizes how tools and data sources connect to models (Chapter 16). Orchestration frameworks (LangChain was the early example, offering prompt templates, memory, tools, chains, and agents) are joined by vendor agent SDKs. The Thought → Action → Observation loop remains the core mental model for agents. Chapter 20 extends it to multi-agent systems.
5.5 The application stack
A production LLM application has several layers, and the model is only one of them:
| Layer | Contents |
|---|---|
| Infrastructure | Compute, storage, and networking, on-premises or cloud; sized for real-time or batch inference |
| Models | Foundation models and adapted models (fine-tuned, LoRA adapters, distilled students) |
| Data and retrieval | Document stores, vector indexes, databases, and their access controls |
| Orchestration and tools | Prompt templates, tool definitions, agent loops, code execution, external APIs |
| Output handling and feedback | Validation of outputs, logging, conversation memory, user feedback for later evaluation and fine-tuning |
| Interface and security | UI or REST API, authentication, rate limits, guardrails |
Logged prompts, completions, and feedback are a valuable asset. They become the evaluation sets and training data for the next iteration, provided users have consented and private data is handled correctly.
5.6 Responsible generative AI
Generative models add specific risks to the usual ML concerns:
| Risk | What goes wrong | Mitigations |
|---|---|---|
| Toxicity and bias | Harmful, offensive, or discriminatory output, including about protected groups | Curate training and fine-tuning data; alignment (Chapter 4); guardrail classifiers on inputs and outputs; diverse annotators and red-teaming |
| Hallucination | Confident, false content: invented facts, sources, or numbers | Grounding with RAG and citations; tools for exact computation; faithfulness evaluation; user education and disclaimers; restrict use to suitable tasks |
| Intellectual property and privacy | Reproducing copyrighted or personal data from training or retrieval sources | Data governance and licensing; filtering and blocking of protected content; policy and legal controls; machine “unlearning” is an active research area, not a solved tool |
| Prompt injection | Instructions hidden in retrieved documents, web pages, or tool outputs hijack the model, for example to leak data or trigger actions (Greshake et al. 2023) | Treat retrieved and tool content as untrusted data; least-privilege tools; human confirmation for consequential actions; output validation |
A workable process for each application:
- Define the use case narrowly. The narrower the task, the easier the risks are to assess and test.
- Assess the risks for that use case: who is affected, and what a failure costs.
- Evaluate on representative data, including adversarial and subgroup cases, against explicit acceptance criteria.
- Monitor and iterate across the whole lifecycle. Models, data, and misuse patterns change after launch.
The NIST AI Risk Management Framework (National Institute of Standards and Technology 2023) and its generative-AI profile (National Institute of Standards and Technology 2024) are widely used, vendor-neutral references for organizing this work. Regulatory obligations (for example the EU AI Act) depend on jurisdiction and use case; check current requirements for a real deployment.
5.7 Key takeaways
- Distillation trains a smaller student on a teacher’s soft outputs or generated text. PTQ lowers precision after training. Pruning removes low-value weights and only speeds things up with structured or hardware-supported sparsity. Measure the quality cost of each.
- Serving efficiency also comes from KV caching, batching, and speculative decoding, which leave the weights unchanged.
- Training cannot fix knowledge cut-offs, exact computation, or access to private systems. Augment the model at inference instead.
- RAG retrieves relevant, permitted text and grounds the answer in it. Evaluate retrieval and faithfulness separately. RAG reduces hallucination but does not eliminate it.
- CoT improves multi-step reasoning in large models but is not a faithful explanation. PAL moves computation to an interpreter. ReAct’s Thought → Action → Observation loop over a fixed action set is the basis of modern tool-calling agents.
- The model is one layer of the application. Responsible deployment means narrow use cases, explicit risk assessment, evaluation, monitoring, and defences against prompt injection.
5.8 Interview checkpoints
- Why use a temperature above 1 in distillation? It softens the teacher’s distribution so that the relative probabilities of wrong answers (the “dark knowledge”) carry training signal.
- PTQ vs. QAT? PTQ converts a trained model and needs calibration data. QAT simulates quantization during training and usually loses less accuracy, at extra training cost.
- Why might pruning not speed up inference? Unstructured zeros need sparse formats and kernel or hardware support to save time.
- RAG vs. fine-tuning for company knowledge? RAG for facts that change or must be cited and permission-filtered. Fine-tuning for behaviour, format, and style. They are often combined.
- Main RAG failure points? Missed retrieval, poor chunking, lost-in-the- middle effects, unfaithful generation, and permission leaks.
- What is cosine similarity, and why use it for embeddings? The normalized dot product. It compares direction (meaning) independently of vector length.
- CoT vs. PAL? CoT writes reasoning in text and computes in the model. PAL writes code and computes in an interpreter.
- What are the three parts of a ReAct step? Thought, Action (from a fixed set), Observation.
- What is indirect prompt injection? Malicious instructions placed in content the model reads (documents, web pages, tool outputs) rather than typed by the user.