Chapter 22 Context Engineering and Harness Engineering

22.1 Why these two terms matter

Earlier chapters covered prompting, verification loops, hooks, skills, and MCP as separate techniques. Two newer terms give those techniques a common frame:

  • Context engineering asks: which tokens should the model see at this step? It covers instructions, retrieved data, tool definitions, tool outputs, conversation history, and memory.
  • Harness engineering asks: what system around the model makes it reliable over many steps? It covers the agent loop, tools, sandbox, state files, hooks, permissions, tests, and evaluation.

Both terms are young, and their boundaries are still debated. This chapter separates three kinds of claim: concepts that are likely to last, vendor practices and features that change quickly, and evidence. The evidence is labeled by strength throughout.

Source labels used in this chapter

  • [Vendor]: an organization reporting on its own product or results.
  • [Secondary]: the original could not be read; the claim comes from a mirror, summary, or aggregator.
  • [Unverified]: no primary source was found.

Product names, API identifiers, and preview status are as of October 2026.

Parts of this chapter began as study notes from the DeepLearning.AI course Context Engineering, built with Chroma and taught by Chroma’s CEO, Jeff Huber. Its first module follows a customer-service agent, and its second a coding agent investigating a customer-feedback app; this chapter borrows both scenarios. Chroma sells retrieval infrastructure, so its research claims carry the [Vendor] label like any other.

22.2 Where the terms came from

22.2.1 Context engineering (mid-2025)

The term spread through a cluster of posts in June and July 2025. No single person has a verified claim to coining it, so “popularized by” is the accurate phrasing.

  • Shopify CEO Tobi Lütke described it as “the art of providing all the context for the task to be plausibly solvable by the LLM” (June 2025, quoted by Simon Willison).
  • Harrison Chase of LangChain defined it as building dynamic systems that give the model “the right information and tools in the right format,” and credited earlier practitioners (LangChain).
  • Andrej Karpathy called it “the delicate art and science of filling the context window with just the right information for the next step.” He compared the model to a CPU and the context window to its RAM (June 2025, quoted by Simon Willison).
  • Drew Breunig named common ways long contexts fail (How long contexts fail; How to fix your context).
  • Manus published lessons from running a production agent (Manus, July 2025).
  • Anthropic’s engineering guide (29 September 2025) gave the most-cited definition: strategies for curating and maintaining the optimal set of tokens during LLM inference (Anthropic).
  • The DeepLearning.AI and Chroma course uses a three-part working definition: giving the agent the right information, in the right format, at the right time for its current task. Section 22.4.1 uses it as a design checklist.

22.2.2 Harness engineering (early 2026)

“Harness” and “scaffold” already meant the code around a model. The phrase harness engineering became common in early 2026.

  • Phil Schmid (January 2026) described the agent harness as the infrastructure that wraps a model to manage long-running tasks, and said it is not the agent itself. He extended Karpathy’s analogy: model = CPU, context = RAM, harness = operating system, agent = application (philschmid.de).
  • Mitchell Hashimoto (February 2026) gave the most practical definition: whenever an agent makes a mistake, engineer a fix so it never makes that mistake again. His fixes were updates to AGENTS.md plus small, purpose-built scripts (mitchellh.com).
  • OpenAI published “Harness engineering: leveraging Codex in an agent-first world” (February 2026). It describes a team that designs the environment while agents write the code (OpenAI). [Secondary]: the page could not be read for this chapter. Its widely repeated figures (code volume, team size, number of pull requests) are therefore not reproduced here.
  • LangChain summarized the idea as “Agent = Model + Harness” (LangChain, March 2026).
  • Birgitta Böckeler’s article on martinfowler.com (April 2026) reframed harness engineering for teams that use coding agents (martinfowler.com).
  • Anthropic documented harness patterns for long-running agents (November 2025; March 2026) [Vendor].

These definitions emphasize different things. Hashimoto’s is incremental, OpenAI’s is organizational, Schmid’s and LangChain’s are architectural, and Böckeler’s treats the harness as a control system.

22.3 How prompt, context, and harness relate

Layer Question it answers Typical artifacts
Prompt engineering How should the instruction be worded? System prompt, few-shot examples
Context engineering Which tokens should the model see at this step? Retrieval, tool selection, compaction, memory files, sub-agent summaries
Harness engineering What system makes the model reliable over many steps? Agent loop, sandbox, hooks, tests and linters, evaluator agents, state logs, permissions

LangChain calls prompt engineering “a subset of context engineering,” and Anthropic calls context engineering its “natural progression.” Treat the chain prompt ⊂ context ⊂ harness as a teaching heuristic, not a settled taxonomy: it appears mostly in explainers, not in primary lab writing.

The more precise statement is that the harness contains context management, but it also includes deterministic machinery, such as linters, sandboxes, and permission checks. That machinery affects the model only when its output is fed back into the context.

Is it just a rebrand? In part. Retrieval, summarization, and RAG all predate the label, and some experienced developers dismiss the term as rebranded prompt engineering (Addy Osmani). What is new is treating these techniques as one budgeted design problem for multi-step agents, which a prompt-only view hides.

22.4 Context engineering

22.4.1 Right information, right format, right time

The course’s definition splits the work into three questions. Each one names a different part of the system to inspect.

Axis Question to ask Typical techniques Closest family below
Right information Did the agent retrieve the most relevant content for this task? Search tools, agentic search, combining search methods Select
Right format Is what was retrieved clean, structured, and free of duplicates and noise? Deduplication, pruning, compaction, summaries, structured metadata Compress
Right time Does the content arrive when it is needed, and not before? Just-in-time retrieval, memory that stores and reloads state Select and write

Formatting covers more than headers and layout. A summary is also a format: it consolidates information so that the important details stand out.

The three axes and the four families of technique (Section 22.4.5) describe the same work from two angles. Use the axes to diagnose a problem: is the issue what was found, how it was presented, or when it arrived? Then pick a technique from the families to fix it. Section 22.4.6 covers the first axis, right information, in depth.

22.4.2 What is in the context window

Before you change anything, list the kinds of context your agent handles. In a simple chatbot there were only three: the system prompt, the user’s input, and the model’s reply. RAG added retrieved documents. An agent adds several more:

  • System prompt. The standing instructions. It sits at the front of every call, so it should be short and stable. Anything that changes per request belongs elsewhere. A stable prefix also keeps the prompt cache warm (Section 22.4.7).
  • User input. The request, plus any later turns in the conversation.
  • Memory. Stored information that is reloaded when relevant, such as user preferences or task instructions.
  • Reasoning traces. The agent’s own planning and intermediate thinking.
  • Tool context. Tool definitions, the inputs the model sends, the outputs that come back, and any retrieved data.

The course’s running example is a ticket-refund agent. A customer asks for a refund and gives an order number. The agent calls a lookup tool to fetch the order, calls a policy-search tool to read the refund rules, calls a refund tool, and replies. Every one of those steps adds tool inputs and outputs to the context.

The context only grows, and it is resent in full. The model is stateless, so the harness sends the whole accumulated context again on every call. That includes the system prompt, every tool definition, and every earlier tool output. Two consequences follow:

  • Growth compounds. If each turn adds 10% to the context, it roughly doubles after seven turns (\(1.1^7 \approx 1.95\)). If each turn adds 150%, which is plausible for a coding agent that reads files and runs tests, a context that starts at 1,000 tokens passes one million tokens during the eighth turn (\(1{,}000 \times 2.5^7 \approx 610{,}000\) and \(1{,}000 \times 2.5^8 \approx 1.5\) million).
  • What people wrote is the smallest part. In the course’s token breakdown for a short refund request, the largest categories were the system prompt, the tool definitions, and the tool outputs. All three are resent on every call. The customer’s message, the tool arguments, and the final answer were barely visible. The cost you pay is the sum of the context sizes over all calls, not the size of the final context.

22.4.3 Context is a finite attention budget

The founding premise is that more context can make results worse. Anthropic describes a finite “attention budget” and names context rot: recall falls as the token count grows. The goal it sets is “the smallest set of high-signal tokens” that makes the desired outcome likely.

Evidence:

  • Lost in the middle. Accuracy was highest when the relevant information sat at the beginning or end of a long input, and worse in the middle (Liu et al., arXiv 2307.03172).
  • Context rot. Across 18 models, performance degraded as input length grew, often unevenly (Chroma, July 2025). [Vendor]: Chroma sells retrieval infrastructure, and the study is not peer reviewed.
  • Focused versus full history. In the same report, Chroma used questions from the LongMemEval benchmark (Wu et al., arXiv 2410.10813), where each question is about one exchange buried in a long chat history. A full prompt held the whole history (about 113,000 tokens on average). A focused prompt held only the few relevant messages (about 300 tokens). The answer was present in both versions. Every model tested did significantly better on the focused prompt. Thinking modes improved both conditions but did not close the gap [Vendor].

A practical consequence: a larger advertised context window is a capacity limit, not a promise that the model will use every token equally well. The course separates two quantities:

  • Advertised context length: the maximum number of tokens a model accepts in one call.
  • Effective context length: how much of that the model can use reliably for a given task. It depends on the model, the task, and how the content is arranged, so there is no universal cut-off.

In the course, Chroma’s instructor says the effective length can be as low as 20% to 40% of the advertised length. That figure does not appear in the context rot report itself [Unverified]. Treat it as an order of magnitude, and measure on your own task.

Context rot is a size problem; the failures below are quality problems. Rot comes from how attention behaves as the window fills, and it happens even when every token is correct. The four failure modes in the next section come from what the content says, and they can occur in a context of a few hundred tokens. A long history makes quality failures more likely, because there are more chances for two pieces to contradict each other, but the two problems need different fixes.

22.4.4 Four ways context fails

Breunig’s taxonomy pairs each failure with a fix (How long contexts fail). The examples he cites are secondhand and were not checked against the original papers for this chapter.

Failure What happens Typical fix
Poisoning A hallucination or error enters context and is reused as if true Validate before writing to memory; prune; quarantine
Distraction A long history crowds out the task; the agent repeats past actions Summarize or compact; start fresh with a handoff note
Confusion Irrelevant tools or documents degrade choices Load fewer tools; retrieve selectively
Clash New information contradicts earlier context Remove stale content; keep one source of truth

One conversation, four failures. The course shows how all four can occur in a single short refund conversation. The company has just extended its refund window from 15 to 45 days, but both policy files are still in the document store.

Step in the conversation Failure Why it happened
The policy search returns the old 15-day file, and the agent denies a valid refund Poisoning Outdated content entered the context, and nothing marked it as outdated
The customer pushes back; a second search returns the 45-day file, and both versions now sit in context Clash Two plausible rules conflict, and neither says which one is in force
The customer says “just cancel it”; the agent calls the cancellation tool instead of the refund tool Confusion The two tool descriptions overlap, so an imprecise word picks the wrong action
A side question triggers a web search that returns a long page of raw HTML Distraction Irrelevant content, some of it sharing keywords like “refund”, crowds out the task

The customer ends up with no tickets and no money. None of these were failures of the model’s reasoning. Each came from what the harness let into the context.

The course uses the same four names with slightly broader meanings than Breunig’s:

  • Its poisoning includes stale retrieved documents, not only hallucinations that get reused.
  • Its confusion is about options too similar to tell apart. Those can be overlapping tool descriptions, or three products that share a name. Breunig’s version is about superfluous tools or documents.
  • Its distraction is irrelevant content crowding the task, not only a long history.

The fixes are the same either way.

Evidence in the context is not enough. In the course’s first lab, a small local model (Qwen2.5-Coder 1.5B) received the whole knowledge base on every query. It still sided with a marketing draft over the binding pricing policy, even though the policy carried an effective date and the draft carried a note saying the policy had not been updated. A frontier model would often resolve that conflict. But companies sometimes run small models on their own infrastructure to keep customer data private, and a production system should not rely on the model making the right inference. Remove the stale or conflicting content before it reaches the model, as Section 22.5.2 shows.

22.4.5 Four families of technique: write, select, compress, isolate

LangChain groups the techniques into four families (LangChain).

  1. Write: persist state outside the window. Scratchpads, progress notes, and to-do files let an agent recover after its history is trimmed. Manus has the agent rewrite a todo.md so its goals stay at the end of the context, a direct counter to lost-in-the-middle.
  2. Select: retrieve just in time. Keep lightweight identifiers (file paths, table names, saved queries) and load content only when needed. Claude Code combines an up-front CLAUDE.md with on-demand file search. Selection applies to tools too: a smaller, relevant toolset reduces confusion.
  3. Compress: summarize or prune. Compaction replaces old history with a summary that keeps decisions, open issues, and key details, and drops redundant tool output.
  4. Isolate: give sub-tasks clean windows. A sub-agent explores in its own context and returns a short summary (Anthropic suggests roughly 1,000 to 2,000 tokens). Isolation costs extra tokens, and some practitioners argue that splitting work across agents fragments shared understanding. Choose it for parallel, separable work, not by default.

Supporting guidance from the Anthropic guide:

  • Write system prompts at the “right altitude”: specific enough to guide behavior, but heuristic rather than a brittle list of rules (Section 6.6 compares the three levels).
  • Use a few diverse, canonical examples rather than every edge case.
  • Make tools self-contained, robust to errors, and clearly distinct from one another. Return compact tool outputs, not raw dumps.

22.4.7 Cost: the overlooked dimension

Agents read far more tokens than they write. Manus reports a ratio of roughly 100 input tokens per output token and calls the KV-cache hit rate the most important production metric [Vendor]. Cached input tokens are much cheaper than uncached ones, which leads to rules such as:

  • keep the prompt prefix stable (a timestamp at the top breaks the cache);
  • append to the context rather than editing earlier turns;
  • serialize JSON deterministically; and
  • disable tools by masking them rather than removing their definitions.

These rules conflict with compaction and dynamic tool removal, which invalidate the cache. Good designs compress rarely and in large steps rather than continuously.

Manus also advises leaving failed attempts in context so the model avoids repeating them, which seems to contradict “poisoning.” One reasonable reconciliation (the author’s judgment): observed error messages are useful feedback, while unverified claims stored as facts are harmful.

22.4.8 Context features are becoming platform features

Vendors now ship these techniques as features, for example Anthropic’s context editing, memory tool, server-side compaction, tool search (Section 22.4.6.3), and the progressive disclosure built into Agent Skills (Chapter 18). Learn the four families. The API names will change, but the families will still apply.

22.5 Harness engineering

22.5.1 What is in a harness

A short working definition: the harness is everything except the model weights. LangChain’s list includes system prompts; tools, skills, and MCP servers (Chapter 21); infrastructure such as a filesystem, sandbox, or browser; orchestration such as sub-agents and model routing; and hooks or middleware for deterministic steps (LangChain).

It helps to separate:

  • the builder harness: the product runtime (Claude Code, the Claude Agent SDK, Codex, LangChain Deep Agents, a custom agent); and
  • the user harness: what your team adds on top, such as AGENTS.md, specs, tests, linters, hooks, permissions, and CI.

Most data scientists work mainly on the user harness, or build a small builder harness inside an application.

22.5.2 Tools and hooks: the two levers

Context-management components enter a harness in one of two forms.

A tool is a function the model chooses to call. It has four parts:

Part What it is Does the model see it?
Tool definition Name, description, and input schema Yes; it is the only part the model ever reads
Tool input The structured request (usually JSON) the model emits Yes; the model writes it
Tool implementation The code that runs No; it stays in the harness
Tool output What the code returns Only after the harness appends it to the next call

Tool use can fail in four places. The model may skip a tool it needed, call the wrong tool, or call the right tool with the wrong arguments. The first three usually trace back to the tool description, which works like a docstring and must say clearly when to use the tool. The fourth failure is the subtle one: the right tool is called with the right argument, but the data behind it is wrong. In the course example, the policy search did exactly what was asked and returned an outdated file. No amount of prompt or description work fixes that, because the model cannot reject content it has no reason to doubt.

A hook is a function the harness runs when a specific event happens. The model never sees it, never requests it, and cannot skip it. Hooks are deterministic.

  • A pre-tool hook runs before a tool executes. Examples: check that the user is allowed to use the tool, or block a request that was already answered.
  • A post-tool hook runs after the tool returns and before its output enters the context. Examples: drop stale documents, deduplicate, truncate, or validate a schema.

Where a hook is wired in varies by system. The course labs call post-tool hooks from inside the tool, which leaves the agent loop untouched. Claude Code fires PreToolUse and PostToolUse hooks from the harness itself (Chapter 17). Either way, the model does not know the hook exists.

The sketch below is a post-tool hook for the stale-policy problem, applied to marketing metric definitions. It assumes every document carries an effective date. That metadata is the precondition, and it is data-governance work (Section 22.6.1). Without it, no hook can tell old from current.

from datetime import date


def keep_current_versions(docs: list[dict], today: date) -> list[dict]:
    """Post-tool hook: keep only the version of each document in effect today."""
    current: dict[str, dict] = {}
    for doc in docs:
        if doc["effective"] > today:  # drafts and future versions are not in effect yet
            continue
        best = current.get(doc["name"])
        if best is None or doc["effective"] > best["effective"]:
            current[doc["name"]] = doc
    return list(current.values())


# What a search tool might return for "how do we define reach and lift?"
retrieved = [
    {"name": "reach.md", "effective": date(2024, 1, 1), "text": "Reach = unique households exposed"},
    {"name": "reach.md", "effective": date(2026, 3, 1), "text": "Reach = unique persons exposed"},
    {"name": "lift.md", "effective": date(2025, 6, 1), "text": "Lift = test rate / control rate - 1"},
    {"name": "lift.md", "effective": date(2027, 1, 1), "text": "Lift (draft) = incremental conversions"},
]

for doc in keep_current_versions(retrieved, today=date(2026, 11, 14)):
    print(doc["name"], doc["effective"], "|", doc["text"])
# reach.md 2026-03-01 | Reach = unique persons exposed
# lift.md 2025-06-01 | Lift = test rate / control rate - 1

One short function prevents two failures. The retired version cannot poison the answer, and with only one version in context there is nothing to clash with. It also saves tokens on every later call, because the dropped documents are never resent. Nothing else changed: not the prompt, the model, nor the agent loop. This is the course’s main argument for hooks: when a rule can be written in code, enforce it in the harness rather than hoping the model infers it.

22.5.3 Guides and sensors

Böckeler classifies harness controls on two axes (martinfowler.com):

Computational (deterministic, cheap, reliable) Inferential (model-based, slower, non-deterministic)
Guides (before the agent acts) Templates, schemas, scaffolding scripts, type definitions AGENTS.md, specs, feature lists, skills
Sensors (after the agent acts) Tests, linters, type checks, schema validation, data-quality checks AI code review, LLM judges, evaluator agents

Prefer computational sensors wherever they can answer the question, and use inferential ones for judgments that code cannot make. Böckeler also notes the limits: tests do not catch a misdiagnosed problem, over-engineering, or a misunderstood instruction.

22.5.4 Each pattern fixes a named failure

Recurring failure Harness response Source
Declares success too early Feature list where every item starts “failing”; checklist before exit; stop hooks that require evidence Anthropic (Nov 2025), LangChain [Vendor]
Tries to do everything at once Work on one feature per session; commit after each Anthropic (Nov 2025) [Vendor]
Edits tests to pass Explicit rule against changing tests; protected test files; CI Anthropic (Nov 2025) [Vendor]
Loses track over long runs Progress log, setup script, descriptive commits, handoff notes Anthropic (Nov 2025) [Vendor]
Praises its own mediocre work A separate evaluator agent that exercises the result Anthropic (Mar 2026) [Vendor]
Repeats the same failing edit Loop detection that interrupts the agent LangChain (Feb 2026) [Vendor]
Repeats a known mistake Add a rule to AGENTS.md or a deterministic check Hashimoto (Feb 2026)

Documentation is part of the harness, and long documents rot. Several practitioners, OpenAI reportedly among them [Secondary], replaced one large instruction file with a short index that points to focused documents. This is the same idea as progressive disclosure in skills and just-in-time retrieval.

Security gaps. The primary harness posts reviewed say little about permissions, sandboxing, secret handling, or prompt injection through tool outputs. Treat these as first-class harness components anyway: apply least privilege, require human approval for actions that write, spend, or send, and keep secrets out of context (see Chapters 17 and 21).

22.5.5 The thin-harness debate

Rich Sutton’s “Bitter Lesson” argues that general methods that scale with compute eventually beat hand-built cleverness. Applied to agents, it raises the question of whether harness logic is a temporary crutch.

  • Thin-harness view. Anthropic writes that every harness component “encodes an assumption about what the model can’t do on its own.” When a stronger model arrived, it removed parts of its own harness (Anthropic, March 2026) [Vendor]. Schmid recommends harnesses that make yesterday’s clever logic easy to delete.
  • Harness-matters view. With the model held fixed, LangChain changed only its harness and raised its Terminal-Bench 2.0 score from 52.8% to 66.5% (LangChain) [Vendor]. A May 2026 position paper argues that harness-induced variance can exceed model-induced variance and can reverse model rankings (arXiv 2605.23950); it is an argument, not a controlled meta-analysis.

The two views are compatible. Specific components (elaborate prompts, rigid step structures, large tool collections) are absorbed as models improve. The need for an environment, verification, persistent state, and permissions remains.

Practical implication: a leaderboard score measures a model–harness pair. When you compare models for your own application, hold the harness constant, and when you change the harness, hold the model constant.

22.5.6 Prompt debt

In a conversation recorded for the course, Drew Breunig described a pattern he calls prompt debt, by analogy with technical debt. This is a practitioner’s opinion, not a measured result.

  1. A team under pressure to ship gives a prompt to the largest frontier model and lets it work out the path. It succeeds most of the time, and the demo looks great.
  2. In use, edge cases keep appearing. Each fix is another line in the prompt: a special rule, capital letters, a threat or a bribe.
  3. Those patches are tuned to one model’s quirks. The prompt becomes long, brittle, and hard to change, and switching to a new model means starting over.

The teams that get out of prompt debt tend to converge on the ideas DSPy is built around: make AI behave more like software, with contracts. In practice that means:

  • typed inputs and outputs;
  • retry logic;
  • structured prompts with schemas; and
  • breaking one large call that does twenty things into smaller steps, each with its own eval.

Breunig also offered a threshold. If an agent is wrong about 10% of the time, people stop using it, because they must check every result and might as well do the work themselves. Teams often add a human in the loop to cover the gap, as coding agents do. But the enterprise value he sees most often comes from narrow workflows with no human in the loop, where the task is small enough to make reliable.

This connects to the rest of the chapter. Hashimoto’s rule, “engineer a fix so the mistake never recurs”, can be followed in two ways. Adding another line to the prompt builds prompt debt. Adding a deterministic check, hook, or test does not. The author’s judgment is to use the prompt for judgment calls and put every rule that can be written as code into the harness.

22.5.7 Look at your traces

A complaint from a user tells you that something went wrong, but the user messages and the agent’s replies alone rarely show why. To diagnose a context problem, you need to see what entered the context at each step.

  • A trace is the full record of one task: user inputs, agent responses, every tool call with its input and output, the agent’s reasoning, every model call, and any housekeeping the harness did, such as compaction.
  • A span is one step inside the trace, for example a single tool call with its input and output.

A trace can hold more than the model saw. A hook’s log of which documents it dropped belongs in the trace, but it was never sent to the model.

Capturing what happened is ordinary logging. A logging hook that fires on each tool call can write to a file or to a tracing platform. The course names Arize, LangSmith, and Weights & Biases; on Databricks, MLflow Tracing plays this role.

Capturing why is harder, because the reasoning is usually not in the replies. The course shows two options:

  • Ask the model to think step by step inside tags such as <think>, and log those blocks.
  • Add a reason field to each tool’s input schema. The model then states its objective, why it picked this tool, and why it chose these arguments, and the reason is logged with the call.
# A tool definition with a reason field; the exact wrapper format varies by vendor.
search_metrics_tool = {
    "name": "search_metric_definitions",
    "description": "Return the governed definition of one or more marketing metrics.",
    "parameters": {
        "type": "object",
        "properties": {
            "metric_names": {"type": "array", "items": {"type": "string"}},
            "reason": {
                "type": "string",
                "description": (
                    "Your current objective, why this tool helps, "
                    "and why you chose these metric names."
                ),
            },
        },
        "required": ["metric_names", "reason"],
    },
}

Two cautions, both the author’s judgment:

  • The reason field costs tokens on every call. Its description is part of the tool definition, which is resent on every call, and the model’s answer adds to the tool input. Measure that cost.
  • A stated reason is the model’s self-report, not ground truth. Use it to find where to look, and confirm with the actual inputs and outputs.

Reading traces is the input to error analysis (Section 10.4). Once you can name a failure in the traces, you can build an eval that measures whether a context-engineering fix worked.

22.5.8 Evaluation is a harness too

Anthropic’s guide to agent evaluation distinguishes the evaluation harness from the agent harness and defines tasks, trials, graders, transcripts, and outcomes (Anthropic, January 2026):

  • Graders can be code-based (tests, exact match, schema checks), model-based (rubrics, pairwise comparison), or human (expert review).
  • Grade outcomes, not paths. Judge what the agent produced, but keep transcripts for debugging and for spotting reward hacking, such as an agent editing the tests it is graded on.
  • versus pass^k. is the chance that at least one of k attempts succeeds; pass^k is the chance that all k succeed. For a production tool that users run repeatedly, pass^k is the more honest reliability measure.

For a statistician, this is familiar ground: define the estimand (task success), use repeated trials to estimate variance, and avoid grading on the training set (tasks the harness was tuned on). Chapter 10 covers the vendor-neutral method: end-to-end and component evals, error analysis, and error bars.

22.6 Applying both to Databricks analytics apps

The design principle that follows from the evidence is:

The model chooses and explains; governed engines compute; sensors verify.

This is the author’s synthesis rather than an official Databricks rule, but each part has support in the sources above.

22.6.1 Metadata is the highest-return context

Text-to-SQL research suggests that semantic context often matters more than which frontier model you pick:

  • Adding a 4 KB semantic-layer document raised accuracy on 100 retail questions from about 46–51% to about 68–69% across three frontier models, and the differences between models were not statistically significant in either condition (arXiv 2604.25149). This is a single study on one domain.
  • On the BIRD benchmark, better column descriptions improved accuracy, although ambiguous columns still needed expert input (arXiv 2408.04691).
  • Automatically profiled metadata helped one team reach the top of the BIRD leaderboard (arXiv 2505.19988).

Databricks’ own Genie guidance is a context-engineering playbook (Genie best practices):

  1. Rely on clear Unity Catalog table and column names and descriptions.
  2. Start with a small number of tables (the docs suggest five or fewer).
  3. Prefer SQL expressions for metrics, filters, and dimensions, then example SQL for complex questions, and use plain-text instructions only as a last resort.

Unity Catalog metric views define each measure once (reach, overlap rate, lift, support, confidence) so that every app and agent computes it the same way.

For an analytics team, this is good news: the highest-return work is data governance work you may already own. Column comments, metric definitions, and verified example queries now have a new consumer.

22.6.2 Mapping to the three workloads

Workload Context engineering Harness engineering Evaluation
Streamlit analytical apps on Databricks Apps Send aggregates (top-N, group-bys, summary statistics), not audience-level rows. Keep few tables in scope. Keep a stable system-prompt prefix On-behalf-of-user auth so Unity Catalog row filters and column masks apply per user. The model proposes SQL or tool parameters; tables and charts render from the result DataFrame, not from model text A code check that every number in the narrative appears in the result table; an LLM judge for groundedness
Reusable jobs (lookalike, audience overlap, basket analysis) Unity Catalog comments on seed, segment, and transaction tables; metric views for overlap, lift, support, and confidence; a few verified example queries for tricky logic such as fiscal calendars Expose validated logic as Unity Catalog functions or job-triggering tools via MCP. For coding agents that maintain the job code, a short AGENTS.md map plus tests act as guides and sensors Compare results with known gold outputs; measure pass^k across repeated runs; protect unit tests from agent edits
AI-assisted apps on Databricks foundation models (Genie, Agent Bricks, custom agents) Follow the Genie order of instruction types; retrieve methodology documents just in time with AI Search rather than preloading them Gateway quotas and budgets; guardrails; durable memory only for state that must persist. Inference logging stores whatever enters context, so keep personal data out Genie benchmarks; MLflow GenAI evaluation with code scorers and judges; labeled cases where the app should ask for clarification or refuse

Relevant Databricks references: Databricks Apps authorization, managed MCP servers (Public Preview), and the Foundation Model APIs supported models page. Product names change often: Genie spaces are now called Genie Agents, Vector Search is now AI Search, and AI Gateway is now Unity Gateway. Check each docs page’s preview banner before relying on a feature.

22.6.3 Three cautions for marketing data

  1. Schema confusion is tool confusion. Too many tables in scope degrade text-to-SQL the same way too many tools degrade tool choice. Curate.
  2. Client data in prompts is both a privacy risk and a cost. Aggregate first, enforce minimum cell sizes, and use per-user authorization.
  3. Do not let the assistant grade its own narrative. A deterministic check against the result table is cheaper and more reliable than asking a model whether its lift or overlap summary is correct.

22.7 A minimal sketch: the number-check sensor

This illustrative check implements the third caution. It is an instructional example, not tested production code, and the regular expression handles only simple number formats.

import re

import pandas as pd


def numbers_in_text(text: str) -> set[float]:
    """Extract plain numbers such as 12, 3.5, 1,200 or 45% from narrative text."""
    tokens = re.findall(r"-?\d[\d,]*\.?\d*", text)
    return {round(float(t.replace(",", "")), 2) for t in tokens}


def unsupported_numbers(narrative: str, result: pd.DataFrame) -> set[float]:
    """Return numbers in the narrative that do not appear in the result table."""
    numeric = result.select_dtypes("number").to_numpy().ravel()
    allowed = {round(float(x), 2) for x in numeric}
    allowed |= {round(float(x) * 100, 2) for x in numeric}  # shares shown as %
    return numbers_in_text(narrative) - allowed


result = pd.DataFrame({"segment": ["A", "B"], "overlap_rate": [0.18, 0.07]})
narrative = "Segment A overlaps 18% with the seed audience, versus 9% for B."
print(unsupported_numbers(narrative, result))  # {9.0} -> flag for review

In a harness, this check runs after the model writes its summary. If it finds unsupported numbers, the app either asks the model to revise using only the table or shows the table without the narrative.

Hands-on. Appendix C walks through the course labs for this chapter: triggering the four failure modes, measuring how an agent’s context grows, filtering stale policies with a post-tool hook, giving a coding agent seven search tools, and loading tool definitions from a registry.

22.8 Exercise: audit one AI-assisted app

Choose an existing assistant or prototype and answer:

  1. Context inventory. List everything in the context window on a typical request, using the components in Section 22.4.2. Estimate tokens for each, and how much each grows per turn. Remember that everything is resent on every call. What could be removed or loaded just in time?
  2. Failure mapping. Read the full trace of one real failure, not just the final answer. Classify it as poisoning, distraction, confusion, or clash, or as a harness failure such as premature success or missing verification. Then say which axis failed: information, format, or timing.
  3. One guide, one sensor. Add one guide (a metric definition, example query, or AGENTS.md rule) and one computational sensor (a schema check, number check, or test).
  4. Measure. Build 20 to 30 representative questions with known answers. Run each several times and report pass^k before and after your change, holding the model constant.
  5. Search audit. For five real questions, write down where a careful colleague would look and how they would match: an exact ID, keywords, a concept, SQL, or logs (Section 22.4.6.1). Compare that with the tools the agent called in its traces. For one question, list the ground-truth chunks and compute and .

22.9 Key takeaways

  • Context engineering decides what the model sees at each step; harness engineering builds the system that makes many steps reliable.
  • Treat prompt ⊂ context ⊂ harness as a useful heuristic, not an official taxonomy; the harness also includes deterministic parts that never enter the context.
  • Give the agent the right information, in the right format, at the right time. Use these three axes to diagnose a problem, and the four families to fix it.
  • Context is a finite attention budget: more tokens can reduce quality. Use the four families—write, select, compress, isolate—and watch the cost trade-off with caching.
  • Context rot is a size problem; poisoning, clash, confusion, and distraction are quality problems. Effective context length is shorter than the advertised length, and the whole context is resent on every call.
  • Agentic search lets the agent choose what to search, where, and how, and search again. It finds better context at the cost of predictability, time, and context size. Match the method to the question: exact match for IDs, BM25 for keywords, embeddings for concepts, RRF to fuse both, SQL for records.
  • When search fails, locate the failure before fixing it: wrong tool, wrong source, bad arguments, or the right call returning the wrong chunks. The last one is a data-preparation problem; measure it with and .
  • Tool definitions are resent on every call. With many rarely used tools, a tool registry loads definitions on demand, at the risk of never retrieving the right one.
  • The model chooses tools, while the harness runs hooks. When a rule can be written as code, enforce it with a hook rather than more prompt text, which builds prompt debt.
  • Look at traces, not just final answers; they are the input to error analysis and evals.
  • Organize harness controls as guides and sensors, and prefer deterministic sensors where possible. Each harness component should fix a named failure.
  • Benchmarks measure a model–harness pair. Hold one fixed when you evaluate the other, and report pass^k for reliability.
  • On Databricks, governed metadata is the best context, deterministic functions and jobs are the best tools, and numbers should come from query results.
  • Expect specific techniques and API names to expire; the separation of concerns is the durable lesson.

22.10 References

Context engineering

Harness engineering

Databricks and text-to-SQL

References

Cormack, Gordon V., Charles L. A. Clarke, and Stefan Büttcher. 2009. “Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods.” Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval. http://cormack.uwaterloo.ca/cormacksigir09-rrf.pdf.