B Agentic AI Labs: Walkthroughs
The course Agentic AI (DeepLearning.AI, Andrew Ng) pairs each design pattern with hands-on labs. This appendix records what each lab built and what it teaches, chapter by chapter. It is written in my own words. The course notebooks are not reproduced: the code blocks below are short sketches of the idea a lab teaches, not runnable copies of the lab.
The graded assignments are described by the concept they practise only. No solutions or exercise answers appear here.
About the results. Outcomes below are as described in the course labs, not reproduced here. The labs call hosted models, so each run can give a different answer, and the model names they use (GPT-4o, GPT-4.1, o4-mini, 2025) will date quickly. Read the outcomes as illustrations of a pattern, not as benchmarks.
B.1 Common setup
| Item | Choice |
|---|---|
| Model access | aisuite, a thin wrapper that calls models from several providers through one OpenAI-style chat interface |
| Models | OpenAI models of 2025: a fast model for drafting, a stronger reasoning model for critique |
| Data and services | Small local datasets (a coffee-sales CSV, a SQLite table of product events), a mock email server, and free public APIs for location and weather |
| Code execution | Generated Python is run with exec() inside the notebook. Fine for a lab, not for production (Section 9.4.1) |
| Output | Charts saved as PNG files; tables shown as data frames |
B.2 Reflection labs
These labs go with Chapter 8.
B.2.1 Lab: chart generation with multimodal reflection
Goal. Turn a plain-English request into a chart, then improve the chart by letting a multimodal model look at the rendered image (Section 8.4).
Steps:
- Generate code (V1). Ask a model for matplotlib code that answers a
request such as “compare first-quarter sales in two years”. The prompt
lists the data frame’s columns with their types and asks for the code
inside
<execute_python>tags, saving the chart to a fixed file name. - Run it. Extract the tagged code and execute it with only the data frame in scope. The result is a PNG.
- Reflect on the image. Send the PNG, the V1 code, and the original request to a vision-capable model. It returns a one-line JSON critique, then revised code in the same tags.
- Run V2 and compare the two charts.
- Wrap it as one workflow with separate model choices for generation and reflection.
The extraction and execution step is a few lines:
import re
CODE = re.compile(r"<execute_python>(.*?)</execute_python>", re.DOTALL)
def run_tagged_code(llm_output: str, df) -> None:
match = CODE.search(llm_output)
if match is None:
raise ValueError("no tagged code in the model's reply")
exec(match.group(1), {"df": df}) # lab only: no isolation, no time limitThe critic’s instructions, paraphrased:
You are reviewing a chart made for this request: {request}
The chart image is attached; the code that produced it is below.
Line 1: JSON {"feedback": "<what to fix and why>"}.
Then: revised code inside <execute_python> tags that saves to {path_v2}.
Lessons:
- The image is the external feedback. The critic sees what a reader would see (overlapping labels, a misleading chart type) rather than guessing from the code.
- Split the models by job. A cheap, fast model drafts; a stronger reasoning model critiques. The critique step is where judgement pays.
- The critic may change the chart type, for example from bars to lines. That is a judgement call, not a bug, which is why chart quality should be scored with a rubric across many requests (Section 8.7), not by looking at one example.
- Watch the prompt for patches. The lab’s prompts carry several emphatic “CRITICAL” rules about date types, each fixing a crash seen in earlier runs. Stating the schema clearly up front is the durable fix; a growing list of emphatic rules is the too-specific end of Section 6.6.
- Parse defensively. The lab’s code tries to read the JSON line, then falls back to searching for any JSON object in the reply. Models do not always follow output formats exactly.
B.2.2 Lab: SQL generation with reflection
Goal. Show that reviewing a query’s text is not enough. The critic also needs to see what the query returned (Section 8.3).
Data. One SQLite table of product events, not product states. Each row is an insert, restock, sale, or price update. The quantity change is positive when stock comes in and negative for a sale. Total sales must be derived from the sale events.
Steps and outcome, for the question “Which product colour has the highest total sales?”:
| Version | What the critic sees | Outcome described in the lab |
|---|---|---|
| V1: generate only | No critic | Valid SQL that sums quantity × price over the events; the “total sales” comes out negative |
| V2a: reflect on the text | Question, schema, V1 SQL | The query looks reasonable on paper; the total is still negative |
| V2b: reflect on the result | The same, plus V1’s output table | The critic notices the negative total and fixes the sign; the answer becomes positive |
A typical V1 and a robust version:
-- V1: runs, but sale rows carry negative quantities
SELECT color, SUM(qty_delta * unit_price) AS total_sales
FROM transactions
GROUP BY color
ORDER BY total_sales DESC
LIMIT 1;
-- Robust: keep only sale events, then flip the sign
SELECT color, SUM(-qty_delta * unit_price) AS total_sales
FROM transactions
WHERE action = 'sale'
GROUP BY color
ORDER BY total_sales DESC
LIMIT 1;A fix that only wraps the quantity in ABS() would still be wrong: insert
events also have positive quantities and a price, so they would be counted as
sales. Filtering on the event type is what makes the query mean “sales”.
The difference between V2a and V2b is a single input to the critic:
def critique_sql(question, schema, sql, result=None):
prompt = f"Question: {question}\nSchema:\n{schema}\nSQL:\n{sql}\n"
if result is not None: # external feedback
prompt += (f"It returned {len(result)} rows. First rows:\n"
f"{result.head(20).to_markdown()}\n")
prompt += ('Does the result answer the question? Return JSON '
'{"feedback": "...", "sql": "<final query>"}')
return llm(prompt)Lessons:
- Semantic errors hide in valid SQL. A sign convention is invisible in the query text; a negative “total sales” is obvious in the result.
- Execution output is cheap external feedback. One query run gives the critic evidence that no amount of re-reading provides.
- Pass a sample of the result, not all of it. The lab passes the whole output table, which is fine for a small aggregate. For large results, pass the row count and the first rows to keep the context small.
- Fail safe. If the critic’s reply cannot be parsed, the lab keeps the original query rather than running something half-read.
- Better context prevents the error. A schema note such as “
qty_deltais negative for sales” would let V1 get it right. Reflection catches what the context failed to say, but fixing the context is cheaper.
B.2.3 Graded assignment: reflective essay writing
The graded lab builds a three-call writing workflow: draft an essay on a topic, critique the draft (optionally with a different, reasoning model) for structure, clarity, argument, and style, then revise the draft using that critique. It practises keeping the three roles in separate calls with explicit inputs, which is the basic pattern of Section 8.1. There is no external feedback, so the gains are the modest ones reported for self-critique alone (Section 8.2). It is also the first building block of the research agent that later graded labs extend.
B.3 Tool-use labs
These labs go with Chapter 9.
B.3.1 Lab: turning functions into tools
Goal. Give a model access to ordinary Python functions, first letting the library run the tool loop, then running it by hand (Section 9.2).
Steps:
- One function, automatic mode. Define a function that returns the
current time, with a one-line docstring. Pass the function itself in the
tool list with a turn limit.
aisuitebuilds the schema from the signature and docstring, runs the function when the model asks, and sends the result back. - Read the trace. The response holds the whole exchange: the model’s tool request, the local result, and the final answer that uses it.
- Manual mode. Write the schema by hand (name, description,
parameters) and drop the turn limit. Now your code must check for a tool
request, run the function, append both the model’s request and a
toolmessage carrying the matching call ID, and call the model again. - Several tools. Add three more: current weather (IP location, then a weather API), write a text file, and make a QR code with an embedded logo. For each request the model picks the one tool that fits.
- A multi-step request. “Make a QR code for this site, and write me a note with the current weather.” The model must fetch the weather before writing the note, whatever order the request names them in.
The two modes differ in who owns the loop:
| Automatic | Manual | |
|---|---|---|
| You pass | Python functions | JSON schemas |
| Who runs the tool | The library, up to the turn limit | Your code |
| Good for | Prototypes and demos | Logging, validation, approval steps, custom error handling |
A tool written for a model has to expect sloppy arguments. The lab’s QR tool strips quotes and backticks from file names and falls back to a plain code if the logo is missing. The same habits in a note-writing tool:
from pathlib import Path
NOTES_DIR = Path("notes").resolve()
def write_note(name: str, content: str) -> str:
"""Save a short text note in the notes folder. Overwrites a note with the same name."""
name = name.strip().strip("`'\"") # models sometimes quote names
path = (NOTES_DIR / name).with_suffix(".txt").resolve()
if NOTES_DIR not in path.parents: # refuse paths outside the folder
return "Error: give a plain note name, not a path."
NOTES_DIR.mkdir(exist_ok=True)
path.write_text(content, encoding="utf-8")
return f"Saved {path.name}"Lessons:
- The docstring is the prompt. It is all the model sees when choosing a tool and filling its arguments (Section 9.3).
- Automatic mode hides the loop. Fine for a demo. In production you want the manual loop, because that is where logging, argument checks, and approval steps go.
- Return errors as text the model can act on. A clear message such as “give a plain note name” lets the model retry; an exception just ends the run.
- Limit what a tool can touch. The lab’s file tool writes to any path it is given. Fixing the folder, as above, is least privilege applied to one tool (Section 9.6).
- The model orders calls by data dependency, not by the wording of the request. Read the trace to confirm it did.
B.3.2 Lab: email assistant with tools
Goal. Run an agent against a simulated inbox and see that the tool list sets what the agent can and cannot do.
Setup. A local mock email service (a small REST API over SQLite, loaded with a handful of messages) and ten thin Python wrappers, one per endpoint: list all, list unread, search, filter, get one, mark read, mark unread, send, delete, and unread-from-sender. Each request is wrapped in a short preamble: you manage email, use the tools, my address is …, and never ask for confirmation. The lab tests every wrapper by hand before any model is involved.
Runs, as described in the lab:
| Request | Tools given | Outcome |
|---|---|---|
| Check unread mail from my boss, mark it read, send a polite follow-up | Six tools, no delete | Unread-from-sender → mark read → send, then a short summary |
| Delete Alice’s email | The same six | Cannot complete the action; no tool can delete |
| Delete Alice’s email | Plus delete_email |
Finds the message, then deletes it |
| Delete the Happy Hour email | All seven | Searches by subject to get the ID, then deletes |
The lab runs with no human check. A production version would put one in front of every send and delete:
NEEDS_APPROVAL = {"send_email", "delete_email"}
def run_tool(name, args, tools, approve):
if name in NEEDS_APPROVAL and not approve(name, args):
return {"status": "declined", "reason": "the user did not approve this action"}
return tools[name](**args)Lessons:
- The tool list is the capability boundary. An action with no tool cannot happen. Leaving a tool out is the cheapest safety control there is.
- Thin wrappers, one job each. Each tool maps to one endpoint and returns compact JSON the model can chain. This is the pattern MCP standardises (Section 9.5).
- “Never ask for confirmation” belongs only in a sandbox. This agent reads private mail, reads mail anyone can write, and can send: all three legs of the lethal trifecta in Section 9.6.
- Describe, then resolve, then act. “The Happy Hour email” must become an ID before anything is deleted. If the search returns more than one match, that is the moment to stop and ask.
- Test the tools before the agent. Checking each wrapper by hand first is a component-level check (Section 10.5). A failure later is then about the model, not the plumbing.
B.3.3 Graded assignment: research report with tools
The graded lab extends the essay workflow of Module 2 into a small research pipeline. A tool-using call writes a report with access to two search tools, one for arXiv papers and one for the web. The tool loop is written by hand: a dictionary maps tool names to functions, and the loop stops at a turn limit (Section 9.2.2). A second call reflects on the report, returns a structured critique and a revised report as JSON, and a third call formats the result as HTML. It practises the manual tool protocol, reflection on a draft, and parsing structured output. The Module 5 graded lab turns these steps into a planned multi-agent pipeline.
B.4 Evaluation lab
This lab goes with Chapter 10.
B.4.1 Lab: a component-level eval for web search
Goal. Score one step of the Module 3 research agent, the search step, on its own instead of rerunning search, drafting, and reflection every time (Section 10.5).
Steps:
- Split out the research step. A single function gives the model three search tools (arXiv, web search, Wikipedia), lets the library run the tool loop, and returns the model’s list of sources. The prompt includes today’s date, which matters for requests such as “recent developments”.
- Define the reference. A hand-made set of preferred domains: journals, universities, agencies, and AI venues.
- Score. Pull every URL out of the output, mark each as preferred or not, and compute the share that is preferred. Pass if the share is at least 40%. Return the flag and a short Markdown report listing each URL.
- Try it yourself. Change the topic, the domain list, or the threshold and rerun.
In the terms of Section 10.2, this is a code-graded eval with a per-example reference: the right domain list depends on the topic. It measures something close to precision (how much of what came back is trustworthy) and says nothing about recall (whether the important sources were found).
Two details in the lab’s scoring are worth fixing. It matches a domain if the
domain’s text appears anywhere in the host name, so summit.edu counts as
mit.edu. It also counts a URL each time the output repeats it. A tighter
version:
import re
from urllib.parse import urlparse
URL = re.compile(r"https?://[^\s)\]>}\"']+")
def on_list(host: str, domains: set[str]) -> bool:
return any(host == d or host.endswith("." + d) for d in domains)
def preferred_share(text: str, domains: set[str]) -> float | None:
urls = set(URL.findall(text)) # count each URL once
if not urls:
return None # no sources: report it, don't score it
hits = sum(on_list(urlparse(u).hostname or "", domains) for u in urls)
return hits / len(urls)Lessons:
- Evaluate the step you are changing. Search quality can be compared across providers, result counts, or prompts in seconds, without the noise of the drafting and reflection steps.
- Decide what the component’s output is. The lab scores the URLs the model wrote, not the results the search tool returned. These can differ: the model may drop good results or cite a URL it never retrieved. The tool results in the trace (Section 10.3) are the cleaner input, and checking that every cited URL appears in them is a second, cheap eval.
- One topic and one run is an anecdote. Build ten or more topics, each with its own domain list, and average the score. Search results and model choices vary between runs.
- A trusted domain is a proxy, not relevance. An off-topic page on a university site still passes. Pair the domain check with precision and recall against sources an expert chose.
- Confirm the gain end to end. A better search score should lead to better reports. Check that it does before moving on.
B.5 Planning and multi-agent labs
These labs go with Chapter 11.
B.5.1 Lab: customer service agent that plans in code
Goal. Let the model write one Python program that is the plan: it filters, computes, and updates in a single pass, instead of emitting a JSON plan for many small tools (Section 11.1.5).
Setup. A sunglasses store in TinyDB, a small JSON-backed document database. One table holds the inventory (ID, name, description, stock, price); the other is a transaction log that starts with an opening balance. The code that runs the plan can see the two tables and two helpers (the current balance, the next transaction ID).
Steps:
- Describe the data. Build a schema block from the live tables: fields, types, and sample rows.
- Prompt for code. The prompt asks for commented Python inside
<execute_python>tags and sets the business rules: derive every filter from the request, treat unclear intent as read-only, write one transaction per item, change nothing if any item is short of stock, always set a customer-facinganswer_textand aSTATUScode. - Execute the extracted code with only the tables and helpers in its namespace, capturing printed logs and errors, and show the tables before and after.
- Wrap it as one agent and try reads, returns, and purchases, including a request for two different items.
Runs, as described in the lab:
| Request | Kind | Outcome |
|---|---|---|
| Any round sunglasses in stock under $100? | Read | One match, the Classic at $60; nothing changed |
| Return 2 Aviator sunglasses I bought last week | Write | A return transaction is added; Aviator stock goes from 23 to 25 |
The prompt asks the model to avoid partial writes, but nothing in the executor enforces it. Running the plan on a copy makes “all or nothing” a property of the code, and gives a natural place for approval:
import copy
def run_plan(code: str, store: dict, approve) -> dict:
"""Run a generated plan on a copy; return the store to keep."""
trial = copy.deepcopy(store) # the plan only touches a copy
scope = {"store": trial}
try:
exec(code, scope) # still not isolated: see the lessons
except Exception as err:
print("plan failed, nothing written:", err)
return store
changed = [k for k in store if store[k] != trial[k]]
if changed and not approve(changed, scope.get("answer_text")):
return store # declined: nothing written
return trialLessons:
- One program replaces a chain of tool calls. Filtering by shape and price, checking stock, and writing a transaction are a few lines of code, and the model already knows the library.
- A restricted namespace is not a sandbox. Python adds its built-ins to
any globals dictionary passed to
exec(), so the generated code can stillimport os. Isolation needs a separate process or container (Section 9.4.1). - Rules in the prompt are requests; rules in the executor are guarantees. “No partial writes” and “read-only when unsure” matter enough to enforce in code, as above.
- Check the business logic, not only the code. The return went through although the transaction log held no sale to return: the plan checked the product, not the purchase history. This is the kind of error that error analysis (Section 10.4) finds and a schema note or rule fixes.
B.5.2 Lab: a market research team
Goal. Chain four role-specific agents into a pipeline that turns live fashion trends and a product catalog into a short campaign report. It is the campaign-launch team of Section 11.2.2, built in its simplest, linear form.
| Agent | Input | Model and tools | Output |
|---|---|---|---|
| Market researcher | The brief, with today’s date | Reasoning model; web search and catalog tools, in a hand-written tool loop | 2–3 trends, matching products, reasons |
| Graphic designer | Trend summary | Text model writes an image prompt and caption as JSON; an image model draws the picture | Image file, prompt, caption |
| Copywriter | Image and trend summary | Vision model | Quote of at most 12 words, with a justification, as JSON |
| Packager | All of the above | Text model rewrites the summary for executives | Markdown report |
Outcome. Two full runs are saved with the lab. Given the same brief, the first chose the Wayfarer and Sport as lead products and the second chose the Mystique, Round, and Aviator.
Lessons:
- A fixed order is a design choice. The developer, not a model, decided the sequence. That makes this the linear pattern of Section 11.2.3: predictable, and enough for a process whose steps are known.
- The same brief gives different answers. Live search results and sampling both vary. Before acting on a recommendation, run the pipeline several times, or save the search results and reuse them so the runs can be compared.
- Agents optimise what the prompt asks for. The catalog lists stock, but the prompt never mentions it, so both runs recommended the Mystique, which has 3 units left. A lead product for a campaign needs inventory behind it.
- Every rewrite can add claims. In the second run the executive rewrite added points about premium pricing, add-on sales, and staying ahead of competitors that no earlier step had produced. Either let the last step format without rewriting, or check the final report against the earlier outputs.
- Keep the tool loop tidy. The researcher’s loop has no turn limit, and it appends the model’s message once per tool call instead of once per turn. If the model asks for two tools at once, the conversation history becomes invalid. The loop in Section 9.2.2 avoids both problems.
B.5.3 Graded assignment: a planned research workflow
The final graded lab completes the research agent built across the course. A planner agent turns a topic into a list of short steps. An executor goes through the steps in order. For each one it picks the agent to do it (researcher, writer, or editor), passes along the outputs of earlier steps, and records the result. By default it runs only the first few steps to keep run time down. The researcher uses the search tools from Module 3, and the editor plays the reflection role from Module 2. The lab practises plan, then execute (Section 11.1.2) and the hierarchical pattern, where a coordinator routes work to specialists (Section 11.2.3).