Chapter 13 Reading Data
Reliable analysis begins by defining the expected schema and unit of observation. Reading a file successfully does not prove that columns have the right types, missing-value rules, encoding, or row-level meaning.
13.1 Read a local CSV reproducibly
Use a project-relative path and inspect a small sample before building later steps around the data.
from pathlib import Path
import pandas as pd
college_path = Path("__REPO") / "data" / "college.csv"
college = pd.read_csv(college_path)
assert college_path.exists()
assert college.shape[0] > 1000
assert college["id"].is_uniqueAvoid absolute paths tied to one computer. Functions should accept a path or receive a configured data directory rather than silently depending on the current user’s Desktop.
13.2 Specify important dtypes
Identifiers may look numeric while functioning as labels. Define important types during input when inference could change meaning or lose formatting.
college_typed = pd.read_csv(
college_path,
dtype={
"id": "string",
"state": "string",
"region": "category",
},
)
assert str(college_typed["id"].dtype).startswith("string")
assert str(college_typed["region"].dtype) == "category"For large files, usecols can avoid reading columns that are not needed.
nrows is useful for inspecting structure, but a development sample must not
be confused with the final analytical population.
13.3 Define missing-value markers deliberately
Source systems often use strings such as NA, NULL, -999, or an empty
field. Treat a sentinel as missing only when the data documentation supports
that interpretation.
from io import StringIO
csv_text = "customer_id,income\n1,50000\n2,UNKNOWN\n3,72000\n"
income = pd.read_csv(StringIO(csv_text), na_values=["UNKNOWN"])
assert income["income"].isna().sum() == 1After reading, validate column presence, types, key uniqueness, allowed ranges, and row counts before continuing.
13.4 Parse dates explicitly
event_text = "event_id,event_date\n1,2026-01-15\n2,2026-02-01\n"
events = pd.read_csv(StringIO(event_text), parse_dates=["event_date"])
assert str(events["event_date"].dtype).startswith("datetime64")
assert events["event_date"].is_monotonic_increasingSpecify a format with pd.to_datetime(..., format=...) when the source format
is fixed or ambiguous. Decide explicitly how invalid dates should be handled;
silently coercing them to missing values can hide a source-data defect.
13.5 Fixed-width records
Some legacy sources encode fields by character position rather than delimiters.
read_fwf() can parse them when widths or column specifications are known.
fixed_width_text = "001Alice 090\n002Grace 095\n"
scores = pd.read_fwf(
StringIO(fixed_width_text),
widths=[3, 10, 3],
names=["student_id", "name", "score"],
dtype={"student_id": "string"},
)
assert scores["name"].tolist() == ["Alice", "Grace"]
assert scores["score"].tolist() == [90, 95]Fixed-width layouts require authoritative metadata. Do not guess field positions from a handful of rows.
13.6 Excel and columnar formats
read_excel()reads Excel worksheets and requires the relevant Excel engine.- Parquet usually preserves types better than CSV and is efficient for analytical tables, but producers and consumers must still agree on schema.
- JSON can represent nested records; normalize deliberately rather than assuming every JSON document is tabular.
Examples should ship with the required local data or be marked non-executable. A public book build should not depend on a private SharePoint folder, another person’s home directory, or an unstable HTTP download.
13.7 Interview checklist
- What is one row, and which columns form the key?
- Which dtypes must be specified rather than inferred?
- Which sentinel values genuinely mean missing?
- Are dates and time zones parsed correctly?
- Is the path portable and the source accessible to other users?
- What schema and row-count checks run immediately after input?