Chapter 16 Transforming and Recoding Data
Prefer vectorized pandas operations for column transformations. They are usually clearer and faster than manually iterating through rows, and they make the intended rule easier to review.
16.1 Build transformations without overwriting the source
Use assign() or an explicit copy when preserving the input matters.
import numpy as np
import pandas as pd
employees = pd.DataFrame(
{
"employee_id": [1, 2, 3, 4],
"salary": [50000, 80000, 120000, 65000],
"department": ["sales", "engineering", "engineering", "sales"],
}
)
with_bonus = employees.assign(
bonus=lambda frame: frame["salary"] * 0.10,
total_compensation=lambda frame: frame["salary"] + frame["bonus"],
)
assert "bonus" not in employees.columns
assert with_bonus.loc[0, "total_compensation"] == 5500016.2 Mapping known categories
Use map() when one value maps to one replacement. Unmapped values become
missing, so validate coverage or use a deliberate fallback.
department_labels = {
"sales": "Sales",
"engineering": "Engineering",
}
labeled = employees.assign(
department_label=employees["department"].map(department_labels)
)
assert labeled["department_label"].notna().all()For targeted replacements that should leave other values unchanged, use
replace().
16.3 Binning continuous values
Use cut() for explicit numeric intervals. State boundary behavior and labels
so a reader can determine where edge values belong.
salary_band = pd.cut(
employees["salary"],
bins=[0, 60000, 100000, np.inf],
labels=["low", "medium", "high"],
right=False,
)
assert salary_band.astype("string").tolist() == ["low", "medium", "high", "medium"]With right=False, intervals include the left boundary and exclude the right
boundary. A salary of exactly 60,000 therefore belongs to medium.
16.4 Multiple vectorized conditions
numpy.select() expresses mutually ordered conditions. The first true
condition wins, so order overlapping rules from most specific to least specific.
16.5 apply() is not automatically vectorized
Series.apply() can be convenient for a genuinely custom scalar function, but
it still calls Python code repeatedly. First look for built-in arithmetic,
string, datetime, categorical, aggregation, or NumPy operations.
def normalize_department(value):
return value.strip().lower()
raw_departments = pd.Series([" Sales ", "ENGINEERING"])
normalized_departments = raw_departments.str.strip().str.lower()
assert normalized_departments.tolist() == ["sales", "engineering"]The vectorized string methods communicate the operation more directly than an equivalent row-by-row function.
16.6 When row iteration is justified
Avoid iterrows() for transformations because it is slow and may not preserve
row dtypes as expected. If an external side effect truly requires row-wise
iteration, itertuples(index=False) is usually clearer.
messages = [
f"employee={row.employee_id}, department={row.department}"
for row in employees.itertuples(index=False)
]
assert messages[0] == "employee=1, department=sales"Do not use this pattern to replace a vectorized calculation. Side effects such as API calls also require retry, rate-limit, and idempotency design.
16.7 Interview checklist
- Can a built-in vectorized operation express the rule?
- Are all categories mapped, and how are unknown values handled?
- Are bin boundaries explicit and tested?
- Does condition order change the result?
- Is mutation of the source DataFrame intentional?
- Does row iteration exist because of a true side effect or only habit?