Chapter 14 Missing and Infinite Values

Missing-data handling is a modeling decision, not merely a cleaning command. Before dropping or imputing values, ask why they are missing, whether missingness itself carries information, and whether the proposed treatment can leak outcome or future information.

14.1 Common missing-value representations

pandas recognizes several missing-value markers, including numpy.nan, pandas.NA, None in compatible columns, and pandas.NaT for missing dates. Use isna() and notna() instead of direct equality comparisons.

import numpy as np
import pandas as pd

values = pd.Series([1.0, np.nan, None, 4.0])

assert values.isna().tolist() == [False, True, True, False]
assert values.notna().sum() == 2

NaN does not compare equal to itself, so code such as value == np.nan does not reliably identify missing values.

missing_value = np.nan

assert not (missing_value == missing_value)
assert pd.isna(missing_value)

14.2 Missingness summaries

Measure missingness by column before choosing a treatment. Counts show scale; rates make columns with different lengths easier to compare.

customers = pd.DataFrame(
    {
        "age": [34, np.nan, 52, 41],
        "income": [70000, 85000, np.nan, 62000],
        "segment": ["A", "B", None, "A"],
    }
)

missing_summary = pd.DataFrame(
    {
        "count": customers.isna().sum(),
        "rate": customers.isna().mean(),
    }
)

assert missing_summary.loc["age", "count"] == 1
assert missing_summary.loc["segment", "rate"] == 0.25

14.3 Infinite values are not automatically missing

Positive and negative infinity often arise from division by zero or numerical overflow. They are not equivalent to unknown data and should be investigated before being replaced.

ratios = pd.Series([1.5, np.inf, -np.inf, np.nan])

infinite_mask = np.isinf(ratios)
missing_mask = ratios.isna()

assert infinite_mask.tolist() == [False, True, True, False]
assert missing_mask.tolist() == [False, False, False, True]

If domain knowledge justifies treating infinity as invalid, convert it explicitly and record that choice:

clean_ratios = ratios.replace([np.inf, -np.inf], np.nan)
assert clean_ratios.isna().sum() == 3

14.4 Dropping versus imputing

dropna() is reasonable only when the information loss and selection effect are acceptable. Imputation requires a defensible value and should usually be learned from training data only.

training_age = pd.Series([30.0, 40.0, np.nan, 50.0])
training_median = training_age.median()
imputed_training_age = training_age.fillna(training_median)

assert training_median == 40.0
assert imputed_training_age.isna().sum() == 0

For a predictive workflow, split the data before estimating medians, modes, scalers, encoders, or other learned preprocessing. Reusing a statistic computed from the full dataset leaks information from validation or test rows.

14.5 Nullable dtypes

pandas provides nullable extension dtypes that can represent missing integers, booleans, and strings without coercing integer columns to floating point.

survey = pd.DataFrame(
    {
        "visits": pd.Series([1, None, 3], dtype="Int64"),
        "converted": pd.Series([True, None, False], dtype="boolean"),
    }
)

assert str(survey["visits"].dtype) == "Int64"
assert survey["converted"].isna().sum() == 1

14.6 Interview checklist

  • What process created the missing values?
  • Is missingness informative or systematically related to the outcome?
  • How much data would row or column deletion remove?
  • Was imputation learned only from training data?
  • Are infinite values data errors, valid domain values, or numerical failures?
  • Does the chosen dtype preserve the intended meaning?

14.7 Exercises

  1. Produce a sorted table of missing counts and rates for a DataFrame.
  2. Add an indicator column showing whether income was originally missing.
  3. Explain why filling every numeric column with zero can distort a model.