Chapter 6 Missing and Non-finite Values
Missingness is part of the data-generating process, not merely a cleaning inconvenience. Before removing or imputing values, ask why they are absent and whether that absence contains information.
6.1 NA, NaN, and infinity
R distinguishes missing values from non-finite numerical results.
special_values <- c(10, NA_real_, NaN, Inf, -Inf)
is.na(special_values)
#> [1] FALSE TRUE TRUE FALSE FALSE
is.nan(special_values)
#> [1] FALSE FALSE TRUE FALSE FALSE
is.infinite(special_values)
#> [1] FALSE FALSE FALSE TRUE TRUE
is.finite(special_values)
#> [1] TRUE FALSE FALSE FALSE FALSEis.na() is also TRUE for NaN, while is.nan() identifies only undefined
numeric results such as 0 / 0.
6.2 Missingness propagates
R does not silently discard missing values from most summaries.
observed_scores <- c(72, 85, NA, 91)
mean(observed_scores)
#> [1] NA
mean(observed_scores, na.rm = TRUE)
#> [1] 82.66667Using na.rm = TRUE changes the population being summarized. Make that choice
deliberately and report how many observations were excluded.
6.3 Row completeness
complete.cases() identifies rows without missing values across selected
variables.
applications <- data.frame(
applicant_id = 1:5,
income = c(60, 75, NA, 90, 68),
debt = c(20, NA, 25, 30, 18),
approved = c(TRUE, FALSE, TRUE, TRUE, FALSE)
)
analysis_columns <- c("income", "debt", "approved")
complete_rows <- complete.cases(applications[analysis_columns])
applications[complete_rows, ]
#> applicant_id income debt approved
#> 1 1 60 20 TRUE
#> 4 4 90 30 TRUE
#> 5 5 68 18 FALSEComplete-case analysis is valid only under assumptions about why values are missing. It can reduce precision and introduce bias when missingness is related to the outcome or predictors.
6.4 Missingness mechanisms
Three common conceptual categories are:
- MCAR: missingness is unrelated to observed or unobserved values.
- MAR: after conditioning on observed information, missingness does not depend on the missing value itself.
- MNAR: missingness still depends on unobserved information or the missing value after conditioning on observed data.
These mechanisms are assumptions about the process, not labels that can usually be proven from the dataset alone.
6.5 Simple imputation
Median imputation can be a reasonable baseline for a numeric predictor, but it reduces variation and does not automatically remove bias.
training_income <- c(55, 62, 70, NA, 88)
training_median <- median(training_income, na.rm = TRUE)
impute_median <- function(x, value) {
x[is.na(x)] <- value
x
}
impute_median(training_income, training_median)
#> [1] 55 62 70 66 88For predictive modeling, estimate the imputation value using the training data only, then apply that saved value to validation, test, and future data. Fitting preprocessing on the full dataset leaks information across the evaluation boundary.
6.6 Missing indicators
Sometimes the fact that a value is missing is predictive. A missingness indicator can preserve that signal, but it does not eliminate the need to understand the missingness process.
applications$income_missing <- is.na(applications$income)
applications$income_imputed <- impute_median(
applications$income,
median(applications$income, na.rm = TRUE)
)
applications[c("applicant_id", "income_missing", "income_imputed")]
#> applicant_id income_missing income_imputed
#> 1 1 FALSE 60.0
#> 2 2 FALSE 75.0
#> 3 3 TRUE 71.5
#> 4 4 FALSE 90.0
#> 5 5 FALSE 68.0