3 Experimentation and Incrementality
Incrementality asks what changed because of a marketing action. The missing quantity is the outcome the treated population would have had without the treatment. Random assignment makes that counterfactual credible on average.
3.1 Define the estimand
Suppose treatment is eligibility for an email offer. Let \(Y(1)\) be a user’s outcome when eligible and \(Y(0)\) the outcome otherwise. A common target is the average treatment effect:
\[ ATE = E[Y(1) - Y(0)] \]
The exact estimand must specify:
- the eligible population;
- assignment versus actual exposure;
- the outcome and follow-up window;
- the unit of randomization; and
- whether effects are reported as absolute or relative changes.
For conversion rates \(p_T\) and \(p_C\):
\[ \text{absolute lift} = p_T - p_C \]
\[ \text{relative lift} = \frac{p_T-p_C}{p_C} \]
Always report the absolute effect. Relative lift can look large when the baseline is small.
3.2 Design before launch
- Choose one primary outcome and a small set of guardrails.
- Select the randomization unit: user, household, store, or geography.
- Define eligibility before assignment.
- Estimate baseline rate, minimum detectable effect, power, and duration.
- Freeze stopping and exclusion rules.
- Verify assignment logging and treatment delivery.
Randomize at a level that limits interference. User-level randomization may be invalid when household members share offers or when a geographic campaign changes market-level behavior.
3.3 Sample-size illustration
This calculation assumes independent observations and a two-sided test. It is a planning approximation, not a substitute for simulation when outcomes are clustered or repeatedly measured.
power.prop.test(
p1 = 0.080,
p2 = 0.088,
power = 0.80,
sig.level = 0.05,
alternative = "two.sided"
)##
## Two-sample comparison of proportions power calculation
##
## n = 18871.45
## p1 = 0.08
## p2 = 0.088
## sig.level = 0.05
## power = 0.8
## alternative = two.sided
##
## NOTE: n is number in *each* group3.4 Analyze effects with uncertainty
The following counts are synthetic. The estimated effect should be accompanied by a confidence interval and the raw group rates.
conversions <- c(treatment = 889, control = 801)
assigned <- c(treatment = 10000, control = 10000)
rates <- conversions / assigned
test <- prop.test(conversions, assigned, correct = FALSE)
data.frame(
group = names(rates),
conversions = as.integer(conversions),
assigned = as.integer(assigned),
rate = as.numeric(rates)
)## group conversions assigned rate
## 1 treatment 889 10000 0.0889
## 2 control 801 10000 0.0801c(
absolute_lift = unname(diff(rev(rates))),
relative_lift = unname(rates["treatment"] / rates["control"] - 1),
p_value = test$p.value
)## absolute_lift relative_lift p_value
## 0.00880000 0.10986267 0.02527148## [1] 0.001091564 0.016508436
## attr(,"conf.level")
## [1] 0.95Statistical significance does not guarantee business value. Translate the effect interval into incremental orders, margin, operational capacity, and risk.
3.5 Essential diagnostics
3.5.1 Sample-ratio mismatch
If a 50/50 experiment receives materially different group sizes, investigate assignment, logging, eligibility, and missing data before interpreting the outcome.
##
## Chi-squared test for given probabilities
##
## data: observed
## X-squared = 11.52, df = 1, p-value = 0.00068853.5.2 Repeated peeking
Stopping an ordinary fixed-horizon test when the p-value first crosses 0.05 inflates false positives. Use the planned horizon or a valid sequential design.