Classification Lab
Interactive companion to Classification and Lookalike Modeling. Every number on this page is computed in your browser from the inputs you set. The inputs are illustrative; no real model or vendor API is called.
How confidence summarizes a probability distribution
Pretend a typed decision model returned these probabilities. Change them and watch the confidence formulas from the TypeSafe documentation respond. Weights are normalized to sum to 1.
Inputs
Result
Confidence
Is "0.95" really 95%? Calibration and cost-based thresholds
This simulates a shadow-mode log. Each case has a true probability of being positive. The model reports a probability that can be pushed toward 0 and 1 (overconfident) or toward 0.5 (underconfident). The reliability diagram compares what the model reports with what actually happens.
What the threshold delivers
| Cases above threshold | – |
| Average reported probability | – |
| Actual positive rate | – |
| Brier score (lower is better) | – |
Choose the threshold from business costs
With calibrated probability p, act when p × CFN > (1 − p) × CFP, that is, when p > CFP / (CFP + CFN).
Lookalike audiences: ranking versus sizing
A population of 1,000,000 people. Each person has a one-dimensional "customer-likeness" score (think of it as a summary of all features). True customers and non-customers have overlapping bell curves. The seed audience is a random share of true customers (the SCAR assumption). Results are computed exactly from the curves, not from a random sample.
Results
| Seed size | – |
| True high-value people | – |
| Naive model implies | – |
| PU-corrected estimate | – |
| True c vs estimated ĉ | – |
| AUC (naive and corrected) | – |
Why the estimate of c is biased when groups overlap
The Elkan–Noto estimate is the average naive score among seed members. Under SCAR that equals c × (average probability of being high value, among true high-value people). If true customers are unmistakable, that average is near 1 and the estimate is right. When groups overlap, many true customers look ambiguous, the average falls below 1, c is underestimated, and the audience size is overstated. The ranking (AUC) never changes, because dividing every score by the same number keeps their order.