Pick the Wrong Metric and You Go Blind

Fifth entry in ML Foundations from First Principles. The hypothesis is the one everyone secretly holds: “accuracy — or AUC — tells you how good the model is.” I re-sat the whole metrics topic as a senior-style mock and rebuilt each piece from its mechanism, and the verdict is FAR FROM IT: a classifier has no single number. It has a confusion matrix, a score, and a dial — and a completely different story under every metric. Pick the wrong one and you don’t just under-measure, you go blind to the failure you shipped.

Everything starts with the confusion matrix (the truth table): the four ways a prediction can land. Using Bouncer Babu — let in (predicted positive) or turn away (predicted negative), crossed with who’s truly a VIP or a gatecrasher:

• ✓TP (true positive) — VIP let in · TN (true negative) — gatecrasher kept out. The two you got right.
• ✗FP (false positive) — gatecrasher sneaked in · FN (false negative) — VIP turned away. The two ways you were wrong.

Here’s the unlock: every metric is just a different slice of these four cells. Click through them:

↑ Click a metric chip and watch the cells it uses light up. Precision is a column, recall is a row, specificity the other row, accuracy the diagonal — and F1 never touches TN.

Two cousins fall straight out of the matrix:

$\text{accuracy} = \frac{TP+TN}{TP+FP+FN+TN} \qquad \text{specificity (TNR)} = \frac{TN}{TN+FP}$

Accuracy — plain: overall fraction correct — is the famous trap: on a 99%-legit stream, “predict legit for everyone” scores 99% accuracy with zero recall. The metric applauds a useless model, because the giant TN cell drowns everything. That’s why accuracy is the first number to distrust on imbalance. Specificity — plain: of the real negatives, how many you correctly cleared — is recall’s mirror, and FPR = 1 − specificity (remember that; it’s ROC’s x-axis).

A classifier hands you a score, not a verdict; a threshold (the dial) turns it into a decision. Two cells flip as you move it:

$\text{precision} = \frac{TP}{TP+FP} \qquad \text{recall} = \frac{TP}{TP+FN}$

Lower the dial → flag more → recall climbs, precision slips. Which way you lean is set by cost asymmetry: fraud → a miss (FN) is costly → recall-first; spam → a real email lost (FP) is costly → precision-first. But “lean toward recall” is hand-wavy — the dial is actually a cost calculation:

↑ Set the cost of a false alarm vs a miss; the green line is the cost-minimizing dial. Crank up the cost of a miss (FN) — fraud — and the optimal threshold slides left, toward recall. That’s how you pick an operating point, not by vibes.

One guardrail: optimise one metric subject to a floor on the other, or it goes degenerate — “100% recall” is free if you flag everyone.

Instead of one dial setting, sweep all of them. Plot catch rate (TPR) against sneak-in rate (FPR = FP/(FP+TN)) as the dial sweeps — that’s the ROC curve, from (0,0) “let nobody in” to (1,1) “let everybody in.” AUC (the area) = the chance Babu scores a random VIP above a random gatecrasher — his “eye,” independent of the dial.

↑ Slide the dial — people cross IN/OUT, the dot slides on both curves. Now hit MOBBED: gatecrashers swamp the VIPs.

Watch MOBBED. The gatecrashers (TN) become an enormous pile, so even hundreds of sneak-ins barely move FPR = FP/(FP+TN) — ROC stays near-perfect — while inside the club those sneak-ins swamp the few VIPs, so precision craters. ROC hides the false positives in a giant TN pile; precision has no TN, so it tells the truth. On rare-positive problems report PR-AUC (baseline = the prevalence, not 0.5), not ROC-AUC.

To collapse precision and recall into one number, use the harmonic mean:

$F_1 = \frac{2PR}{P+R} \qquad\qquad F_\beta = \frac{(1+\beta^2)\,PR}{\beta^2 P + R}$

The harmonic mean is the weakest-link average — dragged toward whichever of P/R is worse. Precision 1.0, recall 0.0 → F1 = 0 (correctly damning), where the plain average would say a comfy 0.5. Fβ tilts it: F2 for recall (fraud), F0.5 for precision (spam). And note from §03: F1 never uses TN, which is exactly why — like PR-AUC — it survives imbalance where accuracy dies.

Go multi-class — a moderation model across 14 languages. Compute a per-class F1, then squash them into one number, and how you squash decides what you can see. Macro averages per-class equally (a tiny language counts as much as a giant); micro pools every prediction into one score — which for single-label is just accuracy — and is dominated by the big classes.

↑ Drag tiny Assamese’s F1 to the floor. MACRO drops and rings the alarm; WEIGHTED / micro barely move. Same model, same failure, opposite verdicts.

So when minority classes matter, report macro-F1 plus per-class precision/recall. A shiny micro-F1 on imbalanced classes is accuracy wearing a lab coat.

MCC (Matthews correlation) is the one that uses all four cells at once — the most honest single number on imbalanced data (+1 perfect, 0 coin-flip, −1 inverted). When someone demands “just give me one metric,” MCC is the least misleading choice.

⚠ Caveat — they all grade decisions, not trust. Every metric here scores a thresholded decision or a ranking. None checks whether the probability itself is believable. If you act on the score — expected-value thresholds, cost math, downstream models — you also need calibration: a predicted 0.9 should be right about 90% of the time (reliability curve / Brier score). A model can have a perfect AUC and badly miscalibrated probabilities. That’s its own experiment — coming later.

These metrics grade one prediction. But a retriever hands back a ranked list — and “is the right chunk in there?” isn’t enough when the LLM only reads the top. Next: MRR and NDCG, and why a chunk being retrieved doesn’t mean it helped (EXP-012). Calibration — whether the probabilities are trustworthy — gets its own experiment after that.