Introduction to Classification Evaluation Indicators: Accuracy, Precision, Recall, and F1. What are they really looking at

Topic

• Examine accuracy, precision, recall, and F1 not just as formulas, but as lenses on classifier behavior, especially under class imbalance.
• Central question: are these metrics properties of a model, or of a deployment involving model, threshold, population, costs, feedback, and labels?

Key Points

• Accuracy measures overall correctness. It is misleading when one class dominates: in fraud detection where 99.8% of transactions are normal, an all-normal classifier scores 99.8% accuracy but catches zero fraud. In rare-disease screening, 99% accuracy may mean missing most patients.
• Accuracy is still useful as a trivial baseline, but only with baselines, confidence intervals, and raw counts.
• Precision asks: of predicted positives, how many are correct? It depends on prevalence and is sensitive to false alarms and population shift.
• Recall asks: of actual positives, how many are found? It matters when false negatives are costly; low fraud recall means blindness to rare positives.
• F1 is the harmonic mean of precision and recall. It ignores true negatives and weights precision and recall equally. It is compact, but hides contested tradeoffs and prevalence effects.
• Metrics describe a triple: model + threshold + population. The same classifier can show different precision and recall in a different hospital or fraud season.
• Metrics feed back: once recall is targeted, thresholds are tuned; a successful fraud detector may deter fraud, lowering prevalence and precision, so success can look like failure.
• Precision and recall grew up in information retrieval, where true negatives are effectively uncountable. F1’s blind spot is habitat-specific, not purely ethical.
• Ground truth can be manufactured by the system: fraud labels come from investigations often triggered by alerts; screening labels come from biopsies after positive screens. Evaluation may be least trustworthy where the model matters most.
• Better evaluation includes costs, calibration, raw counts, alert volume, missed fraud dollars, alert fatigue, holdout periods, randomized controls, and feedback loops.

Main Disagreements

• Accuracy: the enthusiast says it is only useful with balanced classes and symmetric costs. The skeptic says it answers a narrow question and is not misleading alone; reporting it without baseline and counts is the error.
• F1: the enthusiast sees it as a practical alarm bell. The skeptic sees false comfort: it collapses contested tradeoffs into one optimizable number and ignores true negatives.
• Cost judgments: the skeptic argues that favoring false negatives over false positives is a cost decision, not a metric fact. Precision and recall only slice the confusion matrix.
• Decision-aware evaluation: the enthusiast wants calibrated probabilities plus decision impact and feedback loops. The skeptic demands counterfactual evidence, noting dashboards can be gamed, labels are delayed, and calibration fails after policy changes.
• Ground truth: the observer says the confusion matrix assumes external labels, but labels often echo the model and institution. Others focus on deployment and feedback; the observer says the yardstick itself is instrument-dependent.
• Model vs deployment: the enthusiast partly accepts the triple but defends metrics. The observer insists the object is the trajectory of a deployment, not a static classifier.

Conclusion

• No metric is inherently “what matters.” Accuracy measures overall correctness; precision measures positive predictive reliability; recall measures positive coverage; F1 balances them while ignoring true negatives.
• Under imbalance, accuracy can be vacuous. Precision and recall reveal error types but depend on prevalence and threshold. F1 summarizes, but does not decide.
• Metrics are not only descriptive: they shape thresholds, incentives, prevalence, labels, and apparent success.
• Robust evaluation should combine raw counts, baselines, confidence intervals, cost curves, calibration, prevalence monitoring, counterfactual evidence, and scrutiny of label generation. Treat evaluation as dynamic, population-specific, and decision-aware, not as a single score.