A framework for binary classification evaluation metrics

This paper presents a novel framework for analyzing and designing evaluation metrics in binary classification tasks. Traditional metrics—such as Accuracy, Precision, Recall, F1-score, and Cohen’s —often embed implicit assumptions about the relative costs and benefits of correct and incorrect predictions. However, these assumptions are not always transparent and may not align with domain-specific cost–benefit structures. By systematically evaluating classifiers through an underlying reward matrix, the proposed framework reveals that each metric reduces to a single break-even ratio between the resources invested and the value gained. This connects classical confusion matrix based metrics to an explicit cost–benefit interpretation. We derive this ratio explicitly for several widely used confusion matrix based metrics, thereby making their implicit trade-offs directly comparable under a unified interpretation. The paper demonstrates how metric values can be interpreted and applied to comprehensively assess classifier performance. Additionally, the framework allows researchers to define new metrics tailored to specific problem requirements. Experiments with commonly used metrics illustrate the framework’s broad applicability and highlight the value of explicitly modeling both costs and benefits for more context-sensitive performance evaluation.