From mAP to Statistical Metrics: A New Paradigm for Evaluating Model Accuracy in Metrology

Deep learning (DL) models are increasingly used in industrial applications for precise object measurement, particularly in resource-constrained environments where monocular cameras are favored for their cost-effectiveness. However, the traditional evaluation metrics, such as mean Average Precision (mAP) and validation loss, cannot accurately capture measurement task's accuracy and reliability. This study covers the use of Mean Absolute Error (MAE) and Standard Deviation (Std Dev) as specialized metrics to evaluate the trueness and consistency of DL models in measurement applications. We trained two DL based frameworks with different sets of hyperparameter configurations and assessed their performance using a dataset of images captured under identical conditions. The results show no significant correlation between mAP and the proposed metrics of MAE and Std Dev, further indicating the deficiency of the conventional metrics for measurement quality assessment. Instead, a positive linear relationship between MAE and Std Dev was recorded. Our analysis shows a strong correlation between MAE and Std Dev (0.92 for Detectron2 and 0.93 for YOLO). The random forest algorithm confirmed MAE and Std Dev as the most important feature (0.89 for Detectron2, 0.78 for YOLO), while validation loss and mAP had lower significance in feature importance and correlation. This work highlights the importance of incorporating statistical metrics into evaluating DL models to ensure the selection of configurations that deliver reliable, accurate, and efficient camera-based measurements.