The increasing availability of automated tools for assessing data quality and FAIR compliance has enabled large-scale evaluations of publicly available datasets. However, a major challenge remains: many of these tools operationalize FAIR principles differently, resulting in assessments that are not directly comparable. Consequently, the same dataset may receive different evaluations depending on the tool used, potentially leading to inconsistent interpretations of its FAIRness and quality. This work investigates this potential misalignment through a quantitative comparative analysis of FAIR compliance assessment tools. In particular, it compares two state-of-the-art systems, namely KGHeartBeat and FAIR-Checker, to evaluate the extent to which their results align when applied to the same datasets. As an empirical case study, it focuses on biomedical knowledge graphs from the Biomedical Linked Open Data (BLOD) collection. The analysis considers a subset of 1,209 datasets from a larger corpus of 1,314 datasets, selecting those that can be successfully evaluated by both tools. We then perform statistical correlation analyses using Spearman measures to examine the degree of agreement between the metrics produced by the two systems. The results reveal moderate overall FAIR compliance across biomedical datasets, with stronger performance in Findability and weaker performance in Reusability. More importantly, the analysis highlights varying levels of correlation between metrics, revealing both areas of alignment and significant discrepancies in how different tools operationalize specific FAIR principles. This study provides a large-scale empirical comparison of automated FAIR assessment tools and offers insights into the strengths and limitations of current evaluation methodologies. Our findings highlight the need for greater methodological alignment to ensure that FAIR assessments are reliable, interpretable, and comparable across tools.
A large-scale empirical analysis of FAIR compliance in biomedical KGs using KGHeartBeat and FAIR-Checker
Latif, Sana
2026
Abstract
The increasing availability of automated tools for assessing data quality and FAIR compliance has enabled large-scale evaluations of publicly available datasets. However, a major challenge remains: many of these tools operationalize FAIR principles differently, resulting in assessments that are not directly comparable. Consequently, the same dataset may receive different evaluations depending on the tool used, potentially leading to inconsistent interpretations of its FAIRness and quality. This work investigates this potential misalignment through a quantitative comparative analysis of FAIR compliance assessment tools. In particular, it compares two state-of-the-art systems, namely KGHeartBeat and FAIR-Checker, to evaluate the extent to which their results align when applied to the same datasets. As an empirical case study, it focuses on biomedical knowledge graphs from the Biomedical Linked Open Data (BLOD) collection. The analysis considers a subset of 1,209 datasets from a larger corpus of 1,314 datasets, selecting those that can be successfully evaluated by both tools. We then perform statistical correlation analyses using Spearman measures to examine the degree of agreement between the metrics produced by the two systems. The results reveal moderate overall FAIR compliance across biomedical datasets, with stronger performance in Findability and weaker performance in Reusability. More importantly, the analysis highlights varying levels of correlation between metrics, revealing both areas of alignment and significant discrepancies in how different tools operationalize specific FAIR principles. This study provides a large-scale empirical comparison of automated FAIR assessment tools and offers insights into the strengths and limitations of current evaluation methodologies. Our findings highlight the need for greater methodological alignment to ensure that FAIR assessments are reliable, interpretable, and comparable across tools.I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
