Towards Automated FAIR Compliance Diagnosis: Evaluating LLMs on Explanation and Diagnosis Questions

The FAIR principles provide a widely adopted framework for assessing and improving the quality of research data; however, understanding why a dataset attains a given FAIR score and how to improve it remains challenging for dataset producers and consumers. This paper investigates the suitability of large language models (LLMs) for answering Explanation & Diagnosis questions related to FAIR principle evaluation. We introduce the first human-curated benchmark for this task, comprising 40 questions organized into four categories: clarification, metric calculation, improvement, and impact on scoring. Using an in-context learning paradigm, we evaluate five LLMs by providing them with the documentation of a FAIR assessment tool expressed in three semantically equivalent formats: JSON, Markdown, and Turtle. Our results show that LLMs are generally effective at explaining and diagnosing FAIR scores, achieving strong performance on clarification questions, while questions requiring multi-step reasoning about score impact and improvement remain more challenging. Documentation format has no statistically significant effect on overall accuracy; however, Markdown outperforms Turtle for improvement-oriented questions. We further observe that models optimized for reasoning consistently yield higher accuracy on complex question types. These findings highlight both the promise and current limitations of LLMs for automated FAIRness support and offer practical guidance for integrating LLM-based assistance into FAIR assessment platforms.