Analytic Hierarchy Process for an Automated Evaluation of Learning Difficulties of Novice Programmers

Decision support systems represent a relevant tool for data-driven decision-making with limited human resources. It is critical to adopt enabling techniques that avoid automation distortion, that is, delegating the decision to the system without understanding the motivation. The utilization of state-of-the-art AI techniques for decision-making in specific application domains is considered a high-risk practice, such as in Education. In this context, we propose a general methodology that integrates a rule-based classification schema with the Analytic Hierarchy Process, and a practical application to identify and deal with students’ misconceptions through the automatic analysis of compiler logs. Novice programmers, beyond their problem-solving skills, often demonstrate difficulties in writing source code that is error-free. Behind the difficulties of mastering the syntax of a programming language and understanding the compiler messages, is often hidden a wrong comprehension of fundamental coding concepts. A typical decision problem is related to which exercise would allow the student to verify the learning outcome after some remedial activities. We applied the the proposed methodology to a real dataset of compiler logs. The dataset is composed of about two thousand programs developed by novice programmers, who addressed their assignment in five editions of a bachelor’s course in fundamental programming. It also includes compiling errors or warning messages introduced during the development cycle.