The introduction of Large Language Models (LLMs) able to exhibit a number of linguistic and extra-linguistic capabilities has represented, in the last years, one of the main frontiers in Artificial Intelligence (AI) research. Researcher from various disciplines debate about whether or not, among the capabilities of LLMs, there is the one of using knowledge about knowledge-usually considered one of the antechambers of meta-cognition in cognitive agents-about a particular task in order to improve or self-correct previous errors. In this work we propose a novel fine-tuning approach for LLMs, named EXAR, based on a multi-stage process leveraging past predictions from an early version of the same, and aimed at injecting metacognitive features for the task of Question-Answering. The conducted experiments on LLAMA-2-7B-CHAT showed promising improvements on the quality of the outcomes, due to the fact that the LLM acquired the ability to detect its own wrong predictions forcing itself to repeat submissions, thorough a prompt designed to fix inadmissible predictions, whenever detected. Such detection is achieved by enquiring the same LLM acting as meta-validator, through another prompt specifically designed for such purpose.

Eliciting metaknowledge in Large Language Models

Lieto A.
Supervision
2025

Abstract

The introduction of Large Language Models (LLMs) able to exhibit a number of linguistic and extra-linguistic capabilities has represented, in the last years, one of the main frontiers in Artificial Intelligence (AI) research. Researcher from various disciplines debate about whether or not, among the capabilities of LLMs, there is the one of using knowledge about knowledge-usually considered one of the antechambers of meta-cognition in cognitive agents-about a particular task in order to improve or self-correct previous errors. In this work we propose a novel fine-tuning approach for LLMs, named EXAR, based on a multi-stage process leveraging past predictions from an early version of the same, and aimed at injecting metacognitive features for the task of Question-Answering. The conducted experiments on LLAMA-2-7B-CHAT showed promising improvements on the quality of the outcomes, due to the fact that the LLM acquired the ability to detect its own wrong predictions forcing itself to repeat submissions, thorough a prompt designed to fix inadmissible predictions, whenever detected. Such detection is achieved by enquiring the same LLM acting as meta-validator, through another prompt specifically designed for such purpose.
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11386/4910695
 Attenzione

Attenzione! I dati visualizzati non sono stati sottoposti a validazione da parte dell'ateneo

Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 0
  • ???jsp.display-item.citation.isi??? 0
social impact