Performance of Machine Learning models heavily depends on the quality of the training dataset. Among others, the quality of training data relies on the consistency of the labels assigned to similar items. Indeed, the labels should be coherently assigned (or collected) by avoiding inconsistencies for increasing the performance of the machine learning model. This study focuses on evaluating training data consistency for machine learning algorithms dealing with ranking problems, i.e., the Learning to Rank methods (LTR). This work defines a training data consistency measure based on the consensus value introduced in Group Decision Making. It investigates the statistical relationship between the proposed consistency measure and the performance of a deep neural network implementing an LTR method. This measure could drive data filtering at the training stage and guide model update decisions. Experimentation reveals a strong correlation between the proposed consistency measure and the performance of the model.

Data set quality in Machine Learning: Consistency measure based on Group Decision Making

Fenza G.;Gallo M.;Loia V.;Orciuoli F.;
2021

Abstract

Performance of Machine Learning models heavily depends on the quality of the training dataset. Among others, the quality of training data relies on the consistency of the labels assigned to similar items. Indeed, the labels should be coherently assigned (or collected) by avoiding inconsistencies for increasing the performance of the machine learning model. This study focuses on evaluating training data consistency for machine learning algorithms dealing with ranking problems, i.e., the Learning to Rank methods (LTR). This work defines a training data consistency measure based on the consensus value introduced in Group Decision Making. It investigates the statistical relationship between the proposed consistency measure and the performance of a deep neural network implementing an LTR method. This measure could drive data filtering at the training stage and guide model update decisions. Experimentation reveals a strong correlation between the proposed consistency measure and the performance of the model.
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11386/4804692
 Attenzione

Attenzione! I dati visualizzati non sono stati sottoposti a validazione da parte dell'ateneo

Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 7
  • ???jsp.display-item.citation.isi??? 6
social impact