Delphi study for the development and preliminary validation of an item set for the assessment of non-experts’ AI literacy

Abstract

Artificial intelligence literacy is a concept that has been the focus of exhaustive research recently. However, there are very few psychometrically sound and thoroughly evaluated instruments that attempt to assess AI literacy in a valid way. Therefore, this study presents an item set to assess the AI literacy of non-experts. In the context of a Delphi expert study, 53 subject matter experts participated in three iterative questionnaire rounds to generate potential AI literacy items and assess their content validity. In addition, the experts made suggestions on how the items’ wording accuracy could be improved and evaluated the wording suggestions of the other experts. Of 47 potential items, 38 were judged relevant for inclusion in a final AI literacy questionnaire. The result is one of the first freely available AI literacy item sets and represents an important first step in assessing AI literacy and its subconstructs. Finally, the development of the items through the execution of an iterative Delphi study and the strong focus on content validity contribute to the advancement of AI literacy theory.