Are they lit? Developing, testing, and implementing an instrument to measure artificial intelligence literacy

Abstract

This thesis presents the development and application of the "Scale for the assessment of non-experts' AI literacy" (SNAIL). Artificial intelligence (AI) is increasingly influencing various aspects of daily life and is being applied more and more frequently in professional contexts. To achieve a beneficial interaction between non-experts (i.e., individuals without specific AI education) and AI, a certain set of basic AI competencies is necessary. These basic competencies are commonly called "AI literacy" and have been the focus of intensive research in recent years. A particular branch of AI literacy research focuses on the reliable and valid measurement of AI literacy. Early AI literacy studies used unvalidated questionnaires, which were not suitable for reliably determining subjects’ AI literacy. Some researchers, including myself, have therefore started to develop measurement instruments for AI literacy that meet psychometric quality criteria. <br> When this research project was registered and planned, there were no validated instruments for measuring AI competence. Therefore, I conducted a Delphi expert study in which an initial item set for assessing AI literacy was generated through three iterative Delphi rounds. In this initial study, particular emphasis was placed on the content validity of the items, aiming to create a set of questions that would cover the entire field of AI literacy without exceeding its scope. Afterwards, the item set was presented to a large sample of non-experts, who assessed their individual AI literacy using the items created in the first study. Based on the collected data, I conducted an exploratory factor analysis, which examined both the underlying factor structure and reduced the number of items. The result of this second study was the final SNAIL questionnaire. As a subsequent intermediate step, an investigation was conducted into the extent to which the adapted SNAIL could be suitable for evaluating AI courses. The adaptation involved two steps: firstly, the items were systematically translated from English to German. Secondly, all items were presented in both a retrospective version (assessing AI literacy before the start of the AI course) and a post-version (assessment after the completion of the course). Specific statistical methods that are suitable for evaluating learning outcomes were employed to identify strengths and weaknesses of the evaluated course. Finally, a large-scale study was conducted, in which SNAIL was used for the first time to assess the AI literacy of a specific subgroup of non-experts. For this purpose, both SNAIL and a scale for assessing "attitudes towards AI" (ATAI) were distributed to medical students from two German medical schools. Conducting a confirmatory factor analysis revealed that the original three-factor model showed a good model fit for this new data set. At the conclusion of this thesis, avenues for further development and enhancement of SNAIL are presented. One potential area for improvement is to reduce the number of items in the final SNAIL questionnaire to increase the scale’s efficiency while maintaining sufficient reliability of the subscales. Finally, I take another look at AI literacy research as a whole in order to identify potential research directions associated with SNAIL that extend beyond questionnaire development.