Item Analysis of Artificial Intelligence-generated versus Faculty-generated Case-based Multiple-choice Questions in Anatomy: An Observational Study among 1st-year MBBS Students
Muralidhar Reddy Sangam, K. Praveen, G. Vinay, Amandeep Kaur, Roonmoni DekaContext:
The item analysis (difficulty index, discrimination index, and distractor effectiveness) provides input about the validity and reliability of the item. Several early studies indicate that artificial intelligence (AI) systems are capable of producing large numbers of multiple-choice questions efficiently, helping to reduce both the time and cost required. The ability of AI-generated case-based multiple choice questions (MCQs) in Anatomy to accurately assess the knowledge remains unexplored.
Objectives:
The objective of the study was (1) to compare the difficulty index, discrimination index, and distractor effectiveness of AI-generated and faculty-generated case-based MCQs in anatomy and (2) to evaluate the performance of 1 st -year MBBS students on AI-generated and faculty-generated case-based MCQs in anatomy.
Settings and Design:
This observational study was carried out at AIIMS Guwahati with the approval of IEC.
Subjects and Methods:
Two experienced anatomy faculty independently created a set of case-based MCQs ensuring they reflect appropriate cognitive levels, clinical relevance, and standard MCQ construction guidelines. Using a large language model (ChatGPT), the same topics are used as prompts to generate case-based MCQs. The questions are validated by two subject experts (blinded to the source) for accuracy, clarity, and relevance. Students are blinded from the source of MCQs. Students attempted two sets of 50 MCQ (25 AI-generated and 25 faculty-generated) with a crossover. Item analyses were performed.
Results:
Among 100 participants, AI-generated MCQs were easier (mean difficulty index = 57.2 ± 12.42 vs. 50.36 ± 17.4,
Conclusions:
Integrating AI with faculty oversight may therefore serve as a valuable approach in enhancing the quality and quantity of assessment tools for 1 st -year MBBS students.