Analysis of Data with Multiple Representations Conducted by GenAI and a Chemistry Education Expert: Strengths and Weaknesses of GenAI’s Analysis

Abstract

Studies examining how Generative Artificial Intelligence (GenAI) contributes to analyzing data with multiple representations (i.e., symbolic, particulate-level drawings, and written explanations) are scarce. To address the gap, this study compares the potential of GenAI with a chemistry education expert to analyze multimodal data of chemical reactions. Responses to two open-ended chemistry tasks, collected from 30 preservice teachers enrolled in undergraduate programs in chemistry and science education departments, were analyzed by ChatGPT 5.1, a large language model, and a chemistry education expert. The consistency between the expert and ChatGPT was evaluated through qualitative content analysis and thematic analyses of participants’ responses to the tasks. The findings showed that ChatGPT effectively analyzed symbolic representations and written explanations, demonstrating greater consistency with the expert’s assessment. Additionally, ChatGPT recommended novel assessment criteria that went beyond the predefined evaluation for both questions and provided individualized feedback and specific recommendations to address participants’ areas for development in chemistry content knowledge. Nevertheless, it also struggled to identify and assess particulate-level drawings in many cases, leading to reduced consistency due to perceptual errors and limitations in interpreting molecular geometry and stoichiometry. While GenAI tools show promise in assessing open-ended content in chemistry education, ChatGPT 5.1 showed limitations in analyzing the multimodal data used in this study. These findings have implications for chemistry education research and instruction, indicating that GenAI-supported assessment of multimodal chemistry responses may still require expert supervision.