Artificial Intelligence Explanations of Organic Reaction Mechanisms Revisited: Improved Accuracy and Sophistication but Persistent Mechanistic Flaws

Abstract

Generative artificial intelligence tools are increasingly used by students to explain chemistry concepts, but their educational value depends on the chemical accuracy and mechanistic quality of the explanations. Prior work found that ChatGPT could produce sophisticated and convincing organic reaction mechanisms that were frequently inaccurate. Using the same mechanism set and prompt-cueing levels as those in an earlier study, this study evaluates how a reasoning-based large language model changes the relationships among response correctness, explanation sophistication, and chemical concept use. ChatGPT o3-mini responses were generated for mechanisms spanning a year-long organic chemistry curriculum and analyzed for response-level accuracy, nucleophile and electrophile explanation sophistication, sentence-level concept use, and qualitative reasoning patterns. Compared with GPT-3.5, o3-mini increased the response-level accuracy from 28% to 79% and produced more sophisticated explanations. However, inaccuracies remained concentrated in aromatic ring and carbonyl mechanisms. Prompt cueing increased the chemical richness and explanation sophistication without improving response-level accuracy. Qualitative analyses showed that sophisticated terminology could mask flawed reasoning: o3-mini applied correct chemical concepts to inappropriate mechanisms, overgeneralized common mechanism patterns, and used advanced terminology in vague or decorative ways. These findings show that improvements in aggregate accuracy can coexist with persistent flaws in mechanistic reasoning and that great explanatory detail is not a reliable indicator of validity. The fixed benchmark provides a controlled comparison of which dimensions of response quality changed across model generations, while the sentence-level and qualitative analyses identify recurring error patterns that can be examined in future models. This framework gives instructors concrete features around which to structure student critique and provides researchers with a durable approach for studying AI-generated mechanistic explanations beyond point-in-time capability ranking.