AI Chat Bot Use in the Classroom Gone Wrong: Implications for Students, Teachers, and Schools

This article presents a case where a teacher makes an experimental assignment choice using generative artificial intelligence (GenAI). Ninth-grade creative writing students are asked to complete a poetry project using GenAI as a tool to brainstorm or reinforce their work. Because students are prompting GenAI with a creative assignment, the tools may respond differently than they do with non-creative prompts. For example, prompting GenAI to help in a creative writing context may cause the tool to forgo sharing mental health resources and encourage a user to pursue certain dangerous actions. In this case, a bullied student uses his AI tool beyond the assignment parameters, resulting in his suicide.