Law or Flaw: A Double – Blind Study Comparing Student Comprehension of Real and AI Generated Legal Case Briefs

In recent years, the U.S. legal system has seen an increase of legal filings using fictitious court cases or legal propositions generated by artificial intelligence (AI) (Stokel-Walker, 2026). Rules of professional responsibility require that lawyers review filings in which AI was used to ensure their accuracy (Missouri Bar, Office of Legal Ethics Counsel, 2024, Opinion No. 2024‑11) Despite this mandate, filings with fictitious cases and incorrect statements of law are being filed. Besides posing a threat to the parties to an action, such filings may set an unwarranted precedent for future cases. They also tie up court resources searching for nonexistent law or unfounded propositions of law and erode public trust. Further, an AI generated summary of a legitimate court case presents a risk of being misleading as it may fail to capture a true understanding of the case. In this study, undergraduate criminal justice college students provided human review of AI generated legal material. The students were asked to complete a typical undergraduate criminal law and procedure class assignment. At random, students were either given an excerpt from a real legal case decision or an AI generated summary of the same. Students were then asked to read the excerpt or AI summary and answer the same questions based upon their reading. Not surprisingly, students using the AI summary were not able to correctly or fully answer the questions, as the AI summary did not capture the nuances or depth of the legal case that may be found by a full reading of the case excerpt.