AI lexicography and literary texts: applying Gricean maxims to evaluate ChatGPT’s performance on Shakespearean vocabulary

Abstract

This study evaluates the reliability of ChatGPT-3.5, accessed via its 2024 public web interface, in identifying and contextualizing Shakespearean lexical items within Elizabethan English. Using five lexical items (bread, blood, grace, honest, and wit) as test cases, ChatGPT’s outputs are systematically compared with established reference resources, including Shakespeare’s Words and the Shakespeare Complete Works. The analysis adopts Grice’s cooperative principle—Quality, Quantity, Relevance, and Manner—as an evaluative framework for assessing the accuracy, proportionality, relevance, and clarity of artificial intelligence (AI)-generated responses. A total of eighty responses were independently coded by six raters, yielding perfect inter-rater agreement (Fleiss’ κ = 1.00). Results show that violations of the maxim of Quality—particularly factual inaccuracies, misattribution, and fabricated citations—were dominant across all lexical items, despite variation in semantic domain. While ChatGPT demonstrates fluency and occasional contextual sensitivity, its outputs exhibit instability and limited reliability in historically grounded literary tasks. The findings, specific to ChatGPT-3.5, suggest that this model generation may function as a supplementary tool for exploratory or pedagogical purposes but cannot replace curated lexicographic resources in Shakespearean scholarship; whether these limitations persist in more recent model generations remains an open question for future comparative work.