LLMS GO AROUND RATING SENTENCES VERY POSITIVELY WITHOUT ATTENTION TO AFFECTIVE MEANING: AN EXPERIMENTAL COMPARISON OF LLM AND HUMAN SENSITIVITY TO CONSTRUCTIONAL SEMANTIC PROSODY

Semantic prosody describes the affective meanings or connotations of a linguistic element, which are commonly understood in terms of positive and negative alignment. Under the view of Construction Grammar, not only individual words but larger linguistic structures may carry semantic and pragmatic meaning, including positive or negative semantic prosody. The present pilot study takes this approach as a point of departure to investigate LLMs’ sensitivity to subtle pragmatic meanings in the form of semantic prosody. 4 state-of-the-art LLMs as well as a human participant group were tasked with rating the connotation and pleasantness of the go-around-Ving construction said to carry negative semantic prosody as well as two syntactically parallel constructions with neutral/positive semantic prosody. Results showed that while LLMs exhibit some sensitivity to constructional semantic prosody, their rating behavior differed significantly from humans when pooled into one group. Compared to humans, LLMs gave higher median and mean connotation and pleasantness ratings while also assigning less neutral ratings across constructions. Although not generalizable due to the exploratory character of this study, the results demonstrate that subtle pragmatic phenomena like semantic prosody represent a promising research area when it comes to discerning LLM language abilities and how they compare to humans’.