Eye movements reveal a dissociation between prediction and structural processing difficulty in language comprehension

In the process of extracting a meaning from a text, our eyes linger much more on some words than others, and we often reread earlier portions of the text. These disruptions to the reading process are particularly common in syntactically ambiguous sentences. What explains the difficulty presented by these sentences? One prominent hypothesis explains it as a special case of the impact of a word’s predictability (operationalized via surprisal) on the difficulty of processing the word. This contrasts with theories that attribute these disruptions to errors in the structure-building process. Earlier attempts to address this debate have been inconclusive because of small numbers of participants, coarse measurements of the reading process that are ill-suited to disentangling these competing views, and a limited range of surprisal estimates. Here, we conduct a large-scale study ( n = 368 ) examining eye movements during the reading of syntactically challenging sentences, using 409 types of surprisal estimates from language models with multiple architectures and training settings. We find a stark dissociation: Early effects of syntactic disambiguation are well-approximated by language model surprisal, but syntactic disambiguation incurs a significant additional cost, reflected in an increase in rereading that is not explained by language model surprisal. We conclude that surprisal can capture routine structure-building, but not the cost of detecting or correcting errors in the structure-building process.