Don’t Shoot the Messenger: How Source Context Shapes Human and Model Interpretation of Online Discourse

Official crisis messages on social media draw extensive public replies. Some replies contribute new information by offering evidence, corrections, explanations, or concrete requests, whereas others primarily express stance, sentiment, or affiliation. We frame this distinction as an epistemic (gk. epist \(\overline{\mbox{e}} \) m \(\overline{\mbox{e}} \) – knowledge) classification problem and ask whether humans and large language models (LLMs) draw the boundary between discussion and reaction in the same way. We analyze replies to official COVID-19 communication on Twitter/X during the politically salient, and charged, period of the 2020 U.S. election. From conversations initiated by politicians and by officials of government agencies, we construct 2,000 initiator–response pairs and collect human labels under two controlled information conditions: without-context (initiator post and reply text only) and with-context (the tweet pairs as well as labels that identify if the initiator is a politician or a government account). After generating training data through human annotation, we then apply a three-expert LLM ensemble under matched conditions to compare human and model epistemic judgments.

Our central expectation is that source context will influence how engagement is classified, and that this effect will be stronger for human annotators than for LLMs. The results confirm that context shifts the decision boundary between discussion and reaction, but not entirely in the anticipated direction. As expected, human annotators exhibit substantial shifts in their judgments when source information is visible. However, contrary to our directional expectations, this shift occurs for both politician- and government-initiated posts, rather than being more pronounced for one source type.

We further expected the LLM ensemble to remain relatively anchored to the lexical content of the tweet pair, showing limited sensitivity to source context. To investigate this hypothesis, we trained the LLM ensemble with and without source context. When trained without source context, the LLM ensemble remained relatively insensitive to source context added during testing. However, when provided source context during training, the ensemble’s judgments shift toward reaction by a magnitude comparable to the human annotators’, indicating that models reproduce the human contextual effect through their training labels rather than by interpreting source information at inference time. For the LLM ensemble to be context insensitive, our results indicate that context should be purposely withheld at training time. This has practical implications for the training of systems that rely on rapid interpretation of public responses: contextual signals introduced during training, purposefully or inadvertently, may not simplify, but may in fact complicate, decision-making.