Unpacking Bias in LLMs: A Causal Testing Perspective

Large Language Models (LLMs) – the backbone of modern Natural Language Processing applications (NLP) - are notoriously subject to biases that reflect stereotypes present in their training. Fairness testing typically aims to expose bias by generating texts that should yield equivalent outputs, yet differ only in protected attributes such as gender or religion. However, fairness testing for LLMs is more complex than it seems. Bias depends not only on variations in the protected attribute, but also on its causal interactions with non-protected attributes and their combined influence on the output. Ignoring these dependencies risks producing unrealistic scenarios or misleading conclusions about bias. We introduce