Design and Validation of a Novel Decision-Making Task for Unethical AI

Many studies involving the effects of unethical AI on humans center around life-or-death situations. While there are benefits in examining unambiguously unethical AI, the everyday person does not encounter these extreme situations. More commonly, large language models (LLMs) may give unethical advice to its users, and the effects of this are not well studied. Furthermore, there does not exist a task to assess the impact of unethical AI advice in lower stakes settings. For this study, we designed a decision-making task in the context of supply delivery to remote outposts. We then systematically created 54 vignettes consisting of financial risk and ethical risk vignettes and had participants ( N  = 59) rate them on the levels of financial and ethical risk. We ended with 50 vignettes total (23 financial, 27 ethical) that were differentiated on these risks, and demonstrated how they can be organized into matched sets for repeated measures studies.