DOI: 10.1162/rest.a.1882 ISSN: 0034-6535
How Credible Are the Findings from Online Experiments?
Abel Brodeur, Nikolai Cook, Anthony Heyes, Lamis KattanAbstract
Online experiments have surged in popularity, but how credible are their published results? We examine the universe of hypotheses tested on Amazon Mechanical Turk and reported in leading journals between 2010 and 2021. We find evidence of widespread p-hacking, publication bias, and under-powered studies. The extent of the problems varies substantially across fields, with marketing particularly badly afflicted and little evidence of distortion in economics. We explore correlates of increased credibility and compare the online results to laboratory results from the same outlets, finding similar distortions in both, with bunching just above the significance threshold more pronounced on MTurk.