Exploring the Impact of Nonlinearity on Effect Heterogeneity in Experimental Data

Heterogeneity has been proposed as a framework to improve psychological and behavioral science by identifying mechanisms that explain the differences between replications of an effect. The modal response to such differences is to search for relevant moderators—typically with limited success. We argue that monotonic nonlinear relationships can generate heterogeneous effects across replications when baseline levels differ, even in the absence of moderators. To evaluate this possibility, we introduce four descriptors of heterogeneity: effect heterogeneity, control-group heterogeneity, the intercept-slope correlation, and the rank correlation between control and treatment means. These characterize patterns consistent with linear, concave, or convex functional relationships. We apply this analytical framework to an initial pool of 55 meta-analytic multilab data sets. All data sets with heterogeneous effects also demonstrated control-group heterogeneity, and several patterns were in line with a concave functional relationship. In some cases, the association between baselines and effects accounted for a substantial proportion of effect heterogeneity. In addition, only a small number of data sets were directly at odds with a monotonic nonlinear functional form. Moreover, our results suggest that effect heterogeneity in multilab research reflects diverse data patterns rather than providing support for treating moderators as the modal explanation for effect heterogeneity. We discuss implications for multilab and meta-analytic practice, study design, and interpretations of effect heterogeneity. We advocate for a shift from treating only the effect-size distribution as the explanandum of the heterogeneity framework to more elaborate descriptions of heterogeneity.