Beyond mean comparisons: A multidimensional test of equivalence between online and paper-and-pencil surveys

Assessing equivalence across survey data-collection modes requires methodological approaches that go beyond simple mean comparisons. Building on this need, the present study proposes a multidimensional framework for evaluating methodological equivalence, integrating quantitative, qualitative, and auxiliary indicators. Using a homogeneous sample of 424 undergraduate students with comparable digital literacy, data were collected via online and paper-and-pencil administrations of the Nomophobia Scale and the Oxford Happiness Scale. Quantitative equivalence was assessed using the two one-sided tests (TOST) procedure, allowing for direct evaluation of practical equivalence rather than difference testing. Qualitative equivalence focused on distributional characteristics and response patterns, while auxiliary equivalence was examined through response styles. Results show that conclusions about equivalence depend strongly on the dimension examined, underscoring the limitations of relying solely on mean-based comparisons. By demonstrating how different equivalence dimensions may converge or diverge, the study offers a transferable methodological template for evaluating survey mode effects in applied research, extending beyond the specific instruments and sample analyzed here.