Survey Authentication Challenges During a Geriatrics Knowledge Assessment Pilot
Apoorva Rangan, Mengru Wang, Malvika Varma, Yochai Shavit, Stephen Pelletier, E. Nola Sundquist, Deborah Kado, Andrea SchwartzABSTRACT
Background
In this methodological Brief Report, we describe a multi‐institutional survey authentication challenge encountered during pilot deployment of the Age‐Friendly Knowledge Assessment Tool (AF‐KAT), a geriatrics knowledge assessment for graduating medical students.
Methods
The AF‐KAT is a 20‐item case‐based multiple‐choice assessment aligned with the Geriatrics 5Ms framework and American Geriatrics Society minimum geriatrics competencies for graduating medical students. The survey was distributed electronically to graduating medical students at three US medical schools during the 2023–2024 academic year using institutional listservs and included a financial incentive. Following survey deployment, an unexpectedly large volume of responses demonstrated patterns inconsistent with medical student responses. A structured multi‐institutional authentication process was used to develop exclusion criteria and adjudicate responses.
Results
A total of 1029 survey responses were received. Exclusion criteria identified responses with reimbursement codes not linked to institutional email correspondence ( n = 934), graduation years inconsistent with the intended 2024 cohort ( n = 352), duplicated qualitative responses ( n = 76), unrealistic intended specialty responses ( n = 183), duplicated IP addresses ( n = 74), and completion times under 180 s ( n = 43). Exclusion criteria were not mutually exclusive. Following multi‐reviewer adjudication, 88 responses remained in the final analytic sample. AF‐KAT score distributions differed substantially before and after authentication and adjudication procedures. In the pre‐authentication dataset ( n = 1029), the median AF‐KAT score was 5 (IQR: 3–9), compared with 12 (IQR: 6.75–14) in the post‐authentication analytic sample ( n = 88).
Conclusion
We report an example of likely inauthentic participation that distorted performance on a multi‐institutional pilot of a geriatrics knowledge assessment tool. We strongly recommend proactive incorporation of survey security measures in financially incentivized geriatrics education research.