DOI: 10.12688/wellcomeopenres.27560.1 ISSN: 2398-502X

Identifying outstanding research proposals: An examination of reviewers’ judgement processes

Gaëlle Vallée-Tourangeau, Tushna Vandrevala, Ana Wheelock, Hulya Gokalp, Christopher Tomlinson, Fanni Szigetvari, Priscilla Harries
Show PDF Cite
Background Understanding proposal evaluation helps explain the science of peer review. This study aimed to identify how experienced reviewers use application information when making funding recommendations. Methods Social judgement theory methods were used to examine experienced peer reviewers’ funding recommendations. The research had three phases. Phases: I) ‘think aloud’ interviews was undertaken by 7 reviewers on a grant proposal they reviewed for the Wellcome Trust’s ‘Research Fellowship in Humanities and Social Sciences’ scheme, to uncover quality factors used: II) 28 expert reviewers identified and ranked the 10 most important quality factors, which were then used to generate a set of 60 fellowship application vignettes using fractional factorial design III) 24 reviewers made funding recommendations individually, on the vignettes. Multiple regression analysis was used to model the weighting given to the factors. Results Fifty-seven quality factors were initially uncovered through the content analysis of the think aloud protocols, and the 10 factors deemed most important in assessing the quality of fellowship applications were identified. Accounting for 92% of the variance, the 24 expert reviewers used all 10 factors in their decision making; the applicant’s publication record was the most significant in terms of funding recommendation, followed by the applicant’s experience and expertise. Reliability of the mean ratings across 22 reviewers’ recommendations was very high: ICC(2,22) = 0.94 (95% CI, 0.915 to 0.96). Conclusions There was high interrater reliability of reviews in this funding scheme. The focus on the candidate’s publication record and experience is likely to be due to the funding scheme being for ‘fellowships’, where the potential development of the individual is central to the award. This information may be used differently when considering ‘project’ funding applications. The Narrative CV has helped widen recognition of research contributions beyond traditional metrics. Findings will support the ongoing development of peer reviewing approaches.

More from our Archive