Do highly cited papers in AI medical journals reflect clinical maturity?: A matched analysis of 690 ESI highly cited papers and 690 matched controls
Jiayu Luo, Jiawei Luo, Fan Wang, Lu Cheng, Dan Xu, Haiyang Ye, Zhikang Ye, Qi ZhouBackground:
Highly cited papers in journals focused on artificial intelligence (AI) in medicine receive substantial scholarly attention, but it is uncertain whether this citation prominence corresponds to more advanced clinical evaluation. High citation counts may reflect methodological novelty, topic visibility, publication timing, or technological interest rather than real-world implementation or patient benefit.
Methods:
We used a source frame of 78 journals focused on AI in medicine and health and identified papers designated as Highly Cited Papers (HCPs) in Essential Science Indicators within the Web of Science Core Collection. Each HCP was matched 1:1 to a non-HCP from the same journal and publication year, with further matching on a prespecified harmonized document type when possible. The primary analysis comprised 649 exact journal–year–document-type matched pairs. The primary outcome was advanced clinical evaluation (Stages 4–6 vs Stages 0A–3), and the sensitivity outcome was external validation or higher (Stages 3–6 vs Stages 0A–2). We reported group-specific proportions, paired absolute differences, 95% confidence intervals (CIs), and 2-sided exact McNemar
Results:
Among 649 exact-matched pairs, 476/649 HCPs (73.3%) and 528/649 matched non-HCPs (81.4%) were classified as Stage 0A or 0B. Stages 4–6 occurred in 15/649 HCPs (2.3%, 95% CI 1.3%–3.8%) and 8/649 matched non-HCPs (1.2%, 95% CI 0.5%–2.4%); the paired absolute difference was 1.1 percentage points (95% CI −0.5 to 2.8;
Conclusions:
In 649 strictly matched pairs, advanced clinical evaluation was uncommon among both HCPs and matched non-HCPs, and no clear paired difference was observed. Restricted and journal-influence analyses did not change this interpretation. ESI HCP designation should not be assumed to indicate clinical maturity. Patient benefit was not directly assessed.