DOI: 10.1001/jamanetworkopen.2026.40850 ISSN: 2574-3805

Data Extraction Errors and Diagnostic Conclusions of Meta-Analyses

Zixing Huang, Yuan Tian, Ping Gong, Tong Zhang, Hehan Tang
Show PDF Cite

Importance

Systematic reviews (SRs) are often characterized as being inherently reproducible; however, errors arising from the evidence synthesis process can compromise this assumption and lead to reproducibility failure.

Objective

To investigate the data extraction errors in diagnostic meta-analyses and the extent to which such errors affect the results of meta-analyses.

Evidence Review

We included SRs with diagnostic meta-analyses from the Cochrane Library filtered by diagnostic up to May 2023. Data used in the original meta-analyses were extracted from the included SRs and independently reproduced from primary studies. Data extraction errors were identified by comparing datasets and classified by potential mechanism. To assess the consequences, we reanalyzed the meta-analyses using error-corrected data with the same effect estimators and methods. The primary outcomes were the proportions of data extraction errors at the trial, meta-analysis, and SR levels. The secondary outcome was the consequences for diagnostic performance measures (eg, sensitivity, specificity, diagnostic odds ratio [DOR], and area under the curve [AUC]). Analysis was conducted between March and June 2026.

Findings

A total of 46 SRs, 879 diagnostic comparisons, and 6173 primary studies were included. Data extraction could not be reproduced in 338 studies (5.5%) and affected 137 diagnostic meta-analyses (33.1%) and 34 SRs (73.9%). Numerical error (ie, incorrect values) and inferential error (ie, incorrect derivation from statistics) were the most prevalent error types, accounting for 166 of 412 (40.3%) and 140 of 412 (34.0%) errors, respectively. Mismatching error (ie, mismatching data to groups; 69 of 412 errors [16.7%]) and conceptual error (ie, misaligned reference details; 37 of 412 [9.0%]) were also identified. Among 137 reproduced meta-analyses, large changes were observed in 0.7% meta-analyses for sensitivity (absolute differences, >0.20), 3.6% for DOR (fold-changes, >5.0), and 2.9% for AUC (absolute differences, >0.10). Regression analysis of error location showed that combined false-positive and true-negative errors were associated with higher likelihood of moderate or large DOR changes (OR, 1.11; 95% CI, 1.01-1.22).

Conclusions and Relevance

In this SR of diagnostic meta-analyses in Cochrane reviews, data extraction errors were common and differentially affected performance measures, highlighting the need to implement rigorous data extraction processes and to consider careful selection of performance measures.

More from our Archive