Application of large language models in data extraction from population-based observational epidemiology studies: A proof-of-concept study
Zuhaer Yisha, Jin Sun, Keqin Yang, Zechen Liu, Baoqi Zeng, Fengqi Liu, Sihan Yang, Huiwen Deng, Zihan Ma, Chenxi Yan, Tong Li, Cuo Jia, Yihao Hao, Zhuohui Zhao, Feng SunIntroduction:
Data extraction is a labor-intensive and error-prone stage of systematic reviews. Evidence regarding the use of large language models (LLMs) for extracting heterogeneous data from observational studies remains limited.
Methods:
In this prospectively planned, proof-of-concept diagnostic accuracy evaluation, 4 databases were searched from inception through June 2025 for population-based observational studies of long-term exposure to 4 gaseous air pollutants. Studies were selected using prespecified criteria. LLM prompts were developed using an external 30-study pilot set and locked before formal evaluation. LLM-based extraction and duplicate independent human extraction were conducted concurrently, with adjudicated human extractions serving as the reference standard. Accuracy was reported with Wilson 95% confidence intervals (CIs) for 24 data elements, whereas positive predictive value and sensitivity were summarized as point estimates.
Results:
The searches identified 24,644 records. After deduplication and screening, 484 studies met the eligibility criteria, yielding 11,616 study data element pairs. Overall accuracy was 92.6% (95% CI, 92.1%–93.0%). Across data elements, accuracy ranged from 62.4% (95% CI, 58.0%–66.6%) to 100.0% (95% CI, 99.2%–100.0%). Accuracy was lowest for adjusted effect estimates with CIs (62.4%; 95% CI, 58.0%–66.6%), the main exposure–effect statistical model (71.7%; 95% CI, 67.5%–75.5%), and outcome information (77.7%; 95% CI, 73.8%–81.2%). Of 862 extraction errors, 44.9% were complete mismatches and 2.4% were hallucinations.
Conclusions:
The evaluated LLM supported high-throughput extraction of standardized information but was less reliable for interpretive and quantitative data elements. It may be useful as a first-pass assistive tool with targeted human verification, but the findings do not support fully autonomous extraction.