Benchmarking commercial large language models for gene–disease–phenotype extraction from full‐text human genetics literature

Abstract

Manual curation of gene–disease–phenotype relationships from the human genetics literature is a persistent bottleneck for maintaining its bioinformatics databases. Whereas large language models (LLMs) offer a promising alternative, there is currently no systematic benchmark that evaluates whether state‐of‐the‐art commercial LLMs can perform this task reliably on the full‐text articles. To address this gap, we introduce a standardized benchmark comprising 406 full‐text articles covering 180 congenital heart disease‐associated genes, and a multi‐dimensional evaluation framework that incorporates fuzzy matching to account for synonyms and partial matches. We benchmarked seven state‐of‐the‐art LLMs, GPT‐4o, Claude‐Opus‐4, DeepSeek‐R1, Grok‐4, Qwen‐3.5, Gemini‐2.5 (Pro), and GPT‐5 on the extraction of structured gene, disease, and phenotype fields. The top‐performing model, Grok‐4, achieved 97.6% overall accuracy, whereas the lowest‐performing model reached approximately 88%, still surpassing many prior benchmarks employing zero‐shot or n ‐shot prompting in biomedical relation extraction (RE) tasks. Our results provide a rigorous characterization of current LLMs capabilities and limitations. This paper contains two components. First, we conducted a human genetics field benchmark study on LLMs against a curated database. Second we developed the evaluation framework for this task. The benchmark dataset, evaluation framework, and model benchmarking outputs are made available online to support future studies in a reproducible manner.