Evaluation of knowledge-augmented DeepSeek models on nephrology examination questions
Youlu Zhao, Hongyu Yang, Xizi Zheng, Damin Xu, Lingyi Xu, Qingqing Zhou, Mengrui Li, Yuhui Zhang, Lijun Liu, Jicheng Lv, Li YangAbstract
Background
Large language models (LLMs) demonstrate significant potential in medical applications. DeepSeek, as an open-source and customizable LLM, offers potential for nephrology applications through flexible deployment, potential cost reduction, and enhanced data privacy. However, its specific performance and applicability in the nephrology specialty remain to be systematically evaluated.
Methods
We evaluated six large language models (DeepSeek-R1, DeepSeek-V3, GPT-4, GPT-4o, Qwen-2.5, and Gemini-2.5) using 1 294 nephrology multiple-choice questions derived from the ASN NephSAP and KSAP educational programs. The DeepSeek-R1, DeepSeek-V3, GPT-4o, and Gemini-2.5 models were further enhanced through a retrieval-augmented generation (RAG) framework incorporating a nephrology comorbidity knowledge base (NCKB). The NCKB, reviewed by board-certified nephrologists, integrated authoritative sources including standard textbooks, clinical guidelines, and peer-reviewed literature. The NCKB included 50 347 structured knowledge units. Model accuracy, response time, and error types were analyzed using chi-square tests and visualized with bar and radar charts.
Results
A total of 1 294 nephrology exam questions were used for evaluation. The knowledge-enhanced DeepSeek models demonstrated significantly improved performance compared with their base versions (both P < 0.001), with DSR1 + NCKB and DSV3 + NCKB achieving accuracies of 81.22% and 79.13%, respectively. The performance difference between DSR1 + NCKB and DSV3 + NCKB did not reach statistical significance (P = 0.171). Knowledge gaps were the main cause of DSR1 + NCKB’s errors (52.26%), followed by reasoning and comprehension errors (20.16%).
Conclusion
Integrating an expert-curated NCKB significantly improves performance on standardized nephrology questions compared with baseline models. The magnitude of improvement varies across models, with DSR1 + NCKB achieving performance comparable to or approaching that of advanced proprietary models evaluated in this study.