EHRForge: A Reproducible Workflow for Constructing an Evidence-Grounded Synthetic Temporal QA Benchmark from Longitudinal Clinical Notes

Longitudinal electronic health records (EHRs) are a natural source of temporal QA problems, but turning them into a reliable benchmark is not simply a matter of prompting a language model to write questions and answers. We present EHRForge, a reproducible workflow for constructing evidence-grounded synthetic temporal QA benchmarks from governed clinical-note collections. The workflow builds chronological visit-level timelines, selects context according to history length, generates schema-constrained QA candidates, and then subjects those candidates to deterministic temporal and structural checks followed by segment-to-visit evidence alignment against the source notes. In a restricted institutional case study, EHRForge generated 27,558 candidates. Of these, 18,867 (68.5%) passed the first verification stage and 10,742 (39.0% of the original candidates) passed the automated evidence-alignment stage across 829 patients. Human validation included dual review of 300 Round 1 rejections, 300 Round 2 rejections, and 1000 retained items. On the retained set, raw agreement between the two human validators ranged from 96.4% to 99.3% across five quality criteria, with Cohen’s κ ranging from 0.459 to 0.737. In downstream QA experiments, we evaluated four retrieval methods in a retrieval-augmented QA setting, with NV-Embed-v2 achieving 77.0% Hit@5, 59.0% Recall@5, and 41.7% complete Coverage@5 when retrieving from full patient timelines. Qwen2.5-32B reached 40.1 Token F1 with NV-Embed-v2 evidence and 46.7 with oracle evidence. Further analysis showed that much of this oracle gap arose when one or more annotated evidence visits failed to reach the reader. These results position EHRForge as an auditable and configurable approach to benchmark construction in settings where source provenance and institutional data governance must be preserved.