Can agentic AI match the gold standard? multi-agent risk stratification for syncope in the emergency department

Background

Patients with high-risk syncope have a worse prognosis1. Clinical judgment (CJ) remains the gold standard for syncope risk stratification2.

Aim

To compare risk stratification performed by a multi-agent artificial intelligence (AI) framework with CJ.

Methods

We developed a framework to predict 30-day adverse events in patients presenting with syncope. Adverse events were defined as a composite outcome according to the Standardized Reporting Guidelines for Emergency Department Syncope Risk-Stratification Research. The framework consisted of a multi-agent workflow and a decision tree algorithm.

The multi-agent workflow incorporated five knowledge bases: ACC and ESC syncope guidelines, two review articles, and the syncope chapter of UpToDate. The decision tree used the multi-agent output plus six clinical features. The model generated binary risk stratification with a textual explanation. Agentic workflows operated in a zero-shot setting. The decision tree was trained and tested using an 80/20 split.

Performance was evaluated separately in low-, moderate-, and high-risk subgroups of patients admitted for syncope to the EDs of three tertiary hospitals (2015–2017). AI output was compared with CJ regarding hospitalization or discharge decisions. Thirty-day risk stratification was assessed using Kaplan–Meier curves; discrimination by ROC curves. Reclassification was visualized with Sankey diagrams. Incremental value beyond CJ was assessed using a likelihood ratio test (LRT) comparing Cox models (CJ vs. CJ+AI). Net benefit and net interventions avoided were evaluated by decision curve analysis.

Results

Among 266 patients (median age 73 [58–83]), 44 experienced 30-day adverse events. In high-risk patients, AI showed numerically higher discrimination than CJ (AUC 0.76 vs. 0.69, p=0.40) with a lower false-negative rate. AI provided incremental prognostic value (LRT p<0.001) and yielded greater net benefit and fewer net interventions.

In moderate-risk patients, AI discrimination was numerically lower than CJ (AUC 0.59 vs. 0.67, p=0.41), with higher false-negative rates and no incremental value. In low-risk patients, only one event occurred; it was correctly classified by AI but missed by CJ.

Conclusions

The multi-agent AI framework achieved risk stratification comparable to CJ. Its greatest benefit was observed in high-risk patients, reducing false negatives and improving net clinical benefit. Performance in moderate-risk patients was similar to CJ without added value, while conclusions in low-risk patients are limited by the low event rate.Figure 1Figure 2