Neither More Agents nor Different Backbone Models Consistently Reduces Directional Diagnostic Error of Artificial Intelligence Agents in Pulmonary Mucormycosis Versus Invasive Pulmonary Aspergillosis

Pulmonary mucormycosis and invasive pulmonary aspergillosis present alike, yet their first-line treatments are not interchangeable: voriconazole has no activity against Mucorales. Labelling mucormycosis as aspergillosis before culture, histopathology, or molecular results return could point treatment towards a regimen without activity against Mucorales. We asked which way artificial intelligence (AI) diagnostic agents err in this pre-confirmation window. We truncated 140 published pulmonary case reports (47 mucormycosis, 45 invasive aspergillosis, and 48 other) before the final diagnosis was revealed and had two blinded physicians review each report. Agents chose one diagnosis; abstention was not permitted. We compared a single agent with a simulated multi-agent consultation in which specialist-role agents running on the same backbone are aggregated by a moderator; isolated the effects of the design elements, including a one-page guideline primer, factorially in three backbones; and repeated the comparison across ten backbones. Across backbones, the proportion of mucormycosis that was labelled aspergillosis stayed within 4.3–19.1% for the guideline-primed single agent but ranged from 10.6% to 74.5% for the multi-agent consultation; within one model, changing the design moved it from 8.5% to 74.5%. This consultation configuration, which differed from the single agent in several design elements, erred more often in all ten backbones (in 103 of the 113 discordant case–backbone pairs, only the consultation erred), with a penalty that varied roughly thirty-fold between backbones. In the exploratory factorial experiment in three selected backbones, the consultation structure was associated with more such errors, and the guideline primer with fewer, in all 12 matched comparisons; the primer, however, shifted these cases to “other” and did not identify more mucormycosis. Fewer aspergillosis errors did not mean a better overall error profile: across all agent configurations evaluated on the 140-case set, agents identified only 18–55% of mucormycosis cases. Two blinded physicians agreed closely with each other (κ = 0.856) yet matched the reference diagnosis in only 55.8% and 58.1% of a 43-case subset, as did an unscaffolded frontier model (58.1%); these figures describe observed performance on the truncated texts, not the maximum achievable. On the same 14 mucormycosis cases, however, each physician called one aspergillosis; the consultation called twelve. In this exploratory evaluation of classification errors, neither a multi-agent design nor a different backbone model consistently reduced this directional error.