Origin of apparently-improved measurement error with artificial intelligence in Echocardiography

Background

Automated (“Artificial Intelligence”, AI) methods seem to achieve better performance in matching human expert measurements of clinical variables, than experts themselves do. We explored the mechanism for this in a unique 6-observer, 100-patient, regulatory dataset. We examined the 4 contributors to ejection fraction and then 55 other directly measured variables.

Purpose

To determine why AI measurements in echocardiography outperform human expert measurements.

Method

Six experts each directly measured 59 variables in each of 100 echocardiograms. Each independently chose the video, beat, and frame, and then traced the distance or boundary curve. One AI system assessed every beat of every video. We quantified the measurement variance (expert-to-expert, and AI-to-expert) for all 59 variables. We then stratified these expert-expert or AI-expert pairings into whether the 2 measurements happened to be on:The same frame of the same video,A different frame of the same video, orA different video.

The variance in each stratum could then be expressed as a fraction of the variance overall.

To enable comparisons between different variables, we divided the differences by the population standard deviation of that variable. By squaring, we obtained measurement variance as a proportion of population variance.

Results

There were 5,963 pairings of 2 experts measuring the same patient’s A4c LVEDV, of which 426 were same-frame, 2,665 were other-frame, and 2,862 were different-video. The disagreement variance in the same-frame pairings was only 18% of the overall disagreement variance (p<0.001 by F test). All 4 apical volumes showed this pattern: LVEDV and LVESV, in A4c and A2c views (Figure 1 left panels, p<0.01 by F-test). The other 55 directly measurable variables showed a similar pattern: the tracing variability on an identical frame being much smaller than the overall disagreement variability.

Strikingly, the AI-expert differences showed the same pattern (Figure 1 right).

Across all 59 variables both the expert-expert and AI-expert comparisons showed disagreement variance to be dominated by the accident of having chosen a different frame or different video. The results are shown in Figure 2, with same-frame variance and different-frame variance both being significantly smaller than the overall variance (p<0.001 and p<0.001 for expert-expert; p<0.001 and p<0.001 for AI-expert).

Conclusions

The true origin of reduced measurement error from AI measurement is not better accuracy in individual tracing but its comprehensive analysis of all available data.

Individual AI measurements suffer identically to expert measurements from biological variability between different beats (and different acquisitions). However, AI measurements routinely measure every beat and synthesise a summary. Although this has long been recognised as the gold standard for measurement accuracy (1), it has never been practical until the era of AI automation.Figure 1Figure 2