Development of a machine learning ( ML ) model to predict 3‐month response to CGRP monoclonal antibodies in migraine: A pilot evaluation of physicia

Background

Predicting individual treatment responses to anti‐calcitonin gene‐related peptide monoclonal antibodies (CGRP‐mAbs) in migraine remains challenging.

Objective

To develop a machine learning (ML) model for predicting 3‐month treatment response to CGRP‐mAbs and to evaluate its practical clinical utility in improving physicians' predictive performance.

Methods

This retrospective prognostic model development was combined with a prospective, multi‐reader, multi‐case crossover reader study. We used retrospective data from 228 patients with migraine treated with CGRP‐mAbs at the headache outpatient clinic in Dokkyo Medical University from April 2022 to March 2023. The data were divided into training ( n  = 182) and test ( n  = 46) datasets. Phase 1: An ML prediction model was developed using the training dataset with Random Forest and five‐fold nested cross‐validation. The model predicted achievement of a ≥50% reduction in monthly migraine days at 3 months, based on 27 baseline clinical variables. We tested the model's performance on the test dataset. Phase 2: Using the test dataset, five headache specialists and three general neurologists predicted treatment response without the model and then with the model's assistance. Prediction performance was compared between conditions.

Results

Phase 1: The model's performance on the test dataset was as follows: accuracy of 76.1% (95% confidence interval [CI], 62.1%–86.1%), sensitivity (recall) of 87.0% (67.9%–95.5%), specificity of 65.2% (44.9%–81.2%), precision of 71.4% (52.9%–84.7%), F1‐score of 78.4% (64.0%–89.4%), and kappa of 0.52 (0.27–0.74) on the test dataset. Phase 2: The eight participating physicians predicted the treatment effectiveness. Without the model, their performance metrics were median accuracy of 65.2% [interquartile range 61.4%–70.1%], mean sensitivity (recall) of 68.5% ± standard deviation 11.6%, specificity of 62.5% ± 18.4%, precision of 66.3% ± 10.3%, F1‐score of 66.7% [64.5%–71.9%], and kappa of 0.30 [0.23–0.40]. With the model, these metrics were accuracy of 73.9% [72.8%–76.1%], sensitivity (recall) of 79.9% ± 13.1%, specificity of 64.7% ± 13.6%, precision of 70.2% ± 8.1%, F1‐score of 76.6% [74.4%–76.9%], and kappa of 0.48 [0.46–0.52], respectively. The accuracy, sensitivity, F1‐score, and kappa were statistically improved. In the switching analysis, which assessed how physicians' predictions changed after receiving ML model assistance, incorrect‐to‐correct changes occurred in 12.8% (47/368, 8 physicians × 46 cases) of the pooled evaluations, whereas correct‐to‐incorrect changes occurred in 6.0% (22/368). The overall net switching benefit was +6.8%, and incorrect‐to‐correct changes occurred significantly more often than correct‐to‐incorrect changes ( p  = 0.003).

Conclusions

Although external validation is necessary, our findings provide preliminary evidence that ML‐assisted prediction may support the prediction of CGRP‐mAb response.

image