A “foundation model” for precision coronary plaque phenotyping: integration of multi-modal imaging, laboratory, and clinical data via contrastive learning
S K Kim, J C Jakub Chmiel, S A Stefano Andreaggi, I M Iris McNulty, H L Hang Lee, Q L Quanzheng Li, K K Kyungsang Kim, T K Tsunekazu Kakuta, I K J Ik-Kyung JangAbstract
Background/introduction
Although atherosclerosis is a systemic inflammatory disease, contemporary methods for predicting future cardiovascular events rely on a non-integrated, single diagnostic approach such as cardiovascular risk factors, biomarkers, or imaging.
Purpose
To develop and validate a multi-modal foundation model integrating clinical information, laboratory biomarkers, and optical coherence tomography (OCT) image data to enable zero-shot precision plaque phenotyping and interactive clinical assessment.
Methods
We trained the model via contrastive learning on 1 million OCT frames from 1,729 patients with coronary artery disease. OCT images were aligned with clinical data, laboratory biomarkers, demographic characteristics, and cardiovascular risk factors. The framework was subsequently extended to a vision–language foundation model based on an LLaVA backbone.
Results
A novel foundation model was developed which immediately answers to any questions related to OCT images (Figure A). The features of plaque vulnerability were identified with high precision: thin-cap fibroatheroma (TCFA) (AUC 0.86), lipid-rich plaque (LRP) (AUC 0.80), macrophages (AUC 0.72), microvessels (AUC 0.75), and cholesterol crystals (AUC 0.75). The model also identified plaque rupture and erosion with AUCs of 0.96 and 0.94, respectively. (Table A and B)
Conclusion
This multi-modal foundation model enables precision coronary phenotyping by integrating intravascular plaque morphology, laboratory biomarkers, and clinical data. Its interpretable, open-ended OCT visual question-answering interface provides the instantaneous, accurate diagnosis of plaque phenotype.For image description, please refer to the figure legend and surrounding text.For image description, please refer to the figure legend and surrounding text.