Annotation-Free Vision–Language Concept Bottleneck Models for Interpretable Medical Image Classification: A Review

Concept Bottleneck Models (CBMs) route predictions through a layer of human-understandable concepts, but classical CBMs need a concept label for every training image, a cost that is high in medicine. Vision–language models (VLMs) remove that requirement because, in a shared image–text embedding space, a large language model can propose candidate concepts and contrastive similarity can score them, with no concept labels. This review surveys annotation-free VLM-based CBMs and their move into clinical imaging. We cover vision–language foundations (CLIP, ALIGN, LLaVA) and trace concept models from early supervised methods (CBM, CEM, TCAV) to versions that work without manual labels (LaBo, Label-Free CBM, PCBM, OpenCBM, VLG-CBM). We then map these foundations to specific medical fields such as radiology, pathology, cardiology, dermatology, and ophthalmology, highlighting the medical VLM-CBMs closest to actual deployment (CBVLM, AdaCBM, ConcepPath, CBM-RAG). We focus on information leakage, in which task information reaches the prediction through the concept layer without being carried by the named concepts. Two of the seven surveyed medical VLM-CBMs test whether their predictions depend on individual concepts, but none measures leakage, and no shared protocol exists. We identify this gap as a central challenge for interpretable medical AI.