DOI: 10.3390/aieng1020010 ISSN: 3042-8831

Quantifying the Stability–Recovery–Interpretability Trade-Off Between K-Means and Self-Organizing Maps for High-Dimensional Imbalanced Data

Imtiaz Ahmed, Hamdy Soliman

High-dimensional engineering datasets often combine class imbalance, noisy structures, and limited ground truth, making unsupervised analysis difficult to evaluate reliably. This study quantifies how three properties—partition stability, minority class recovery, and topological interpretability—are traded off across clustering methods, using a capacity-matched 25-seed comparison on a TCGA-derived RNA expression dataset (10,095 samples, 19 cancer types, 13,634 genes). We compare K-means across cluster counts k∈{19,…,400}, self-organizing maps (SOMs) across lattice sizes from 25 to 625 nodes, consensus K-means, a granularity-matched SOM-Super20 control, and four modern baselines (HDBSCAN, spectral clustering, Gaussian mixtures, and Leiden). At matched prototype budgets, K-means is both more reproducible and substantially better at recovering minority classes than SOMs: at 400 prototypes, K-means achieves pairwise NMI 0.819 versus 0.621 for the 20×20 SOM and recovers the smallest cancers 6–14× more effectively (pancreas effective coverage 0.760 vs. 0.054).Crucially, the SOM does not close this gap even when given more prototypes (0.07 at 625 nodes), so, under matched capacity, minority recovery is better explained by representational capacity and centroid allocation freedom than by topology preservation. The recovery is not free: increasing k overfragments the partition and lowers the pairwise ARI stability (0.643→0.419 from k=20 to k=400), while the NMI remains robust (≈0.82). The hardest minority, pancreas, is recovered only by high-capacity K-means and by no other method evaluated, including SOMs at any size, consensus K-means, SOM-Super20, HDBSCAN, Gaussian mixtures, spectral clustering, and Leiden. The SOM’s distinct value is therefore not stability or recovery but the interpretable two-dimensional topological visualization that it uniquely provides, including a gradient-organized structure that is reproducible across seeds for kidney (weaker for uterus). No single method optimizes all three properties; the appropriate choice depends on whether a task prioritizes reproducibility, minority recovery, or visual interpretability. Because these conclusions follow from the shape of the data and the allocation behavior of the algorithms rather than from biological semantics, we expect them to transfer to high-dimensional imbalanced engineering data, such as those from fault clustering, condition monitoring, and anomaly detection.

More from our Archive