Fuzzy Adaptive Adam (FA-Adam): A Hierarchical Fuzzy-Logic-Enhanced Adaptive Optimizer

This study introduces Fuzzy Adaptive Adam (FA-Adam), a novel optimization method that enhances the Adam algorithm through a hierarchical, two-level fuzzy inference framework. Although Adam is widely used due to its adaptive moment estimation and strong empirical performance, its fixed update dynamics limit its ability to adapt to the non-stationary nature of deep learning training. Optimization typically evolves through distinct phases, where early training benefits from aggressive and exploratory updates under high gradient noise, while later stages require more stable and conservative behavior. FA-Adam addresses this limitation by enabling dynamic, phase-aware adaptation of optimization behavior throughout training. The proposed method integrates fuzzy inference directly into the optimizer’s internal update process rather than using it solely for external hyperparameter tuning. An upper-level fuzzy system analyzes epoch-level indicators, including loss trends, gradient variance, oscillatory behavior, and convergence rate, to classify the training state as stable, balanced, or fast. Based on this classification, a lower-level fuzzy system adaptively modulates the first- and second-moment estimates of Adam through specialized aggregation operators that blend conservative and aggressive update strategies. This hierarchical design enables smooth transitions between exploration and exploitation while preserving the stability of Adam. We evaluate FA-Adam across 35 experiments involving five neural architectures and seven benchmark datasets. FA-Adam outperforms standard Adam in 29 settings, with two ties and four losses, achieving an average test accuracy improvement of +3.69%. Notably, it delivers substantial gains in challenging training regimes, including cases where Adam fails to converge, and consistent improvements on difficult classification benchmarks. A comparison with modern adaptive optimizers on a representative subset of configurations shows that FA-Adam clearly outperforms Adam and AdamW and is competitive with RAdam and AdaBelief, while a sensitivity analysis indicates robustness to its main hand-tuned constants.