Memorization and forgetting in a learning Hopfield neural network: Bifurcation mechanisms, attractors, and basins

Despite the explosive expansion of artificial intelligence based on artificial neural networks (ANNs), these are employed as “black boxes,” since it is unclear how, during learning, they form memories or develop unwanted features, such as spurious memories and catastrophic forgetting. Much research is available on isolated aspects of learning ANNs, but due to their high dimensionality and non-linearity, their comprehensive analysis remains a challenge. Here, we comprehensively analyze mechanisms of memory formation and destruction in an 81-neuron Hopfield network undergoing Hebbian learning. We propose a method to analyze bifurcations in a learning ANN in terms of the stage of learning. We discover that individual memories, represented by attractors and their basins, are formed and destroyed, thanks to bifurcations. We show that the same bifurcations cause both the abrupt memory loss while the ANN is trained on a single task, and the conventional catastrophic forgetting while it switches between training tasks. We point to a possible mechanism of spurious-memory formation. We reveal the structure of attractor basins formed at the end of learning. Our strategy to analyze high-dimensional learning ANNs is principally applicable to recurrent ANNs of any form. The demonstrated mechanisms of memory formation and of catastrophic forgetting shed light on the operation of a wider class of recurrent ANNs and could aid the development of approaches to mitigate their flaws.