Emotion AI on the Edge: A TinyML ‐Driven Framework for Speech Emotion Recognition in Social Environments

ABSTRACT

The rise of Emotion AI is transforming how human emotions are detected, interpreted and responded to in real‐time social systems. Speech emotion recognition (SER) enables machines to understand human emotions from voice, fostering more empathetic and context‐aware interactions. However, the currently deployed SER systems are frequently based on highly resource‐consuming large models, which are unsuitable for real‐time deployment on edge devices with limited memory and processing power. Most previous studies have focused on single‐language or single accent datasets, which results in ineffective extrapolation to various speakers, accents and the real world. This study introduces a scalable and lightweight SER system designed for the TinyML environment and suitable for deployment on resource‐constrained systems, including social networks using IoT technologies, assistive technologies and embedded mental health devices. Combining RAVDESS, TESS and SAVEE increases dataset diversity. Their effectiveness in capturing both spectral and temporal emotion cues is tested across six hybrid frameworks of deep learning, such as CNN + BiLSTM and CNN + BiGRU with multi‐head attention. The best model achieved 74.15% accuracy, with a macro F1‐score of 0.73, a weighted F1‐score of 0.74, and a highest class‐level F1‐score of 0.82, supporting low‐latency emotion recognition for affect‐aware edge applications. The quantization through TensorFlow Lite further reduces the model size by up to 94.5% and achieves inference latency as low as 3.4 ms, making it suitable for deployment on microcontrollers. This study contributes to Emotion AI as it allows detecting emotions on edge devices to facilitate affect‐aware customer service, support mental health, improve education and more broadly, computational social systems.