A Hybrid Anomaly Detection Framework for Reliable Physiological Signal Extraction in Multimodal Wearable Sleep Monitoring

Wearable sleep monitoring systems provide a scalable and low-burden alternative to laboratory-based polysomnography, but overnight physiological recordings collected from wearable sensors are frequently corrupted by poor skin contact, sensor displacement, flatline behavior, saturation, abrupt autoscaling, outliers, and non-physiological noise. These low-quality segments can prevent reliable extraction of clinically relevant biomarkers, including heart rate, heart rate variability, pulse rate, oxygen saturation, and electrodermal activity, thereby limiting the robustness of downstream sleep stage classification and sleep apnea prediction. In this work, we present a multistage hybrid anomaly detection framework for signal quality assessment in a custom-designed multimodal wearable sleep monitoring platform. The proposed framework is shared across modalities: each sensor stream is segmented into windows, screened using signal processing algorithms for obvious sensor failures, and then analyzed using an autoencoder trained on normal windows to detect subtler morphology-level deviations from normal physiological patterns. On chest ECG recordings collected over 10 overnight sessions, the hybrid fusion detector achieves an accuracy of 0.9067, AUROC of 0.9068, F1 score of 0.8739, precision of 0.9470, and recall of 0.8202, outperforming rule-based detection and autoencoder-based detection alone. Additional experiments on PPG, EDA and ExG recordings show consistently high F1 scores of 0.9431, 0.9567 and 0.8728, respectively. These results demonstrate that hybrid anomaly detection can serve as an effective quality-control layer for multimodal wearable sleep monitoring and provide a foundation for future work on robust sleep stage and apnea prediction under real-world sensing conditions.