Automated Multimodal Driver Stress State Estimation Using Contrastive Learning-Based Missing Data Imputation

Abstract

Stress is a major contributing factor to various lifestyle diseases as well as mental and emotional disorders. Stress can be detected using biosignals acquired from wearable sensors. Wearable sensors often suffer from missing or corrupted data due to factors such as poor sensor contact and calibration issues.Existing approaches often address this problem by discarding incomplete data or applying simple averaging and interpolation methods, which are error prone. In this study, we propose a contrastive learning-based framework to handle missing data in multimodal driver stress assessment. For this, a dataset with N=30 was collected using a Hexoskin wearable sensor in a controlled driving scenario comprising Electrocardiogram, Abdominal Respiration, and Thoracic Respiration signals. An imputation module with stacked autoencoders refines these features for stress classification and can reconstruct missing modality features during prediction. Experimental results demonstrate that the framework was able to classify five levels of stress with one modality missing with an average accuracy of 81.4% and average F1-score = 81.2% across 5 fold cross validation. The proposed model learns modality-invariant feature representations by aligning all three modalities to a shared latent space during pretraining. Moreover, ECG features were imputed most effectively with classification accuracy of 87.5% and an F1 score of 89%.