Frequency–Time Domain Alignment for Cross-Subject Time-Series Classification Under Distribution Shift

Background: Distribution shift is one of the most pervasive data-related challenges for time-series classification. In sensor-based Human Activity Recognition (HAR), a model trained on one population of subjects often degrades substantially when deployed on another, even with identical hardware and activity labels. This paper addresses that setting specifically: cross-subject distribution shift. Methods: We propose FTDA (Frequency–Time Domain Alignment), an unsupervised domain-adaptation method that processes raw signals and their FFT-magnitude spectra through a dual-branch encoder, combines a Gradient-Reversal adversarial loss on the joint feature with a Multi-Kernel MMD on the frequency branch, and applies a symmetric-KL time–frequency consistency loss on unlabeled target data. Results: On UCI HAR, over four cross-subject transfer tasks and 10 random seeds (n=40 paired observations), FTDA reaches 95.01% target accuracy against 92.46% for Source-only, 92.29% for DANN and 93.32% for a RAINCOAT-style baseline. Because the tasks share subjects and two of them are mirror configurations, no pooled significance test is offered; inference is confined to within-task paired tests (10 seeds each, Holm-corrected), under which FTDA is ahead of every single-view baseline on all four tasks, while the difference from the two dual-view competitors (+1.69 pp against RAINCOAT-style, descriptive task-level interval [−0.40,+3.78]) is not established. Over a common set of seeds, controls widened to FTDA’s parameter and multiply-accumulate budgets shifted accuracy by at most +0.35 pp against DANN (intervals including zero), where FTDA gains +2.97 pp on the same runs, so added capacity alone does not appear to explain the improvement, and a target-only reference model, which is optimistic because it is evaluated on the windows it was trained on, sits 0.70 pp above FTDA. An ablation shows that the frequency branch is useful chiefly because it enables a coupled mechanism of cross-view consistency and auxiliary source supervision (+2.02 pp for that final step) rather than concatenated features (+1.53 pp); for the frequency MMD term, no measurable benefit was detected (−0.57 pp, 95% CI [−1.28,+0.13]), and it can be removed. On a second dataset, HHAR, the cross-subject advantage does not reproduce under cross-device shift: we detect no difference from time-only adversarial baselines (paired difference +0.48 pp against DANN, 95% CI [−1.20,+2.16]; exploratory, since all device pairs share the same subjects), so the benefit of the spectral view does not appear to carry across shift types. Conclusions: Under cross-subject shift, the operative ingredient is not the spectral view as an extra set of features but the coupled consistency-and-auxiliary mechanism that the second view makes possible; the alignment objectives themselves contribute little, and no measurable benefit was detected for the frequency MMD term. FTDA realizes this view at a modest 0.31 M/7.55 MMAC footprint with no OT solver in the training loop and no hyper-parameter that requires target labels. Transfer across devices and operating environments is not evaluated here and remains open for future work.