DR-Transformer: A Dual-Regularized Transformer Combining Sparse Attention and Supervised Contrastive Learning for Interpretable Stress Detection in Social Media Text

Automatic detection of stress in social media text holds promise for supporting digital mental health, but most existing Transformer-based approaches are opaque and computationally demanding. This work presents DR-Transformer, a Dual-Regularized Transformer that combines two complementary mechanisms: (i) a group sparsity penalty (L2,1/L2 elastic net) applied to the query and key projection matrices of every attention head, which encourages whole-row sparsity, producing more concentrated and inspectable attention patterns; (ii) a supervised contrastive loss on the [CLS] projection, which organizes the latent space according to the stress label. The architecture is intentionally lightweight (six layers, eight heads, 256-dim embeddings; ∼9.5 M parameters) and runs entirely on consumer-grade hardware (NVIDIA GTX 1660, 6 GB). Experiments on the publicly available Dreaddit dataset (binary stress classification, 2838 train/715 test segments) compare DR-Transformer against Logistic Regression, BiLSTM, a Standard Transformer of identical architecture, and MentalBERT. Across five seeded runs, DR-Transformer (Full) reaches F1=0.876 (bootstrap 95% CI 0.852–0.898), outperforming the Standard Transformer (F1=0.842; McNemar p<0.001 with Bonferroni correction) and performing comparably to the much larger MentalBERT (F1=0.879; p=0.421). Sparse regularization increases the fraction of near-zero attention weights (below 0.01) from 0.215 to 0.682, while the supervised contrastive loss improves the silhouette score of [CLS] embeddings from 0.312 to 0.483. Dual regularization thus combines accuracy, efficiency, and structurally induced attention concentration in a single model which can be trained without specialized infrastructure. We use the term “interpretable” throughout in this restricted, structural sense—to refer to concentrated and inspectable attention—rather than in the sense of established causal or mechanistic faithfulness; this is only partially and indirectly supported by our token deletion analysis.