PaCMoE: Pairwise Cross-Modal Attention Fusion with Mixture-of-Experts for Stock Movement Prediction

Stock movement prediction remains challenging because financial time series are nonstationary, noisy, and interdependent across heterogeneous information sources. Conventional multimodal forecasting methods often integrate price, text, and inter-firm relationships into a single representation, making it difficult to capture modality-pair-specific interactions and their varying relevance across market conditions. To address this issue, this study proposes Pairwise Cross-Modal Attention Fusion with Mixture-of-Experts (PaCMoE), which explicitly decomposes multimodal interactions into three modality pairs—Price–Text, Price–Graph, and Text–Graph. Each pair is modeled by an independent Cross-Modal Attention expert, and a Dense Mixture-of-Experts router adaptively integrates the resulting representations according to the input. This design enables separate modeling of pairwise interactions and input-dependent weighting of their contributions. The modality encoders combine PatchTST for prices, FinBERT with canonical correlation analysis for text, and a single-relation R-GCN for daily correlation graphs. PaCMoE is evaluated using chronological data splitting, validation-based model selection, and ten random seeds on the ACL18 (StockNet) benchmark. The proposed method achieves an average accuracy of 60.69% and an MCC of 0.2200, which are comparable to or higher than the multi-seed values reported for LSTM-based Adv-ALSTM and Transformer-based DTML. Additional evaluations include single-run ablation studies, three-class prediction incorporating neutral days, precision–recall analysis, and visualizations of router and attention weights to examine component contributions and model behavior. Training takes approximately 30 min per run on a single NVIDIA GeForce RTX 3090. These results support the effectiveness of explicitly modeling pairwise cross-modal interactions and adaptively integrating their contributions for stock movement prediction on the ACL18 benchmark.