DOI: 10.35377/saucis...1939701 ISSN: 2636-8129

Multi-Task Transformer Architectures for Simultaneous Sentiment and Urgency Classification of Turkish Social Media Crisis Communications

Abdulrahman Alsamaraie, Can Yüzkollar
Show PDF Cite
After the Kahramanmaras¸ earthquakes on 6 February 2023, approximately one million online messages related to the crisis were generated in a short period of time. Although these online messages contain abundant information, they also include a mixture of urgent requests and noise. Furthermore, the short length of such messages and the difficulty of interpreting them during crisis situations make it challenging to extract critical information. Instead of modeling emotion and urgency as separate components, an integrated model may reveal deeper underlying patterns in Turkish crisis discourse. In the revised experimental design, 25,000 tweets were manually annotated. Sentiment analysis was formulated as a binary task with two labels, negative and neutral, while urgency was defined as a multi-class task with five escalating labels from U1 to U5. Within this revised setting, classical TF–IDF methods, sequence-based neural models, and transformer-based architectures were implemented as baseline approaches. Subsequently, a shared-encoder multi-task framework with separate heads for sentiment and urgency was developed. In the single-task setting, BERTurk (cased) achieved the highest Macro-F1 and AUC scores for sentiment classification, while XLM-RoBERTa-large (thresholded) obtained a slightly higher accuracy. BERTurk (cased) achieved 92.44% accuracy, 87.17% Macro-F1, and 95.39% AUC for sentiment classification. For urgency classification, BERTurk (cased) achieved the strongest overall performance, reaching 77.20% accuracy, 77.11% Macro-F1, and 94.35% Macro-AUC. In the multi-task setting, XLM-RoBERTa-large achieved the strongest joint performance among the evaluated models. It reached 90.62% accuracy, 85.26% Macro-F1, and 95.66% AUC for sentiment classification, together with 79.18% accuracy, 78.50% Macro-F1, and 95.30% Macro-AUC for urgency classification. The manually labeled training and evaluation dataset used in this study is publicly available at https://github.com/AbdalrahmanM/turkish-crisis-sentiment-urgency-dataset.

More from our Archive