DOI: 10.54569/aair.1971719 ISSN: 2757-7422

Development of a Hybrid Artificial Intelligence System to Reduce the Risk of False Positives in Turkish Emails

Hüseyin Kaplan, Fatih Ahmet Şenel
Show PDF Cite
The detection accuracy of spam filters has improved significantly over the years; however, the false positive rate of these filters can result in legitimate corporate emails being incorrectly routed to spam folders. In this study, an integrated hybrid artificial intelligence system was developed to reduce the risk of false positives in Turkish emails. The system consists of two modules: (i) a detection module comparing Naïve Bayes, Logistic Regression, and Linear Support Vector Machine (Linear SVM) classifiers using TF-IDF features, and (ii) a text smoothing module based on mT5-small, retrained using the task transfer learning method. In the detection module, the Linear SVM model achieved the best performance with a spam-class F1 score of 98.86% on a balanced dataset of 6,727 examples, which was compiled, cleaned, and organized from five public data sources and additional examples collected during the project. In a subsequent controlled re-evaluation where records with high similarity were filtered out, the F1 score adjusted to approximately 97.3%. The smoothing module was retrained using a total of 1,235 paired samples, one focused on pure advertising content and the other on legitimate but aggressively toned corporate email patterns. False positive reduction effectiveness was evaluated in a classifier-based simulation environment using a held-out test set containing 118 emails not included in the training. In the high-risk subset (>50% spam likelihood), the average spam score dropped from 74.5% to 41.0% (a decrease of 33.4 points); 25 out of 40 emails (62.5%) were reclassified as normal/legitimate after softening. Output quality was evaluated with a BERTScore F1 of 0.7146; it was observed that brand, URL, and numerical information were preserved at rates of 100%, 91.4%, and 93.8%, respectively.

More from our Archive