DOI: 10.18485/judig.2026.2.ch18 ISSN:

Linguistic Annotation of South Slavic Languages with Large Language Models

Katerina Zdravkova
Show PDF Cite
This study investigates the applicability of ChatGPT 5 Plus to linguistic annotation in four South Slavic languages: Bulgarian, Macedonian, Serbian, and Slovene. While recent studies have demonstrated the potential of large language models (LLMs) for linguistic annotation, their ability to learn detailed language-specific morphosyntactic annotation conventions remains insufficiently understood. Using the corresponding translations of the first 1235 tokens of Orwell's 1984 together with their MULTEXT-East annotations, we evaluated the model for lemmatization, part-of-speech (PoS) tagging, and morphosyntactic description (MSD) annotation. The evaluation consisted of zero-shot annotation, one-shot calibration, iterative calibration through corrective human feedback, and an independent blind evaluation. Zero-shot annotation demonstrated high lemmatization accuracy and reliable PoS tagging, while iterative calibration substantially improved adherence to language-specific annotation conventions, particularly for MSD annotation. Blind evaluation on previously unseen sentences confirmed high lemmatization (94.95–99.22%) and PoS-tagging accuracy (94.69–96.61%) across all four languages, with MSD accuracy ranging from 75.16% to 89.21%. Most observed errors reflected the model's tendency to rely on own linguistic knowledge rather than consistently adopting the language-specific annotation conventions introduced during calibration. Regardless of their relatively frequent presence, especially during MSD annotation, the current results show that this chatbot can effectively support morphosyntactic annotation using iterative calibration based on the human-in-the-loop concept and highlight the importance of expert feedback for adapting large language models to corpus-specific annotation standards.

More from our Archive