DOI: 10.18485/judig.2026.2.ch9 ISSN:

Comparative Intrinsic Evaluation of LLM Tokenizers on Serbian Text

Mihailo Škorić, Nikola Janković, Tanja Samardžić
Show PDF Cite
This study presents a comparative intrinsic evaluation of large language model tokenizers trained specifically for Serbian, a language characterized by rich morphology and dual script usage. We construct several tokenizers using diverse strategies, including GPT2-style and trailing space BPE, as well as some experimental approaches: minimum redundancy exploration, casing and script compression, and morphology-aware tokenization (based on a Serbian morphological dictionary). In addition, we benchmark tokenizers from contemporary models such as Qwen3.5 and Mistral3.5. Training is conducted on a Serbian corpus of approximately one billion words, with evaluation performed on a held out set one-third of that size. Intrinsic properties are analysed using Tokenizer Analysis Suite, calculating metrics such as fertility, compression ratio, Rényi Entropy and MorphScore. Our results highlight systematic differences different tokenizer building strategies: morphology driven approaches achieve higher MorphScore, while script compression strategies result in lower fertility and improved compression, especially while handling mixed Cyrillic–Latin corpora. In comparison, existing general purpose tokenizers reveal that they are consistently inferior to language specific tokenizers across most intrinsic metrics, underscoring the importance of tailored tokenizer design.

This work provides the initial comparative study of Serbian tokenizers, offering insights into how design choices affect the intrinsic properties. By systematically evaluating both newly trained and existing models, we establish a foundation for future research on tokenizer development for morphologically rich languages, and contribute benchmarks that can help future designing of tailored, specific purpose tokenizers.

More from our Archive