DOI: 10.18485/judig.2026.2.ch26 ISSN:

Approaches to Automatic Annotation of Semantic Roles and Syntactic Relations

Svetlozara Leseva, Ivelina Stoyanova
Show PDF Cite
We investigate the capabilities of several openly available large language models (LLMs) for the automatic annotation of frame-semantic structure in Bulgarian, a morphologically rich, pro-drop Slavic language for which the application of LLMs to FrameNet-style annotation remains largely unexplored. The study includes a controlled pilot annotation on 350 sentences containing cleaning verbs (e.g. чистя, мета, мия, бърша) that evoke two closely related FrameNet frames, Removing and Emptying , which share the same core elements —
Agent
,
Theme
,
Source
, but differ in their syntactic profiling which distinguish frame-semantic alternations. We evaluate three models — Gemma 4 E4B-IT, BgGPT-Gemma-3-12B-IT and Qwen3-8B, on frame element assignment, with the additional evaluation of four annotation subtasks (clause-boundary identification, phrase span identification, coreference resolution, pro-drop resolution). The Bulgarian-adapted BgGPT-Gemma-3-12B-IT achieves the highest scores on all five tasks (F 1 of 58.4), followed by Qwen3-8B with chain-of-thought reasoning (50.9) and Gemma 4 E4B-IT (38.7). Bulgarian fine-tuning matters most where morphosyntax is decisive — pro-drop, clause segmentation, phrase boundaries, while reasoning partially compensates on the more conceptual tasks. The results motivate further work on retrieval-based prompt construction, iterative self-refinement, and supervised fine-tuning.

More from our Archive