DOI: 10.35377/saucis...1865546 ISSN: 2636-8129

A Semantic and Lexical Analysis of LLM-Generated Responses in Comparison with Stack Overflow Answers

Kerem Can Çelepkolu, Yasin Ortakcı, Kürşat Mustafa Karaoğlan
Show PDF Cite
Large Language Models (LLMs) have become indispensable tools in programming education and professional software development. However, their output quality remains poorly characterized, particularly in terms of the semantic and linguistic quality of the explanatory text. Traditional evaluation metrics such as ROUGE primarily rely on lexical overlap and therefore only partially reflect the semantic adequacy of code explanations. This study proposes a multi-metric evaluation framework integrating semantic embeddings and lexical metrics to assess the quality of explanations generated by LLMs. We conducted experiments on 442 question-answer pairs produced by human experts regarding Python programming language from Stack Overflow, comparing them with GPT-4o-mini outputs. Three embedding models (MiniLM-L6, MPNet-Base, Paraphrase-MiniLM) were employed to compute the measure of semantic similarity by using cosine similarity, and ROUGE metrics (ROUGE-1, ROUGE-2, ROUGE-L) were used for lexical analysis. Results reveal a high semantic alignment (cosine similarity: 0.65–0.75) but substantially lower lexical overlap (ROUGE:0.10–0.32), indicating that LLMs produce semantically sound yet lexically distinct explanations. Pearson correlation analysis reveals weak to moderate correlations (r=0.29–0.47) between semantic and lexical metrics, suggesting that traditional metrics alone are insufficient for a comprehensive evaluation. Statistical testing (paired t-tests, p < 0.001) shows significant differences among embedding models, with MPNet-Base achieving the highest semantic consistency. Our findings underscore the critical need for multidimensional evaluation frameworks in assessing LLM-generated explanations, particularly in technical domains where semantic accuracy is crucial.

More from our Archive