On the Diffusion of Test Smells in LLM-Generated Unit Tests

LLMs promise to transform Java unit test generation from a manual burden into an automated solution. Yet, beyond metrics such as compilability or coverage, the quality of LLM-generated Java tests (particularly their susceptibility to test smells, design flaws that undermine readability and maintainability) remains insufficiently understood. Recent studies have begun to examine test smells and broader quality attributes in LLM-generated tests, but a comprehensive, multi-benchmark analysis of test smell diffusion in Java unit tests, comparing LLM-generated suites with SBST-generated and human-written tests under multiple detectors, is still missing. This paper presents a multi-benchmark, large-scale analysis of test smell diffusion in LLM-generated Java unit tests. We contrast LLM outputs with human-written suites (as the reference for real-world practices) and SBST-generated tests from EvoSuite (as the automated baseline), disentangling whether LLMs reproduce human-like flaws or artifacts of synthetic generation. Our aim is to systematically characterise the prevalence, distribution, and co-occurrence of test smells (what we term their diffusion) in LLM-generated Java tests across models, prompting strategies, and generation granularities, and to compare these diffusion patterns with SBST-generated and human-written suites. Our study draws on 20,505 class-level suites generated by four LLMs (GPT-3.5, GPT-4, Mistral 7B, Mixtral 8×7B) over three Java benchmarks (Defects4J, SF110, and a curated dataset, CMD), 972 method-level cases from TestBench generated by three LLMs (GPT-3.5, GPT-4, CodeLlama-13B-Instruct), 14,469 EvoSuite tests, and 779,585 human-written tests from 34,635 open-source Java projects (CAT-LM dataset, SF110, Defects4J). Using two complementary detection tools (TsDetect and JNose) together with a manually validated sample of 240 classes and 447 methods, we analyze prevalence, co-occurrence, and correlations with software attributes and generation parameters. Results show that LLM-generated Java tests consistently manifest smells such as Assertion Roulette and Magic Number Test, with patterns strongly influenced by prompting strategy, context length, and model scale. Comparisons reveal overlaps with human-written tests, especially under certain detectors, which are consistent with shallow imitation of common testing idioms or possible training-data overlap, while EvoSuite exhibits distinct, generator-specific flaws. These findings highlight both the promise and the risks of LLM-based Java test generation, and call for the design of smell-aware generation frameworks, prompt engineering strategies, and enhanced detection tools to ensure maintainable, high-quality test code.