Configuration Factors for Compact LLM‐Based Story Point Estimation: A Controlled Project‐Level Empirical Study

Story point estimation is widely used in agile software planning, yet automated estimation remains challenging because issue descriptions and story point distributions vary substantially across projects. This study examines configuration sensitivity and comparative performance in compact large language model (LLM)‐based story point estimation through a controlled project‐level study using five TAWOS projects and two ~1–1.5B‐parameter compact backbones with frozen parameters and trainable prediction heads. The study comprises four‐factor configuration analysis covering maximum sequence length, metadata concatenation, learning rate, and soft‐target mixing; metadata‐free sensitivity analysis; learning‐rate–soft‐target‐mixing interaction analysis; comparison with three aligned baselines; assessment of alternative temperature‐selection objectives; and computational‐cost evaluation. Soft‐target mixing and learning rate show the larger average mean absolute error (MAE) effects among the four evaluated factors, with mean high‐minus‐low effects of −0.0535 and −0.0458, respectively, although both vary in direction across project–backbone–seed scenarios. The metadata‐free analysis shows similar average effect magnitudes for learning rate and soft‐target mixing, while the interaction between these two factors is heterogeneous, with 11 negative and 9 positive scenario‐level contrasts. When compact configurations and baseline methods are selected independently using seed‐averaged validation MAE, the corresponding test results favor the compact configuration in three projects and the selected baseline in two, with additional project‐specific variation among TFIDF‐LGBM, MiniLM‐MLP, and CodeBERT‐FT. Temperature‐selection rules produce different MAE, snapped‐expected calibration error (ECE), and negative log‐likelihood (NLL) profiles. The compact configurations require very small trainable prediction heads but retain substantially higher inference latency and memory requirements than the evaluated baselines; method‐level mean raw‐text‐to‐logits latency ranges ~0.37–25.72 ms/sample in the reported NVIDIA A10 environment. Overall, the results characterize compact‐LLM story point estimation by heterogeneous configuration sensitivity, project‐dependent model ranking, and accuracy–cost trade‐offs rather than a single consistently preferable configuration or model family.