Large language model developments

Abstract

Large language models (LLMs) are generative artificial intelligence (AI) models trained with vast amounts of data and context, and excel at processing, understanding and generating human language, using transformer models to learn statistical relationships between tokens. LLMs are a subset of transformer models, and hence are closely related but not interchangeable. Given the ‘explosion’ of LLM research, the challenge is tracking emerging developments and matching LLMs to applications. This paper reviews LLM developments with the aim of understanding the underlying models’ computational strengths and weaknesses, building on previous work focusing on LLM optimization (Sharkey & Treleaven 2025 Optimising large language models: taxonomy and techniques. SSRN Scholarly Paper no. 5278456. Rochester, NY: SSRN). We find that despite their power, LLMs (transformers) face several key challenges. These include: (i) computational complexity and resource requirements—training and deploying large models is extremely costly and slow, and energy consumption is a concern; (ii) understanding long sequences and dependencies—even with enhancements, models can struggle with very long-range dependencies or context lengths beyond their window (e.g. maintaining coherence in a long document), which reflects the more fundamental absence of continual learning; (iii) interpretability and bias—it remains difficult to interpret why a model produces a given output, and models can inadvertently amplify biases present in training data; and (iv) optimization challenges, including fine-tuning for specific tasks without overfitting or losing generality and ensuring efficient inference (latency, the delay before a response is completed) for real-time applications. In summary, this review focuses on architectural, optimization and interpretability developments in LLMs. It does not aim to provide a systematic benchmark comparison, exhaustive implementation details or empirical performance evaluation, which are covered extensively elsewhere. The authors are committed to updating this review aligned to constant changes in the technology; please reach out with any additions.