Application of LLM for Multitasking Workload Prediction in Mixed‐Reality Environments

ABSTRACT

Advancements in mixed‐reality technologies have enabled increasingly complex multitasking scenarios, amplifying the need for accurate workload prediction to support effective human–machine collaboration. Although the NASA Task Load Index (NASA‐TLX) remains the most widely adopted subjective workload assessment tool, its tabular nature and limited sample availability constrain the application of advanced learning models. This study proposes a novel hybrid workload prediction framework that integrates large language models (LLMs) with NASA‐TLX data and generative adversarial networks (GANs) for data augmentation. Specifically, categorical task‐difficulty descriptors derived from mixed‐reality multitasking conditions are encoded as text inputs and processed with a pre‐trained BERT model, whereas numerical NASA‐TLX scores and participant attributes are modelled using a neural network. To address data scarcity and class imbalance inherent in human‐subject experiments, a GAN‐based tabular data augmentation strategy is used to generate 5000 high‐quality synthetic samples that preserve the statistical characteristics of the original dataset. The hybrid architecture jointly optimises representations from both textual and numerical modalities using a weighted loss function, enabling deeper semantic modelling of workload‐related context beyond traditional neural networks. An experimental study involving 36 participants who performed mixed‐reality multitasking tasks demonstrates that the proposed approach significantly outperforms a conventional neural network, achieving a substantially lower root mean square error (RMSE) for workload prediction. The primary novelty of this work lies in the explicit fusion of LLM‐based semantic encoding with NASA‐TLX workload measures and GAN‐augmented tabular data, offering a scalable and data‐efficient framework for predictive workload modelling in mixed‐reality environments.