Design and Evaluation of an LLM-Powered Virtual Guide for Rupestrian Churches Using XR Technologies

In recent years, Large Language Models (LLMs) have begun to find application in immersive Extended Reality (XR) environments, raising questions regarding their reliability, grounding, and evaluation in multimodal interaction, especially with Cultural Heritage (CH) contents. To address these challenges, this work presents the design and systematic assessment of an Large Language Model (LLM)-powered virtual guide for rupestrian churches in Matera, implemented within a high-fidelity immersive eXtended Reality (XR) application. The system combines speech transcription, visual object detection, spatial contextualization, and Retrieval-Augmented Generation (RAG) to deliver context-aware responses grounded in curated Cultural Heritage (CH) knowledge. Beyond system implementation, the study frames the XR application as a controlled experimental platform to investigate the causal impact of retrieval augmentation on response quality, grounding fidelity, robustness, and stability. To isolate this effect, a paired RAG ON/OFF design was adopted across multiple large-scale language models (70B–120B parameters), enabling within-query comparisons under identical prompting and decoding conditions. We evaluated the system using a hybrid methodology that integrates automated retrieval-aware metrics (Retrieval Augmented Generation Assessment (RAGAS)) with a blind expert assessment by domain specialists in medieval archaeology. Results demonstrate that retrieval augmentation yields statistically robust improvements in grounding-related metrics, particularly faithfulness and factual correctness, and significantly increases expert-rated accuracy, completeness, coherence, clarity, and overall acceptability. Correlation and delta-based alignment analyses reveal selective convergence between automated metrics and human judgments, highlighting answer relevancy as the most human-aligned diagnostic signal in immersive CH contexts. Stability analyses further indicate that RAG improves epistemic grounding without introducing additional generative variability. These findings support retrieval augmentation as a structural mechanism to enhance reliability in XR-based CH applications and provide a replicable framework for evaluating grounded conversational intelligence in multimodal settings.