Integrating Real Tool Interaction and Multimodal Operator Monitoring in Immersive Simulation for Human-Centred Assessment

The transition from Industry 4.0 to Industry 5.0 is increasing the need for design approaches that place human centrality, safety, and ergonomics at the core of system development. In this context, immersive simulation is evolving from a training-oriented technology into a controlled environment for the observation of human behaviour and task execution. However, many Virtual Reality applications still rely on generic interaction devices that are not sufficiently representative of real tool-mediated operations, limiting the reliability of ergonomic and behavioural assessment. This paper proposes an anthropocentric framework for human-centred assessment in immersive simulation. The framework integrates four main components: a task-oriented Scenario Digital Twin, a Physical-Tool-in-the-Loop module based on a real instrument synchronised with its virtual counterpart, an Interaction Engine for state-dependent action management, and an Operator-in-the-Loop module coupled with a Human-Centred Assessment Layer. The framework is instantiated through an immersive hazelnut pruning simulator, selected as a representative case study because pruning involves non-neutral postures, irreversible actions, and strong dependence on tool handling. The reference implementation combines reality-based reconstruction of hazelnut trees, interactive branch-cutting logic, an instrumented electric pruning shear, full upper-body embodiment, and multimodal data acquisition. In particular, the proposed architecture supports the extraction of body motion through markerless multi-camera pose estimation and the acquisition of eye-related variables through the head-mounted display. A preliminary experimental session demonstrates the technical feasibility of the proposed architecture by showing that multimodal data, including reconstructed 3D body kinematics and eye-tracking signals, can be successfully acquired during immersive task execution, providing a basis for subsequent ergonomic analysis. The results show that a real instrumented tool, a task-oriented digital twin, and a continuous monitoring pipeline can be integrated within a single immersive platform, as well as that the assessment pipeline is sensitive to differing postural configurations during simulated task execution. They do not establish equivalence between behaviour in the simulator and behaviour during real-world pruning; dedicated cross-modal validation is therefore required. The framework is intended to support the evolution of immersive simulation toward assessment-oriented applications, contributing to a broader Safety-by-Design perspective in which human behaviour and interaction quality are considered as early-stage design inputs.