Spatio-Temporal Segmentation of Eye Movements During Complex Decision Making Using Switching Hidden Markov Models

Understanding how visual exploration evolves over time during complex decision making remains challenging, as the dynamics of underlying cognitive processes are difficult to observe. Hidden Markov models are widely used to model eye movement sequences, but their assumption of stationary transition probabilities limits their ability to identify successive phases. This study investigated whether Switching Hidden Markov Models (SHMMs), trained only on fixation positions, could segment visual exploration during an obstacle-avoidance task requiring a left/right decision. Individual SHMMs with two or three high-level (HI) states were trained without using reaction times. HI states were characterized through transition-matrix variability, saliency maps, cumulative dwell times in regions of interest, and their association with reaction times. Both models produced temporal segmentations consistent with decision making. Early HI states showed greater interindividual variability and exploration focused on obstacles, whereas later states were more homogeneous and reflected decision confirmation followed by reorientation toward the center of the screen. The transition to the final HI state of the three-state model best explained reaction time. Furthermore, the average log-likelihood of the Viterbi state restoration improved the fit quality of this explanatory model, indicating that SHMM confidence provided complementary behavioral information. These findings suggest that SHMMs trained only on fixation positions can identify phases broadly consistent with those traditionally observed in the decision-making process, such as early evidence accumulation and late decision confirmation.