From General Audio to Sports Vibration: Cross-Modal Learning for Vibration-Based Crowd Monitoring

Crowd monitoring in sports stadiums is essential for public safety and audience experience. Traditional methods rely on manual observation, cameras, and microphones, which can be disruptive and raise privacy concerns. Floor vibration sensing offers a non-intrusive alternative, but the lack of open-source vibration datasets limits data-driven model development.

In this paper, we propose Vi bration L everages A udio (ViLA) , a vibration-based crowd monitoring framework that reduces reliance on labeled vibration data through cross-modal pre-training. Specifically, ViLA is first pre-trained on general-purpose audio data in an unsupervised manner, then fine-tuned using a small amount of labeled vibration data. This approach enables the model to learn general spectral representations from audio and transfer this knowledge to the vibration domain, which is especially useful in environments with high variability and limited sensing duration, such as live sports events.

We evaluate ViLA in two stadium environments across different sports. We also analyze audio–vibration relationships to understand how pre-training influences model performance. In a preliminary single-run modality comparison, general-purpose audio produced the highest accuracy among the evaluated pretraining modalities. Across five-seed same-day evaluations, audio pretraining improved performance in some games, particularly when the scratch baseline was weak, while the magnitude of benefit varied across games.