SoleSense: A Shoe-Based Wearable for Sound Event Localization and Detection in Urban Environments

Detecting and localizing acoustic events from a person or device is important for a wide range of applications, including improving pedestrian safety, increasing situational awareness, or for interacting with digital assistants. Compared to radio frequency waves, acoustic waves are naturally more affected and attenuated by the atmosphere, making the range of sensing limited. Additionally, naturally quiet sounds like footsteps are difficult to detect. We propose SoleSense, a shoe-based platform with microphone arrays attached to the soles of both shoes, for sound event localization and detection (SELD). SoleSense takes advantage of two novel design choices to overcome attenuation and low signal strength. First, placing microphones on the shoe near the ground, compared to other human-centered platforms that are higher (e.g., headsets), provides natural near-ground acoustic amplification through pressure zone effect and access to ground-coupled cues. Second, SoleSense incorporates a novel transformer-based architecture called Soleformer that augments the standard attention-based global feature extractor (multi-head attention) with a convolutional neural network based local feature extractor, resulting in more salient features for detecting and localizing (weaker) sounds. Through realistic experiments across five different scenarios (environmental sounds, vehicles, speech, footsteps and multiple concurrent sound sources), we demonstrate that SoleSense can improve performance over current state-of-the-art SELD systems by up to 70%.