Human Tracking by a Mobile Robot Using Machine Learning of Clothing Features and Body Thermal Images

This study proposes a human tracking algorithm that combines vision-based object detection and body heat tracking. The small autonomous mobile robot developed in this study is equipped with an ultracompact camera and three thermal array sensors. We propose a hybrid human tracking method based on camera images and body heat information. In particular, the proposed machine-learning-based image recognition method employs Faster Objects, More Objects, an object detection algorithm optimized for resource-constrained edge devices such as microcontrollers. This algorithm generates an inference model focusing on the clothing features of the target person, which is implemented in the robot control system. Furthermore, a human-robot distance estimation model is derived to calculate the tracking distance between the target person and the robot using a monocular camera. Experiments demonstrated that the proposed inference model enables robust tracking even when the target is partially occluded by another person. Additionally, the body-heat-distribution-based tracking method employs the summation of multiple thermal pixels to improve tracking performance. In this study, tracking performance was improved by emphasizing the target region in the thermal image using four-pixel, nine-pixel, and one-column pixel summation methods. The experimental results demonstrate that the four-pixel summation method, which minimizes fluctuations in the robot posture angle during motion, is the most suitable for the proposed tracking algorithm. The proposed algorithm is effective for hybrid control in environments where illumination conditions change abruptly between bright and dark areas.