3D Scene Reconstruction and Immersive VR Environment Generation Using Smartphone-Based Panoramic RGB-D Data

This paper proposes Sensor-Initialized Gaussian Splatting (SIGS), a method that uses panoramic depth data acquired by the LiDAR sensor in a smartphone for three-dimensional (3D) scene reconstruction and VR space generation. Traditionally, 3D scene generation has required specialized knowledge and significant time, posing challenges for its application in VR. In particular, point clouds estimated by Structure from Motion (SfM), which are used for initializing 3D Gaussian Splatting (3DGS), have had limitations in density and accuracy. SIGS addresses these challenges by utilizing high-accuracy point clouds directly acquired from an iPhone’s LiDAR sensor for 3DGS initialization. For this research, a dedicated smartphone application called Panoramic Depth Recorder (PDR) was developed to simultaneously capture RGB images, depth images, point cloud data, and camera position and rotation information while the iPhone is rotated. These point clouds are then integrated into a common world coordinate system. For outdoor scenes, geometric information over a wider range is supplemented by combining near-field LiDAR data with far-field point clouds estimated by SfM. Experiments demonstrated that SIGS improved rendering accuracy (Structural Similarity Index Measure and Learned Perceptual Image Patch Similarity) and visual quality compared to conventional methods initialized solely with SfM. The generated scenes exhibited fewer artifacts and reproduced more faithful geometric shapes, with improved floor surfaces and ceiling irregularities. The mesh data of the generated 3D scenes is designed for use in VR environments. After conversion from PLY to the FBX format using Blender, they can be imported into Unity, enabling collision detection and user movement control within the VR space. This opens up possibilities for applications such as creating digital twins of robot training environments to reproduce real-world spaces in VR applications.