Multimodal 3D Medical Image Fusion in Augmented Reality

A patient’s imaging record is fragmented across modalities and viewers. CT and MRI arrive as DICOM volumes, whereas pathology is captured as gigapixel whole-slide images (WSI), each in its own coordinate system and scale. This paper presents a post-processing pipeline and augmented reality (AR) viewer that fuse separate CT, MRI, and WSI samples into a single common virtual space, so that effectively all a patient’s imaging data can be co-located and inspected together, simultaneously, within one interactive 3D space. Each modality is placed in a shared physical coordinate system through an externally supplied per-modality affine transform rather than by landmark registration. A central difficulty is the extreme resolution difference between millimeter-scale radiology and micrometer-scale histology. We bridge it without resampling the modalities onto a common grid, preserving each one’s native resolution, and we store the gigapixel volume as a sharded multiresolution pyramid whose native level is kept lossless for diagnostic use while the coarser overview levels are lossy, keeping the volume streamable without compromising the diagnostic layer. To render this span stably, we adapt the floating-origin principle from large-world rendering to the cross-scale medical setting as a focus-relative local coordinate frame that decouples the global scale span from GPU single-precision arithmetic and eliminates the positional jitter it would otherwise produce; we characterize this effect quantitatively, against a naive global frame, under progressive magnification. A single-pass GPU ray-marching traversal composites all modalities with independent per-modality controls (clip plane, opacity, intensity window), and a focus-and-context streaming strategy bounds on-device GPU memory for the visualization device. We evaluate conversion efficiency and fidelity, precision under extreme zoom, and native-resolution streaming performance on the headset against a fixed on-device memory budget.