Attention Based Optimization for 3D Shape Registration

Abstract

Transformers are sequence‐to‐sequence architectures originally designed to handle structurally rigid and order‐sensitive data, such as text and images. At their core, they exploit the attention mechanism, which is permutation‐equivariant and relies on computing token‐to‐token relationships. These models have been applied to 3D geometry in several instances, achieving discrete success across tasks such as shape generation, segmentation, classification, shape matching, and registration. While existing 3D geometry methods use transformers as traditional learners, we present an approach that reinterprets the transformer as an optimization pipeline for shape correspondence. By fitting the model directly to a shape pair, our method eliminates the need for large training datasets, providing a category‐agnostic solution. In particular, we focus on the use of attention weights, tailored to encode token‐to‐token information, to inject and extract point‐to‐point information during the processing of one or more meshes. We demonstrate, that self‐ and cross‐attention mechanisms can, by design, serve as feature extractors and matching solvers, respectively. Furthermore, instead of deriving correspondence from the final output of the network, we exploit the cross‐attention weights directly as the permutation matrix. This framework not only achieves robust shape matching and registration but also provides a theoretically grounded, interpretable approach to attention for unstructured 3D data. Notably, our work represents an approach that leverages the transformer architecture as an end‐to‐end pipeline for shape correspondence, operating effectively without requiring additional training data.