CLASP: General-purpose clothes manipulation with semantic keypoints

Clothes manipulation, such as folding or hanging, is a critical capability for home service robots. Despite recent advances, most existing methods remain limited to specific clothes types and tasks due to the complex, high-dimensional geometry of clothes. This paper presents CLothes mAnipulation with Semantic keyPoints (CLASP). CLASP aims at general-purpose clothes manipulation across diverse clothes types, such as T-shirts, shorts, skirts, long dresses, … and across different tasks, such as folding, flattening, hanging, …. The key idea of CLASP is semantic keypoints —for example, “left sleeve” and “right shoulder”—a sparse spatial-semantic representation, salient for both perception and action. Semantic keypoints of clothes can be reliably extracted from RGB-D images and provide an effective representation for a wide range of clothes manipulation policies. CLASP uses semantic keypoints as an intermediate representation to connect high-level task planning and low-level action execution. At the high level, it exploits a vision-language model (VLM) to predict task plans over the semantic keypoints. At the low level, it executes the plans with the help of a set of pre-built manipulation skills conditioned on the keypoints. Extensive simulation experiments show that CLASP outperforms state-of-the-art baseline methods on multiple tasks across diverse clothes types, demonstrating strong performance and generalization. Further experiments with a Franka dual-arm system on four distinct tasks—folding, flattening, hanging, and placing—confirm CLASP’s performance on real-life clothes manipulation.