SeededGrasp: Language-Guided Grasping in Complex Scenes with Multiple Embodiments

Abstract

Practical robotic grasping in complex scenes requires both 3D spatial reasoning and alignment with task-specific requirements. Vision-language models (VLMs) offer a natural way to specify these requirements using language, but existing approaches either use a VLM to predict the grasp directly with limited spatial awareness, or train the VLM together with the grasping model, which requires significantly more data and compute. These limitations impede performance and have prevented scaling to multiple embodiments in complex scenes. We address this by proposing SeededGrasp, a novel data-efficient framework that enables a VLM to predict a seed point to be used as conditioning for a subsequent lightweight grasp-generation model. Our architecture decouples high-level semantic reasoning from low-level geometric execution, enabling multi-embodiment support while bypassing the need for expensive end-to-end training. To enable training such models, we release the first multi-embodiment tabletop grasping dataset comprising over 2.5M grasps in cluttered scenes. Experimental results demonstrate that our approach outperforms existing baselines, achieving 72% success in simulation and 78% in real-world grasping experiments.

Dataset

To the extent of our knowledge, we are releasing the first public multi-embodiment dataset for grasping in cluttered tabletop scenes.

Key Statistics:

• Size: 2.56 million in-scene grasp poses.
• Environments: 610 synthetic tabletop scenes with random object clutter (504 train, 106 eval).
• Objects: 334 unique objects sourced from the GSO and YCB datasets (284 train, 50 unseen eval).

Poses by Embodiment:

Method

Method Overview: Our method allows users to guide robot grasping in cluttered environments using simple language instructions. Starting with an RGB-D image of the scene, we use a pre-trained Vision-Language Model (VLM) to translate the user’s text into a 2D target point on the image. We then project this point into a 3D point cloud and pass it through a lightweight flow-matching model, which generates a stable, 6DOF grasp pose for the robot gripper. Because it relies on a pre-trained VLM, this approach provides intuitive, language-conditioned grasping without requiring complex language-annotated datasets or extra training time.

Results

We evaluate our method in a real-world setup using a Delto DG-3F-B gripper by mapping Robotiq 3-Finger gripper-conditioned predictions, achieving a 78% success rate.

Acknowledgements

This research was supported in part by the Vector Institute and Digital Research Alliance of Canada.