Enabling Large Model Inference in Dynamic and Heterogeneous Mobile Robots by Incorporating Collaborative Inference and Model Context Protocol

Modern robotic systems increasingly rely on transformer-based AI models to process sensor input and support high-level decision making. However, due to limited GPU resources in each ego robot, it is difficult to run a large model locally. Outsourcing the model inference to cloud providers could address this issue, which however suffers from limitations including unreliable network connection, high latency, and high monetary cost. In this work, we have developed an effective decision making framework for robotic systems by leveraging a few state-of-the-art technologies, such as collaborative inference and model context protocol (MCP). Our framework seamlessly integrates collaborative inference between edge servers and mobile robots, supporting inference of large models by relying on local resources instead of remote cloud-based resources. It excels in a few feature designs. First, it is designed for mobile robot deployments where the available computing resources may change over time. This is achieved by introducing “anchors”, adding redundant nodes to hold the model partitions, and switching among model variants of different sizes. Second, it can allow heterogeneous robotic peers to seamlessly join the system through MCP, a standardized interface for modern AI applications. We have implemented a prototype of the proposed framework and experimentally validated its efficiency.