Ablation Study of Multimodal Perception, Language Grounding, and Control for Human-Robot Interaction in an Object Detection and Grasping Task

arXiv cs.RO / 5/5/2026

📰 NewsIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper presents a controlled ablation study of a multimodal human-robot interaction system, focusing on three key modules: an LLM for action extraction, a perception module for visual grounding, and a motion controller for execution.
Rather than redesigning the entire pipeline, it isolates each component’s contribution using a consistent experimental protocol and then evaluates strong end-to-end combinations.
The study compares three different language models, five perception configurations, and three controllers, followed by a second-stage factorial experiment over the best-performing candidates.
The analysis aims to determine which design choices most affect execution time versus task success rate and to identify where future engineering improvements are likely to yield the biggest gains.

Abstract

This manuscript extends our previous multimodal human-robot interaction system by introducing a controlled ablation study of the three modules that most strongly influence end-to-end performance: the large language model used for action extraction, the perception system used for visual grounding, and the controller used for motion execution. The goal is not to redesign the full pipeline, but to isolate the contribution of each component under a common experimental protocol and then evaluate the best combinations end-to-end. We therefore compare three language models, five perception configurations, and three controllers, followed by a second-stage factorial study over the best candidates. The resulting analysis is intended to clarify which choices primarily affect execution time, which primarily affect success rate, and where the largest engineering gains are likely to come from in future revisions of the system.