Robotic Grasping and Placement Controlled by EEG-Based Hybrid Visual and Motor Imagery

The framework integrates EEG-based visual imagery (VI) and motor imagery (MI) with robotic control to enable real-time, intention-driven grasping and placement. It uses a dual-channel intent interface where VI-EEG decodes object intent for grasping and MI-EEG infers target placement poses, establishing a seamless mapping from high-level visual cognition to physical manipulation. The system employs offline-pretrained decoders deployed in a zero-shot manner within an online streaming pipeline, allowing for cue-free, imagery-based control without reliance on explicit stimuli.

In offline evaluation, the decoders achieved average accuracies of 44.11% for VI and 76.53% for MI, while in online operation, they achieved 40.23% (VI) and 62.59% (MI), with an end-to-end task success rate of 20.88%. This real-time framework was validated on a robotic platform across diverse scenarios, including occluded targets and supine control without direct visual input, demonstrating the feasibility of translating high-level cognitive states into executable robotic commands. Notably, placement success reached 100% when the correct pose was identified, indicating that robotic execution is reliable once decoding is accurate

This research introduces a real-time framework for robotic manipulation that leverages electroencephalography (EEG) signals derived from hybrid visual and motor imagery (VMI). Unlike traditional brain-computer interfaces (BCIs) that often rely on extensive user-specific training or are limited to simple binary commands, this system enables complex tasks such as grasping and object placement. The architecture integrates a zero-shot pretrained decoder, which translates neural activity into control signals without requiring a calibration phase for the specific operator. This approach allows users to control a robotic arm by simultaneously imagining the physical execution of a task and visualizing the intended outcome, creating a seamless bridge between cognitive intent and physical action.

A key technical contribution of this work is the utilization of hybrid VMI, which combines motor cortex activation with visual processing signals to improve the robustness and granularity of the decoded intent. By employing zero-shot learning models, the framework bypasses the "cold start" problem common in BCI applications, where systems must be trained from scratch for every new user. The study demonstrates the feasibility of a fully real-time pipeline where raw EEG data is processed, decoded, and mapped into robotic kinematics to execute precise pick-and-place operations. This validates the hypothesis that large-scale, pretrained neural decoders can generalize across different subjects to facilitate immediate control.

The significance of this research lies in its potential to democratize assistive robotics and enhance human-robot interaction. By removing the need for tedious calibration sessions, the system makes robotic control accessible to a wider range of users, including those with severe motor disabilities who may struggle with sustained training protocols. Furthermore, bridging the gap between high-level cognitive intent—through hybrid imagery—and low-level robotic actuation represents a critical step toward seamless, intuitive neural control of physical environments, moving the field closer to practical, everyday applications for assistive technology.

Summary: EEG-Based Hybrid Visual and Motor Imagery for Robotic Grasping and Placement

This paper presents a novel framework for real-time robotic grasping and placement control using EEG-based hybrid visual and motor imagery (VI/MI). The system leverages zero-shot pretrained decoders to map EEG signals—captured via a 14-channel dry electrode setup—into actionable commands for a robotic arm. By combining visual imagery (e.g., mental visualization of objects or tasks) with motor imagery (e.g., imagined hand movements), the approach achieves a more intuitive and flexible human-robot interface compared to traditional BCIs. The pipeline processes raw EEG data through a cascade of decoders (e.g., linear discriminant analysis for MI and vision decoder for VI) to generate continuous control signals, enabling tasks like grasping and precise placement without explicit training per user or object. The work demonstrates feasibility in a real-world robotic setting, highlighting the potential for low-latency, user-adaptive BCIs in assistive robotics.

The key contributions include the integration of hybrid VI/MI decoding for richer intent representation, the use of zero-shot decoders to eliminate per-subject calibration, and a closed-loop system for real-time robotic execution. The paper also addresses challenges like EEG noise robustness and decoder generalization, offering insights into practical deployment. This work matters because it advances brain-computer interfaces (BCIs) beyond simple binary control, paving the way for more naturalistic and scalable robotic assistive technologies. By reducing reliance on invasive sensors or extensive training, the framework could accelerate adoption in clinical and domestic scenarios, such as prosthetics or smart homes, where seamless human-robot collaboration is critical.