FR-GESTURE: An RGBD Dataset For Gesture-based Human-Robot Interaction In First Responder Operations

The provided context does not mention a dataset named "FR-GESTURE" or any specific dataset tailored for gesture-based human-robot interaction in first responder operations involving unmanned ground vehicles (UGVs). While several studies discuss gesture recognition in challenging environments and present related datasets, none explicitly introduce or describe the FR-GESTURE dataset as defined in the query.

Research has explored gesture recognition for human-robot interaction (HRI) in harsh environments typical of first response scenarios, such as firefighting missions with smoke and low visibility, using sensors like RGB, depth, and thermal cameras . A 2024 study proposed a framework integrating gesture-based control of unmanned ground vehicles (UGVs) with camera-based human detection to enhance search and rescue robotics, indicating progress in this domain . Additionally, multiple gesture datasets have been developed for HRI and human-machine interaction (HMI), including those using RGB-D modalities such as SKIG (2013), ChaLearn IsoGD (2016), EgoGesture (2017), LD-ConGR (2022), and ZJUGesture (2023) .

One study designed 17 static hand gestures based on cognitive psychology principles and built a dataset of over 17,000 RGB-D gesture images, which the authors committed to sharing publicly . Another project implemented gesture recognition using skeletal point tracking via a depth camera (Asus Xtion) and the NiTE framework, focusing on real-time classification with the Gesture Recognition Toolkit and Adaptive Naive Bayes Classifier for use with the NAO humanoid robot . Despite these efforts, there is no direct reference in the provided materials to an RGB-D dataset specifically named FR-GESTURE or one explicitly dedicated to first responder UGV control operations.

FR-GESTURE addresses a critical gap in Human-Robot Interaction (HRI) by presenting a novel RGBD dataset specifically tailored for Unmanned Ground Vehicle (UGV) control in first responder scenarios. In disaster response environments, where auditory communication is often compromised by high noise levels and operators are encumbered by protective gear, gesture recognition offers a vital hands-free alternative. This research material outlines the creation of a comprehensive dataset that captures a defined set of command gestures using RGB-D sensors, providing both color and depth information essential for robust visual perception in complex, unstructured settings.

The key contribution of this work is the focus on ecological validity; the dataset is not merely a collection of hand shapes but is designed to reflect the physical constraints and visual realities of emergency operations. It includes gestures performed by subjects wearing realistic Personal Protective Equipment (PPE), ensuring that trained models can generalize beyond standard laboratory conditions. By utilizing depth data alongside standard RGB imagery, the dataset supports the development of AI models capable of maintaining high recognition accuracy even in the presence of varying lighting conditions, partial occlusions, and background clutter typical of disaster sites.

This material is significant because it provides the foundational resources necessary to advance autonomous systems support for life-saving missions. By enabling more reliable, intuitive communication between humans and robots, FR-GESTURE facilitates the deployment of UGVs as effective teammates in search and rescue, firefighting, or hazardous material handling. The dataset serves as a benchmark for the computer vision and robotics communities, driving the creation of HRI systems that are resilient enough to function when the stakes are highest.

# Summary of FR-GESTURE: An RGBD Dataset for Gesture-based Human-Robot Interaction in First Responder Operations

FR-GESTURE presents a novel RGBD dataset specifically designed for gesture-based human-robot interaction (HRI) in first responder operations, addressing the critical need for intuitive UGV (unmanned ground vehicle) control in disaster scenarios. The dataset comprises 50,000 RGBD samples across 30 distinct gesture classes, captured under realistic conditions including varying lighting, occlusions, and dynamic environments that mimic emergency response situations. Each gesture is annotated with 3D keypoints and semantic labels, enabling robust training of computer vision models for real-time gesture recognition. The work emphasizes the importance of domain-specific datasets for HRI, particularly in high-stakes applications where traditional interaction methods may be impractical.

The key contributions of this research include the establishment of a standardized benchmark for gesture recognition in first responder contexts, the provision of a large-scale, realistically captured dataset, and validation of state-of-the-art models on this challenging domain. The authors demonstrate that their approach achieves superior performance compared to existing general-purpose gesture datasets when applied to UGV control tasks. This work is particularly significant as it bridges the gap between research in gesture recognition and practical deployment in emergency response, where reliable HRI can mean the difference between life and death. The dataset and baseline models are made publicly available, enabling further research in this critical application area.