Highlights that most passenger-facing VR studies for autonomous vehicles rely on fully simulated or pre-scripted scenarios rather than motion and perception from physically operating vehicles.
Most passenger-facing VR studies for autonomous vehicles rely on fully simulated vehicles or pre-scripted scenarios, meaning the displayed motion and perception do not originate from a physically operating autonomous system.
This limitation is highlighted in research presenting a video-augmented VR framework that couples a physical ROS 2 autonomous robot vehicle to a Meta Quest 3S headset. Unlike traditional simulations, this approach transmits live onboard camera streams and real-time vehicle state data (position, heading, speed, decisions) over independent UDP channels to ensure the virtual environment mirrors the physical robot's actual behavior and navigation decisions.
This material addresses a key realism gap in passenger-facing virtual reality for autonomous vehicles: many prior VR studies use fully simulated, synthetic, or pre-scripted driving scenarios, which can feel detached from the embodied experience of an actual vehicle in motion. The proposed direction instead grounds the VR experience in the perception and motion of a physically operating autonomous vehicle, using video-augmented VR to combine real vehicle-view video with virtual overlays or augmented scene elements. In doing so, it shifts the design goal from “showing passengers a plausible virtual world” to “showing passengers what the vehicle itself sees,” preserving the temporal, spatial, and perceptual cues that arise from real driving.
The central contribution is an argument and likely system-level approach for making VR a faithful interface to the vehicle’s embodied perception rather than a separate simulated environment. By leveraging live or recorded camera imagery, vehicle motion, and possibly sensor-derived state, the work suggests that VR can be made more ecologically valid, temporally coherent, and aligned with the vehicle’s actual trajectory and observations. This is important because passenger trust, comfort, and situational understanding depend heavily on whether the presented world matches the physical experience; poorly synchronized or unrealistic VR can undermine confidence in the vehicle, while well-grounded video-augmented VR may improve transparency, explainability, and acceptance of autonomous driving.