Developed and user-studied an LLM-powered 'sighted guide' AI for blind/low-vision users in social VR with 16 participants, finding it effective when users were alone in virtual environments.
A study explored the use of a large language model (LLM)-powered 'sighted guide' AI to enhance accessibility in social virtual reality (VR) for blind and low vision (BLV) people, building on prior work that investigated human-guided support in VR environments. The AI guide system was designed to assist BLV users by interpreting visual scenes and responding to natural language queries in real time, leveraging GPT-4 and speech-to-text APIs to process user requests and generate spoken responses. The guide could classify user queries into categories such as holistic descriptions, specific visual questions, navigation requests, and social interactions, enabling tailored support within a virtual environment developed in Unity and running on a Meta Quest 2 headset.
The system allowed users to request environmental descriptions, ask to be guided to specific objects, or even modify the scene—such as adding audio beacons or adjusting object sizes—for improved accessibility. In testing, the AI guide demonstrated effectiveness in supporting BLV users, particularly when they were navigating virtual spaces independently. However, challenges were noted, including occasional hallucinations by the LLM and difficulties with spatial reasoning when using raw 3D coordinates, which participants found unintuitive. To improve usability, the system was enhanced with egocentric spatial references (e.g., “to your left” or “in front of you”) to better align with how users naturally perceive space.
Participants expressed diverse preferences based on their level of vision, with low-vision users benefiting more from visual enhancements like brightness adjustments, while blind users emphasized precise spatial orientation. The research highlighted the potential of conversational AI to shift accessibility control from static, developer-defined features to dynamic, user-directed modifications, allowing personalized and real-time adaptations in VR. This work aligns with broader efforts to integrate LLMs into assistive technologies for BLV individuals, such as in applications like Be My Eyes and AIRA, which connect users with remote human assistants for visual interpretation.
This research investigates the integration of Large Language Models (LLMs) as accessibility agents within social Virtual Reality (VR) environments. The authors developed a "sighted guide" system designed to assist Blind and Low Vision (BLV) users by leveraging the generative capabilities of LLMs to interpret virtual scenes and provide real-time, descriptive audio commentary. Through a user study involving 16 participants, the paper evaluates the efficacy of this AI-mediated approach in translating the inherently visual medium of VR into an accessible experience. The system functions as an autonomous agent, describing the virtual environment, identifying objects, and offering navigational cues to users who cannot rely on visual stimuli.
The study’s key contribution is the empirical evidence regarding the specific contexts in which LLM-powered accessibility succeeds and faces limitations. The results indicate that the AI guide is highly effective in aiding navigation and environmental understanding when users are alone in the virtual space. In these solitary scenarios, the system allowed participants to successfully build mental models of the environment and interact with the virtual world. However, the research highlights a critical boundary condition: the utility of the guide was predominantly confined to these solitary experiences, suggesting that the current implementation struggles to manage the complexity of social interactions or the presence of other avatars, which may overwhelm the user or the AI's focus.
This work is significant because it addresses a major barrier to inclusivity in the emerging landscape of the Metaverse and immersive social platforms. By demonstrating that LLMs can serve as scalable, dynamic interpreters of visual data, the research proposes a solution that avoids the resource-intensive need for manual, scene-by-scene audio tagging. The distinction drawn between solitary efficacy and social complexity provides a vital roadmap for future development, indicating that while generative AI can bridge the accessibility gap for exploration, future iterations must evolve to handle the multimodal nuances of social presence to ensure full inclusion for BLV users.
This paper presents the development and user evaluation of an LLM-powered "sighted guide" AI designed to enhance accessibility in social virtual reality (VR) for blind and low-vision individuals. The study involved 16 participants who interacted with the AI guide in VR environments, with findings indicating that the system was most effective when users were alone in virtual spaces. The guide leveraged large language models (LLMs) to provide real-time auditory descriptions of the environment, navigation assistance, and contextual awareness—functions traditionally performed by human sighted guides. The work builds on prior research in AI-driven accessibility tools, particularly in VR, where spatial awareness and social interaction pose unique challenges for visually impaired users.
The key contributions of this study include: 1. Validation of LLM-based guidance in VR—demonstrating that an AI can effectively replace or supplement human guides in controlled virtual environments. 2. Identification of contextual limitations—highlighting that the guide’s effectiveness diminished in highly dynamic or socially dense VR spaces, where real-time tracking of multiple agents and unpredictable interactions overwhelmed the LLM’s capabilities. 3. User feedback on trust and usability—participants reported that the AI improved independence but also expressed concerns about reliability in complex scenarios, suggesting that hybrid human-AI systems may be necessary for full accessibility.
This research matters because it advances AI-driven accessibility in emerging technologies like social VR, where visually impaired users often face barriers to participation. While the study confirms the potential of LLM-powered guides, it also underscores the need for context-aware, adaptive AI systems that can handle the unpredictability of multi-user virtual environments. The findings could inform future developments in assistive VR tools, particularly as social platforms like VRChat, Horizon Worlds, and Meta’s Quest ecosystem expand their user bases.
Source: [arXiv:2603.09964](https://arxiv.org/abs/2603.09964)