On October 1, Google unveiled its innovative Guided Vision feature within the Gemini Live platform, designed to enhance the experience of users with visual impairments. This mode provides real-time descriptions of what the camera is focused on, offering verbal guidance on how to adjust the phone for optimal viewing. Google emphasizes that Guided Vision serves as a “voice-forward, real-time visual interpretation” tool, specifically aimed at assisting individuals who are blind or have low vision, as well as those seeking intuitive visual support. However, it is crucial to note that this feature is not intended to replace medical devices, mobility aids, or traditional white canes.
Gemini tells you how to move the camera
The standout aspect of Guided Vision is its coaching capability. Users will receive natural verbal prompts to help them adjust their camera positioning. For instance, if the camera is aimed too high or too close, Gemini will suggest actions like panning slowly to the right or tilting downward to capture clearer visual context. This enhancement builds upon existing camera-sharing functionalities in Gemini Live, adding a layer of spoken guidance and follow-up questions.
Google envisions various practical applications for Guided Vision, including:
- Reading small print on nutrition labels or appliance dials.
- Identifying items like a dropped earbud or spices in a crowded cabinet.
- Choosing outfits by asking about colors, patterns, and shapes.
- Understanding the layout of unfamiliar rooms.
Moreover, the assistance extends beyond its primary audience, benefiting older adults, individuals with low literacy, and anyone struggling to read fine print in dim lighting.
Aira trained it, with more than 1,000 testers
To develop this feature, Google collaborated with Aira, a visual interpreting service, to train the Gemini Live model through extensive real-world data interpretation. The partnership involved tens of thousands of hours of visual interpretation, with over 1,000 members from Aira’s Trusted Tester network participating in rigorous testing and refinement processes. Aira specialists contributed their expertise to establish safety protocols and evaluate the model’s performance.
The development process included feedback from blind and low-vision communities across various countries, including India, Brazil, Singapore, Indonesia, and Japan, ensuring that Guided Vision is capable of handling conversations in multiple languages.
Three ways to start it on Android
Users can activate Guided Vision on their Android devices through three straightforward methods:
- Via the Gemini app: Navigate to profile settings, enable “Use Guided Vision in Live,” then open Gemini Live and share the camera.
- Using an Android accessibility shortcut: Go to Settings, then Accessibility, followed by Vision assistance, and select Guided Vision. This can be accessed through a floating accessibility button, a two-finger swipe, or by pressing both volume keys.
- For TalkBack users: Open the TalkBack menu with a three-finger tap and select Guided Vision from the options.
Guided Vision is compatible with Android devices running version 9 and above, available in regions and languages where Gemini Live is supported. Users are advised to keep both the Gemini app and system software updated to access this feature. Initially previewed in the September Android Drop, Guided Vision is positioned as an assistive utility that may have limitations, underscoring the importance of traditional navigation aids for safe travel.
