Microsoft has significantly enhanced the capabilities of its Copilot Vision AI, integrating it into the Copilot Windows app to assist users with a variety of content. In a recent blog post, the tech giant announced that this innovative feature allows users to engage with any item displayed on their screens, providing a new level of interactivity and support.
A big leap forward
Currently available in the United States for Windows 10 and 11, with plans to roll out to additional non-European countries, Copilot Vision serves as a virtual assistant, offering insights and assistance for files, applications, and other on-screen items. Users can request analyses or summaries of content, or pose questions about specific applications. For instance, if you find yourself puzzled by a particular software feature, the Highlights option enables you to ask Copilot for guidance, which will then direct you through the necessary steps.
Consider a scenario where you’re playing a challenging game and hit a roadblock. By consulting Copilot Vision, you can receive tips to help you advance. Alternatively, if you’re editing a photo in Adobe Photoshop Elements and wish to enhance the lighting, simply ask Copilot for advice, and it will provide detailed instructions tailored to your needs. Furthermore, the AI can facilitate connections between two applications or files simultaneously. For example, by sharing your calendar alongside a webpage listing interesting events, you can inquire about available dates for attending those events. Copilot Vision will identify suitable options and guide you on how to add them to your calendar.
How to try it
To experience this feature, users can access the Copilot Windows app on their Windows 10 or 11 devices. By clicking the eyeglasses icon adjacent to the prompt, a list of open files, applications, and windows will appear. Users can activate the switch for the desired item they wish to share with Copilot. The AI will then respond in the voice selected during setup, providing a personalized touch to the interaction.
Once engaged, users can pose questions regarding the shared content, and Copilot Vision will offer the necessary information or step-by-step guidance. If you wish to include another item in the discussion, simply click the eyeglasses icon again and activate the switch for the additional window, allowing for a more comprehensive inquiry that references both items.
My experience
In testing Copilot Vision, I explored several scenarios. For instance, while editing a photo in Photoshop Elements that featured distracting light reflections, I asked Copilot how to remove them. It promptly guided me on utilizing the Spot Healing Brush tool effectively. Additionally, I opened my calendar alongside a schedule of upcoming New York Yankees games and requested assistance in finding a date to attend a game against the Orioles. Copilot not only identified a suitable date but also offered to help me add the event to my calendar.
Privacy concerns often accompany screen-sharing technologies, but with Copilot Vision, users maintain control. The AI can only analyze content that has been explicitly shared, ensuring that your privacy is respected. As Microsoft noted in their blog post, “Copilot Vision on Windows is an all-new way to engage with your Windows PC, assisting you when needed.” This feature acts as a second set of eyes, capable of analyzing content, providing insights, and answering questions in real-time, thereby enhancing productivity and user experience.