Windows 11 will soon be able to describe images on your screen using AI — and it’ll all be done locally

Microsoft is set to enhance the user experience on Windows 11 with the introduction of a new AI action within its Click To Do feature. This innovative addition will empower users to receive descriptions of images displayed on their screens, all thanks to advanced on-device AI capabilities. Exclusively available for Copilot+ PCs, users can easily access this feature by holding down the Windows key and clicking the mouse.

On-Device AI for Enhanced Privacy

The describe image function leverages sophisticated on-device AI models, ensuring that users can generate descriptions without the need for an internet connection. This offline capability not only enhances accessibility but also prioritizes user privacy. As noted in a Microsoft blog post, “When you use the action for the first time, the required models are set up, and the descriptions are generated locally on your device making sure your sensitive data stays on your PC.”

Windows Recall utilizes these on-device AI models to analyze screen content, offering tailored tasks and actions based on what it identifies. The feature’s versatility allows it to function seamlessly across various applications within the Windows environment.

The describe image popup will work in any app on Windows.
(Image credit: Microsoft)

Currently, Click To Do is capable of a variety of functions, including text analysis, list creation, and rewriting tasks. It can also identify images, offering quick actions such as blurring or removing backgrounds, and even searching the web for more information about the displayed image. Users will appreciate the customizable nature of these actions, allowing them to tailor the Click To Do menu to their preferences by disabling features they find unnecessary.

The introduction of the describe image feature positions Click To Do as a formidable competitor to Google’s Circle To Search, which has garnered popularity among Android users. In many respects, Click To Do surpasses its counterpart, offering a more advanced suite of actions powered by local AI models.

Another significant advantage of this on-device AI approach is the enhanced security it provides. Microsoft emphasizes that the describe image feature operates securely without relying on cloud services, ensuring that images remain private and are processed locally through the onboard AI model and the NPU within Copilot+ PCs.

Currently, the describe image feature is accessible through the Windows 11 Insider Program, specifically for users in the Beta and Dev Channels. A broader rollout is anticipated later this year, with the preview currently limited to Copilot+ PCs equipped with Snapdragon processors. Intel and AMD chip users can expect to gain access to this exciting feature in the coming weeks.

Winsage
Windows 11 will soon be able to describe images on your screen using AI — and it'll all be done locally