Copilot on Windows 11 is currently undergoing testing for a new OS-level integration that enables users to share their favorite app screens directly with the AI assistant. This feature is being rolled out to Windows Insiders in the United States, and while it may initially raise eyebrows due to its nature of screen sharing, Microsoft assures users that it offers valuable use cases.
For instance, if a user finds themselves navigating a complex application, sharing their screen with Copilot Vision can provide real-time assistance. The AI can analyze the screen content, offer insights, and respond to queries as they arise. Imagine being in the Settings app, struggling to locate a specific toggle; Copilot Vision can guide you precisely to the right spot.
Source: BleepingComputer
While the exact methods by which Copilot Vision processes screen data remain somewhat ambiguous, it is important to note that this functionality is cloud-based, meaning Microsoft is not relying on local AI models for its operation. For those participating in the Windows Insider program, engaging with this feature is straightforward: simply click the glasses icon in the Copilot app, select the desired browser or application, and request assistance from Copilot. Users can easily cease screen sharing by clicking the Stop button or the ‘X’ in the app.
Copilot’s File Search
Another noteworthy addition is the “File Search” feature, designed to streamline file retrieval on your device. Users can interact with Copilot using natural language, asking questions like “Can you find my resume?” In response, Copilot efficiently scans the PC’s storage to locate the specified document, such as resume.docx.
According to a recent blog post from Microsoft, Copilot on Windows 11 is capable of reading various file formats, including .docx, .xlsx, .pptx, .txt, .pdf, and .json files.
Source: BleepingComputer
Currently, both the Copilot Vision (screen sharing) and File Search features are being gradually introduced within the Windows Insider Program. While Microsoft has yet to provide clarity on how data is processed or encrypted, further details regarding privacy policies are anticipated as the features move beyond the testing phase.
