Windows 11 is poised to unveil its third feature update, a significant leap forward that integrates a variety of innovative AI functionalities. This update, designated as version 24H2 (2024 Update), introduces the Windows Copilot Runtime, a robust framework that enables a multitude of AI features to operate seamlessly on the device itself. Among the new capabilities are Windows Recall, Live Captions, Auto Super Resolution, Voice Clarity, Studio Effects, and several others designed for built-in applications.
<h2 class="article-bodysection” id=”section-what-ai-changes-are-coming-to-windows-11-2024-update”>What AI changes are coming to Windows 11 2024 Update
This feature update from Microsoft promises nearly a dozen AI-driven enhancements, most of which function without requiring an internet connection.
Windows Copilot Runtime
At the core of this AI evolution is the Windows Copilot Runtime, a sophisticated layer within the operating system that integrates over 40 distinct AI models capable of executing tasks locally. Notable models include Phi Silica, a Small Language Model (SLM) that encapsulates many advantages of larger language models, alongside tools like the Screen Region Detector, Optical Character Recognizer, Natural Language Parser, and Image Encoder.
Additionally, Microsoft is rolling out the Windows Copilot Library, a collection of APIs that facilitates the integration of AI capabilities into applications, ensuring these features are always ready for use without consuming substantial resources.
While some features, such as Copilot, rely on cloud processing, the Windows Copilot Runtime empowers many functionalities to operate independently, safeguarding user data from being transmitted online.
Windows Recall
Among the most anticipated and debated features is Windows Recall, which continuously captures screenshots of user activity, leveraging multiple on-device AI models aided by a Neural Processing Unit (NPU) to analyze and store content in the new Windows Semantic Index. Users can access the “Recall” app to search for information using natural language or a timeline slider, allowing for easy interaction with past content.
The feature requires adequate storage, with a default allocation of 25GB for devices with a 256GB hard drive, adjustable based on total storage capacity. A new “Recall & snapshots” page in the Settings app will allow users to manage the feature’s settings, including storage and filters to exclude specific websites and applications.
Importantly, Windows Recall respects user privacy by not tracking activity in InPrivate or Incognito modes on Chromium-based browsers. Despite its potential, the feature has raised concerns regarding security and privacy, prompting Microsoft to enhance its safeguards, including data encryption and authentication requirements for accessing Recall data.
Windows Studio Effects
Windows Studio Effects is another exciting addition, utilizing the NPU to apply various enhancements to webcam and microphone inputs. This feature, now broadly available in version 24H2, includes Background Blur, Eye Contact, Auto Framing, Voice Focus, Portrait Light, and Creative Filters, all accessible through the Quick Settings flyout.
Users can manage these effects via the Settings menu, tailoring their video call experience to enhance visibility and engagement.
Live Captions
Live Captions, a feature that has been part of Windows 11, is now enhanced with AI capabilities, including live translations. This accessibility tool aims to facilitate understanding of audio content by providing real-time transcription and translation from various audio sources, such as YouTube videos or voice calls.
Voice Clarity
Another noteworthy addition is Voice Clarity, which employs AI to eliminate background noise during conversations or recordings. This feature operates automatically, ensuring a clear audio experience without requiring user intervention. Originally exclusive to Surface devices, it will now be available across a broader range of hardware.
Auto Super Resolution
Auto Super Resolution (Auto SR) is set to enhance gaming experiences by using on-device AI to upscale graphics, improving frame rates and image quality without necessitating developer updates. This feature functions similarly to existing technologies like NVIDIA DLSS but operates independently, processing upscale tasks through the NPU for improved energy efficiency.
Users can enable Auto SR through the Graphics settings, with notifications indicating compatible games and options to activate the feature.
Restyle Image for Photos
The Photos app will now feature a “Restyle Image” tool, utilizing on-device AI to transform photos with various styles akin to mobile filters. Users can input text prompts to guide the AI in altering backgrounds and other elements of their images.
Cocreator for Paint
In the Paint app, a new feature called “Cocreator” will assist users in enhancing their artwork by scanning drawings and offering creative suggestions. This feature allows for a more intuitive artistic process, utilizing a diffusion-based algorithm to generate high-quality images with minimal effort.
Copilot app
The 2024 Update will also see a redesign of the Copilot AI integration, transitioning from a side panel to a dedicated app. This new interface mirrors the ChatGPT design, featuring a left-side panel for prompt history and options for initiating new conversations and utilizing plugins.
Copilot suggestions
While the Copilot app will no longer manage system settings, it will introduce “Copilot suggestions,” which will appear contextually within applications to offer helpful prompts and actions. For instance, users may receive suggestions for summarizing document attachments in emails or modifying images in File Explorer.
Important: The Windows 11 2024 Update began partial availability on June 18, 2024, for Copilot+ PCs, though not all features, including Windows Recall, will be immediately accessible. A broader rollout is expected in September or October, with some features potentially requiring new hardware.
More resources
For additional insights, articles, and answers to common queries regarding Windows 10 and Windows 11, please explore the following resources:
