In 2024, the landscape of on-device AI within consumer electronics has witnessed significant developments, particularly with major players like Microsoft and Apple making strides in their respective operating systems. Microsoft’s introduction of the “Copilot+ PC” branding for AI-capable laptops and Apple’s launch of Apple Intelligence have set the stage for a new era in computing.
These initial offerings have produced mixed outcomes. While features such as real-time translations and on-device speech-to-text have proven beneficial, others, like Microsoft’s Windows Recall, are still in the process of demonstrating their value.
As we look ahead to 2025, the anticipation grows for mainstream developers to integrate on-device AI into their Windows applications. This shift will compel consumers to pay closer attention to the AI capabilities of modern Windows laptops when considering new purchases.
Big gains are coming for NPUs
For those interested in the AI performance of Windows laptops, the term “TOPS” (Trillions of Operations Per Second) will likely become a focal point. This metric measures an NPU’s capacity to execute matrix multiplications for on-device AI tasks, making it a crucial factor in evaluating laptop performance.
The year 2024 has already seen substantial advancements in TOPS performance among Windows laptops. To qualify for Microsoft’s “Copilot PC+” designation, a laptop must achieve a minimum of 40 TOPS. For context, Qualcomm’s inaugural Copilot+ PCs reported around 45 TOPS, a remarkable leap from Intel’s “Meteor Lake” Core Ultra 7 165H, which offered only 11 TOPS.
Karl Freund, founder and principal analyst at Cambrian AI Research, noted that Qualcomm’s innovations have prompted swift responses from competitors like AMD and Intel, who are now releasing chips that deliver similar performance enhancements.
By the end of 2024, consumers seeking premium Windows laptops—such as the Microsoft Surface, Asus ProArt, or Dell XPS—can expect a three- to four-fold increase in NPU performance compared to their 2023 counterparts. This trend raises the question: will this momentum continue into 2025?
Ryan Shrout, president of performance testing lab Signal65, believes it could. He speculates that we might see further doubling or even tripling of performance metrics, although he anticipates that the most significant improvements will materialize towards the latter part of 2025, possibly extending into 2026.
TOPS may not stay on top for long
While a potential two- to three-fold enhancement in on-device AI performance is noteworthy, both Freund and Shrout caution against placing too much emphasis on the TOPS figures provided by chip manufacturers.
Freund humorously dubbed TOPS as “Terribly Overused Performance Stat,” suggesting it lacks substantial value. Shrout echoed this sentiment, likening TOPS to the TFLOPS metrics often touted by AMD and Nvidia for GPUs—figures that, while impressive, do not always translate to real-world performance.
The current state of real-world AI performance remains somewhat unpredictable, primarily due to Windows’ lack of a unified API for leveraging an NPU’s capabilities. This inconsistency poses challenges for users of Copilot+ laptops that do not feature Qualcomm chips.
Although AMD and Intel have introduced chips that meet the Copilot+ criteria, Qualcomm has enjoyed a competitive edge thus far. Their machines were the first to support Windows Recall and several popular applications, including Blender and Affinity Photo, which are now exclusive to Qualcomm Snapdragon X hardware.
However, as 2025 unfolds, this landscape is expected to shift. Microsoft is actively promoting its low-level machine learning API (DirectML) and the Windows Copilot Runtime, which encompasses various task-specific AI APIs, some of which are yet to be released. The potential for growth in Copilot+ PCs is evident, and experts believe that Microsoft will address these challenges in 2025.
Proving the case for Windows AI
Enhancements in NPU technology and a cohesive API that simplifies the utilization of an NPU’s full capabilities are crucial advancements. However, these developments alone do not guarantee that on-device AI will become a standard feature.
Developers still have the option to leverage cloud-based AI solutions from companies like OpenAI and Anthropic, which offer more powerful models accessible via the internet. These cloud-hosted models can outperform on-device AI in terms of capabilities and speed.
Yet, as we approach 2025, the cost implications of cloud-based AI services will become increasingly significant. Shrout highlights the advantage of small language models running continuously on an NPU, a feat that is either impossible or prohibitively expensive in the cloud.
The recent introduction of ChatGPT Pro by OpenAI, which offers premium access to advanced AI models at a steep monthly fee, underscores this point. In contrast, users and developers who utilize a Windows laptop’s on-device NPU can do so without incurring additional costs, potentially paving the way for a broader adoption of on-device AI.
As the industry progresses, the pace at which on-device AI gains traction within Windows’ software ecosystem remains to be seen. However, 2025 is poised to be a pivotal year in this transformation.
