DirectML Archives

Winsage

December 14, 2024

Rising to the TOPS: How will NPUs and Windows AI grow in 2025?

In 2024, Microsoft introduced the "Copilot+ PC" branding for AI-capable laptops, while Apple launched Apple Intelligence. These developments have led to mixed outcomes, with features like real-time translations and on-device speech-to-text being beneficial, but others, such as Windows Recall, still proving their value. By 2025, mainstream developers are expected to integrate on-device AI into Windows applications, influencing consumer purchasing decisions. The term "TOPS" (Trillions of Operations Per Second) is becoming important for evaluating the AI performance of Windows laptops, with a minimum of 40 TOPS required for Microsoft's "Copilot PC+" designation. Qualcomm's Copilot+ PCs reported around 45 TOPS, significantly higher than Intel's 11 TOPS. By the end of 2024, premium Windows laptops are expected to see a three- to four-fold increase in NPU performance compared to 2023 models. Analysts speculate further performance improvements may occur towards the end of 2025. Despite the potential for a two- to three-fold enhancement in on-device AI performance, experts caution against overemphasizing TOPS figures, which may not accurately reflect real-world performance. The lack of a unified API for leveraging NPU capabilities in Windows complicates matters for users of Copilot+ laptops without Qualcomm chips. Although AMD and Intel have released competitive chips, Qualcomm currently holds an advantage with exclusive support for certain applications. Microsoft is promoting its low-level machine learning API (DirectML) and the Windows Copilot Runtime, which may enhance the Copilot+ PC ecosystem. While cloud-based AI solutions remain an option, the cost of these services is expected to rise, making on-device AI more appealing. The introduction of ChatGPT Pro highlights the financial implications of cloud access compared to on-device NPU usage, which incurs no additional costs. The pace of on-device AI adoption in Windows' software ecosystem is anticipated to accelerate in 2025.

Winsage

November 19, 2024

Microsoft and NVIDIA Supercharge AI Development on RTX AI PCs

Generative AI-powered laptops and PCs are advancing sectors like gaming, content creation, productivity, and software development, with over 600 Windows applications and games utilizing AI on more than 100 million GeForce RTX AI PCs globally. At the Microsoft Ignite event, NVIDIA and Microsoft introduced tools for Windows developers to build and optimize AI applications on RTX AI PCs, enhancing workflows for AI agents and digital humans. NVIDIA's interactive digital human, James, utilizes NVIDIA NIM microservices, NVIDIA ACE, and ElevenLabs technologies for immersive interactions. NVIDIA ACE enhances digital entities' engagement through visual perception, allowing context-aware responses. The multimodal small language models developed by NVIDIA process text and imagery, optimizing rapid response times. The upcoming NVIDIA Nemovision-4B-Instruct model operates on RTX GPUs while maintaining accuracy, enabling digital humans to interpret visual imagery for relevant responses. NVIDIA will also launch the Mistral NeMo Minitron 128k Instruct family, offering large-context small language models in various parameter versions for efficient digital human interactions. These models process extensive datasets without segmentation, improving efficiency on low-power devices. NVIDIA announced enhancements to the TensorRT Model Optimizer for Windows, addressing challenges in model deployment due to limited memory and computational resources. The updates streamline models for ONNX Runtime deployment, utilizing GPU execution providers. The TensorRT-ModelOpt includes advanced quantization algorithms, significantly reducing memory footprint and improving throughput performance on RTX GPUs, achieving up to a 2.6x reduction in memory footprint compared to FP16 models.

Winsage

November 13, 2024

DirectML unlocks new silicon for AI experiences across Windows Copilot+ PCs

October saw the introduction of various applications utilizing the Neural Processing Unit (NPU) on Copilot+ PCs, enhancing AI innovations on the Windows platform. Adobe Premiere Pro became the first Adobe application to leverage NPU capabilities, integrating features like the Audio Category Tagger, which automatically tags audio clips. Capture One announced two AI-powered features, Match Look and AI Crop, for Copilot+ PCs, utilizing Qualcomm's NPU. Affinity Photo 2 introduced AI-enhanced Object Selection capabilities, automating the creation of layer masks using machine learning with Qualcomm's Hexagon NPU. DirectML facilitates compatibility across hardware architectures, supporting these advancements in AI applications on Windows.