Most software applications provide a clear indication of their functionality. Video editors enhance your footage, games respond to your inputs, and chat applications keep you connected with friends. However, when it comes to antivirus utilities or security suites, the assurance of their effectiveness is often less tangible. Without a notification alerting you to thwarted attacks, it can be challenging to gauge their performance. This is where PCMag steps in, rigorously evaluating these products through a series of comprehensive tests that validate their claims.
Each review not only presents the results of these evaluations but also shares firsthand experiences with the products. The testing process is tailored to the specific features of each antivirus solution. For instance, while many utilities offer phishing protection, not all do. Some suites come equipped with parental controls, while others do not. Regardless of the features included, every product undergoes thorough scrutiny.
We Test Antivirus Protection Using Actual Malware
At the core of every robust antivirus tool are two essential features: an on-demand scanner that identifies and removes existing malware, and a real-time monitor that blocks new threats. To verify the efficacy of these protective measures, we employ real-world malware samples. Utilizing virtual machines for this testing ensures that any missed infections do not spread.
We Collect Real-World Malware
Each spring, following the annual update cycles of most security vendors, we compile a fresh batch of malware samples for testing. This process begins with a feed of the latest malware-hosting URLs, from which we download thousands of samples and refine them to a manageable quantity. Each sample is meticulously analyzed using a suite of hand-coded tools. Samples that detect their operation within a virtual machine and refrain from malicious activity are excluded from our tests. Our focus is on a diverse range of malware types, particularly those that alter the file system and the Registry.
During this selection process, we cross-reference each sample’s identifying hash value with a database maintained by Google subsidiary VirusTotal. This database scans files using approximately 70 antivirus engines, reporting which ones identify a file as malicious. Samples that receive fewer than half of the engines’ classifications as malicious are discarded. Once we have a refined collection, we meticulously document the system changes made by each sample.
We Challenge Each Antivirus to Defend Against Malware
To evaluate a product’s malware-blocking capabilities, we initiate the opening of our samples. In some instances, real-time protection activates immediately, eliminating known malware. If necessary, we employ various methods—such as single-clicking each sample or transferring the collection to a new folder—to trigger the real-time protection. We meticulously record which samples the antivirus eliminates upon detection.
Following this, we execute each surviving sample to observe how the antivirus responds. We document the total percentage detected, regardless of when the detection occurs. While detection is crucial, it is equally important for the antivirus to actively prevent the attack. A small, hand-coded program checks the system for any Registry changes or newly installed files. For executable files, we also verify whether any processes are running. Once the measurements are complete, we shut down the virtual machine. Products that successfully prevent the installation of all executable traces from a malware sample earn high scores, while those that fail to do so receive lower ratings. The average of these scores culminates in the product’s final malware protection score.
We Evaluate Web-Level Malware Protection
The most effective way to thwart a malware attack is to intercept it before it reaches your computer. Many antivirus products integrate with browsers to redirect users away from known malware-hosting sites. Some operate below the browser level, ensuring protection without requiring additional extensions. If the initial protection fails, there remains an opportunity to eliminate the malware payload during or immediately after download.
Our basic malware protection test utilizes the same set of samples for several months, while the URLs used for testing web-based protection are updated regularly. These links are sourced from a feed of the latest malicious URLs identified by MRG-Effitas, typically no more than a few days old. Using a specialized utility, we sequentially launch each URL, discarding those that do not lead to a malware download or return error messages. For the remaining URLs, we note whether the antivirus prevents access, eliminates the download, or remains passive. This process continues until we gather data from 100 verified malware-hosting URLs. The score for this test reflects the percentage of URLs from which the antivirus successfully prevented malware downloads, whether by blocking access or deleting the downloaded file. Many tested antivirus tools achieve scores of 90% or higher.
We Test Using Actual Phishing Sites
Creating complex data-stealing Trojans is unnecessary for cybercriminals when they can simply deceive individuals into revealing their credentials. Phishing websites, which mimic legitimate banks and other sensitive platforms, pose a significant threat, as they can operate across any operating system that supports web browsing. These fraudulent sites are typically blacklisted shortly after their creation, so we utilize only the most recent phishing URLs for testing. Our goal is to maintain an equal distribution between verified phishing pages and those reported as fraudulent but not yet confirmed.
For this test, we deploy four virtual machines: one protected by the antivirus under evaluation and one each using the phishing protection built into Chrome, Edge, and Firefox. A small utility program launches each URL across the four browsers. Any URL that returns an error message or does not actively imitate another site or attempt to capture user credentials is discarded. We meticulously document whether each product successfully detects the fraudulent page. Unlike the web-level protection test, antiphishing scores can vary significantly, with some products achieving perfect detection rates while others fall short compared to built-in browser protections.
We’ve Simplified Antispam Testing
In today’s digital landscape, most consumers find their email accounts are effectively shielded from spam by their email providers or server-side utilities. Consequently, the average consumer’s need for spam filtering has diminished. Previous tests conducted by AV-Comparatives in 2016 revealed that even Microsoft Outlook alone blocked nearly 90% of spam, with many suites performing even better. However, the lab has not published further tests since then, as the consensus is that hosted mail solutions like Gmail have rendered traditional spam filters largely unnecessary.
In the past, we conducted our own antispam tests using a carefully curated real-world account designed to receive both spam and legitimate emails. This involved downloading thousands of messages and manually verifying whether any spam infiltrated the inbox or if valid emails were misclassified as spam. Given the minimal importance of this feature today, we have shifted our approach. While we still evaluate a suite’s antispam capabilities, we no longer engage in extensive testing.
Security Suite Performance Doesn’t Require Testing
As security suites diligently monitor for malware attacks, defend against network intrusions, and prevent access to dangerous websites, they utilize system resources such as CPU and memory. Historically, security suites garnered a reputation for consuming excessive resources, leading users to disable protection due to performance issues. However, advancements over the years have significantly improved the efficiency of these products, eliminating noticeable slowdowns. Consequently, we have discontinued our performance impact tests, as the majority of recent suites demonstrate minimal or even improved performance post-installation.
We Test Firewall Protection Several Ways
A typical personal firewall serves two primary functions: safeguarding the computer from external threats and ensuring that programs do not misuse network connections. In the past, we employed physical computers connected directly to the internet via a router’s DMZ port for testing. However, advancements in router technology have made such connections more challenging. Given that most computers are now connected through routers, which effectively conceal them from the broader internet, our previous port-scan tests have become less relevant. The built-in Windows firewall already manages port stealthing, prompting us to retire this test.
We Evaluate the Firewall’s Program Control
In earlier iterations, personal firewalls required users to interactively approve or deny network access for new programs, a method that often proved ineffective. Users frequently lacked the knowledge to make informed decisions, leading to either excessive allowances or unnecessary blocks. Modern suite firewalls typically include program control features without making them the default setting. For our hands-on evaluation, we enable these features prior to testing. Many firewalls now come preconfigured with access permissions for known programs, enhancing usability.
We Check Protection Against Exploits
Software is inherently imperfect, and cybercriminals diligently seek vulnerabilities in popular operating systems, browsers, and applications. They develop exploits to compromise system security, necessitating timely security patches from the software developers. The most effective firewalls intercept these exploit attempts at the network level, preventing them from reaching the computer. For those that do not operate at this level, the antivirus component often neutralizes the exploit’s malware payload. We utilize the CORE Impact penetration tool to assess each test system’s resilience against approximately 30 recent exploits.
We Probe the Firewall’s Defenses
Finally, we conduct a sanity check to determine whether a malware coder could easily disable security protections. This involves searching for on/off switches in the Registry, testing whether security processes can be terminated via Task Manager or third-party utilities, and verifying the integrity of essential Windows services associated with the product.
We Verify Parental Control Features
Currently, PCMag does not rate or recommend standalone parental control services, instead advocating for the free screen time and parental features integrated into modern operating systems. However, when a security suite includes parental controls, we ensure these features perform as promised. Typical parental control utilities aim to shield children from inappropriate content, monitor internet usage, and set time limits for online activities.
To validate the content filtering capabilities, we conduct tests to ensure that inappropriate websites are effectively blocked. Finding explicit content for testing is relatively straightforward, as URLs often follow predictable patterns. Most products pass this test with ease. We employ a custom browser utility to confirm that content filtering functions independently of the browser in use. Additionally, we verify that the time-scheduling feature operates correctly and cannot be easily circumvented by altering system settings.
We also examine the program’s claimed features, such as the ability to block specific applications or filter out inappropriate language in communications. By testing these functionalities, we ensure that the parental control features meet their advertised capabilities.
We Release Real World Ransomware, Safely
Modern antivirus solutions are designed to defend against a spectrum of malware, including the particularly insidious ransomware. This type of malware poses a unique threat, as failure to detect it can result in irreversible data loss. Consequently, many antivirus programs incorporate specialized defenses against ransomware, such as monitoring for suspicious behavior or requiring authorization for changes to critical files. Our testing rigorously evaluates these ransomware protection measures.
To initiate this test, we disable all other real-time antivirus protections, simulating a scenario where a zero-day ransomware attack bypasses existing defenses. We isolate the test virtual machine from the internet and systematically release a series of real-world ransomware samples. When the antivirus successfully detects and blocks an attack, we verify its efficacy, ensuring that no files were encrypted before detection occurred. We also document instances where the ransomware presents its ransom note, even if it fails to encrypt any files.
We Monitor and Interpret Antivirus Lab Tests
While we lack the resources to conduct exhaustive antivirus evaluations akin to those performed by independent laboratories, we diligently track their findings. We focus on five labs that consistently release scored test results, which inform our reviews.
AV-Test’s Three-Way Evaluation
Based in Magdeburg, Germany, the AV-Test Institute continuously subjects antivirus programs to a variety of assessments. We concentrate on their three-part test, awarding up to 6 points in each of three categories: Protection, Performance, and Usability. To achieve certification, a product must accumulate a total of at least 10 points without any zeros in any category. The highest-performing products can attain a perfect score of 18 points.
For the protection evaluation, researchers expose each product to a reference set of over 100,000 samples alongside several thousand widely prevalent samples. Products earn credit for preventing infestations at any stage, whether by blocking access to malware-hosting URLs, detecting malware through signatures, or preventing execution based on behavioral analysis. The best products often achieve 100% success in this category.
Performance is crucial; if an antivirus noticeably slows down system operations, users may disable it. AV-Test researchers measure the time taken to perform various common system actions with and without the security product, identifying any performance impacts.
Multiple Tests From AV-Comparatives
We monitor results from three specific tests conducted by AV-Comparatives, an Austrian lab collaborating with the University of Innsbruck. Products that pass these tests receive Standard certification, while those that do not are labeled as merely Tested. Exceeding minimum requirements can earn Advanced or Advanced+ certification.
The file-detection test runs each antivirus against approximately 100,000 malware samples, while a parallel false-positives test ensures accuracy. Similar to AV-Test, the performance evaluation measures any impact on system performance. The dynamic whole-product test is particularly significant, as it simulates a real user’s experience, allowing all components of the security product to engage in combating malware.
SE Labs Replays Actual Malware Attacks
Unlike AV-Test and AV-Comparatives, SE Labs typically assesses a limited number of products, focusing on realism in their testing. Researchers capture real-world malware-hosting websites and employ a replay technique to ensure each product encounters the same drive-by downloads or web-based attacks. A program that completely blocks an attack earns three points, while partial successes receive lower scores. If malware runs unchecked, the product incurs a penalty.
Tests by MRG-Effitas Are Pass/Fail
We utilize a feed of samples from MRG-Effitas for our hands-on malicious URL blocking test and follow their quarterly assessments. Their 360 Assessment & Certification test simulates real-world protection against current malware, similar to AV-Comparatives’ dynamic test. Products that entirely prevent infestations receive Level 1 certification, while those that allow some traces but later eliminate them receive Level 2 certification. Any product failing to achieve either level is deemed unsuccessful.
AVLab Reports Malware Defense Percentages
The AVLab Cybersecurity Foundation in Poland is a recent addition to our lab collection, challenging antivirus applications with in-the-wild malware and reporting the percentage caught both immediately and post-launch. Success is defined by either detection method.
We Calculate an Aggregate Testing Score
Summarizing lab results presents challenges, as different labs utilize varying scoring systems and test different collections of programs. We have developed an algorithm that normalizes scores to a scale of 0 to 10. Our aggregate lab results chart reflects a weighted average of these scores, accompanied by the number of lab tests conducted. The pinnacle of achievement is a perfect score of 10, derived from results across all five labs.
What About VPNs?
It is worth noting that our testing methodologies do not encompass virtual private networks (VPNs). Evaluating a VPN requires a distinct approach, which is detailed in a separate article dedicated to testing VPN services.
