Thoughts on "The PC Security Channel" malware tests

Greetings to this great community!

I have been using windows defender on windows 10 and have seen that windows defender is trusted nowadays.

Recently I came across "The PC security channel" [TPSC], a youtube channel where you can find various antivirus vs malware tests.

Windows defender had a very poor performance on the TPSC test as opposed to third party AV like Kaspersky or Bitdefender.

See the test here:

and the methodology described here:
https://www.thepcsecuritychannel.com/

I am confused, since Windows defender usually scores really well on AV-test and AV-comparatives.

So, the questions is: What are your thoughts on this channel's tests? Do you think they are reliable?

One thing to note is that he disables protection in order to get the malware onto the machine in the first place...

A direct quote from the video:

"The way I usually run these is I turn the product off for a second just so I can grab my malware. Don't worry this is only temporary. I will actually re-enable it before we execute anything. It's just to give me a window of time to be able to grab my malware without it interfering."

You should want your antivirus to interfere. That is the whole purpose of the program. The thing with antiviruses is that they are a multi-layered approach with different mechanisms in place to prevent malicious activity from running. Disabling the antivirus defeats the purpose of evaluating its protection capabilities as a whole.

The methodology on their site even admits "TPSC tests focus on the last line of defense i.e. protection during execution of malware."

Overlooking aspects of an antivirus, such as its ability to prevent you from downloading the malicious files in the first place or blocking suspicious activity (i.e. scripts), isn't how you should evaluate an antivirus in my opinion.

Thanks Protomartyr for your response.

What I still don't get is: if windows defender doesn't detect the threat when the user executes the executable (when it has all the info available to analyze the actual file and behavior of it), how would it be more effective at detecting the threat even before downloading the malicious file?

It depends a bit on the AV, but most have different protection layers. And for example URL protection is different than file protection. So, if you have a malicious url that downloads a malicious file, and the products already has a detection for that malicious url in place, then it will just block the file based on the fact that it's source is bad. It does not further analyse the file. That is good of course, because it adds an extra layer of protection. But at the same time when testing, you really want to know if a product also can block the actual file that is executed, no matter it's source. 

The latter is done based on the file's actual behavior, not it's source. That means in practical terms that, if the file is detected, it doesn't matter it ends up on your computer using a drive-by download, an email attachment or social media spam, it'll be blocked regardless. If you rely on URL-based blocking first in a test, without looking at actual file/behavior blocking, there's no way to know if the product would also have blocked another infection vector (so you know a drive-by download would have been blocked, but you have no way to know if the same malware would still have been blocked if it would have been dropped by a malicious email attachment).

When it comes to AV testing imho transparency is a must; if you as the product user don't know how a test is performed, there's no way for you to make a decision on how well the product will protect you; a 100% score on a test means nothing if a test is not conducted properly or if you have no idea how the test was done in the first place.

Finally to answer your question:

 

 

if windows defender doesn't detect the threat when the user executes the executable (when it has all the info available to analyze the actual file and behavior of it), how would it be more effective at detecting the threat even before downloading the malicious file?

It would be more effective is a known malicious link is used; that relies only on url blocking without looking at the file.

Thanks Elise for your valuable info. It's clearer now

I wouldnt trust the tests done by AV comparatives, they never show their tests, the antiviruses they are testing  always gets a detection rate close to 100%, they even tested a scam/fake antivirus (TotalAV) and it got a 98% detection ratio, I would say they are bogus.

Edited by EmanuelJacobsson, 02 December 2019 - 03:31 PM.

They are not bogus, but since tests like these ones cost a lot of money for AV vendors to participate its more likely a vendor will be willing to participate only if the tests performed will turn out well for their product. Otherwise it would be like paying a lot for bad marketing.