BleepingComputer.com: Test Results

I would read this whole topic, but my brain's kind of on the fritz ATM....maybe later tonight, I'll read it when my brain isn't as racked with stuff.

This post has been edited by scff249: 03 November 2008 - 05:47 PM

Hello.

Billermo said:

Grinler said:

I think this point says it all. The files used in tests are just that: files. They are not active. In real life, infections don't sit there waiting to be detected.

An active infection can evade detection and even disable the AV all together. I doubt there is a test that can replicate an environment where this is happening and judge an AV's ability to fight active infections.

Here, in this topic, we mainly discusses detection. What about an AV's ability to remove infections? Interested in bringing that into the fray?

With Regards,
The Panda

Sorry I'm meaning to reply again but have been very busy last couple of days -- will get back on here, catch up and respond as soon as I get some down time, probably tomorrow.

Sorry again. Thanks for the replies

Galadriel, on Nov 3 2008, 09:16 PM, said:

I think we have 2 separate issues identified here:
1. are there tests that are trustworthy?
2. if so, is the methodology used going to be effective in finding real-world like results?
If it was possible to satisfy these criteria, I tend to think a test would be of some value in evaluating an AV program.

Galadriel, on Nov 3 2008, 09:16 PM, said:

I'm not necessarily against considering more than tests -- it's just that if good tests exist (see above) then I would tend to think those should also be considered. To what degree, I admit I'm still trying to figure out. (see below)

Galadriel, on Nov 3 2008, 09:16 PM, said:

OK but my devil's advocate argument has to kick in now -- no one can be sure if their test-winning AV made them safer or not -- absent any statistics on their usage or incidence of running up against viruses, there's no quanitifiable way to know this for sure. It's possible they have great AV but spend all their time surfing warez and porno sites, and for that reason were at much higher risk than most of catching viruses. How much safer did their AV make them? We can't really know. Maybe it killed off 99% and they got hit with something out of that 1%.

Galadriel, on Nov 3 2008, 09:16 PM, said:

No arguments here -- the kind of advice given out here about safe practices seems wise to me. Good habits like these will reduce risks. I'm just saying that an AV that performs better in virus detection than others to some degree would also reduce risk just that much further. If I said (or seemed to say) earlier that the detection rates were the main thing in keeping someone safe, then I can see where that would be wrong. I still might argue that the detection rates would be the main criteria in choosing which AV to go with though, if someone's highest priority concern was being as secure as possible (worrying less about, say, scan speed or if it slowed the system down).

Galadriel, on Nov 3 2008, 09:16 PM, said:

OK sorry for putting words in your mouth then. Nope, never did attempt uninstalling Norton that I can remember.

Quote

To be fair, I think we would need to look at the methodology anyway because I'm not sure that they run the test by hitting the AV with thousands of samples at once. They may be doing them one at a time, for all I know.

Quote

OK I didn't realize the significance of Grinler's points here at the time, didn't envision it the way you described it, as all the samples hitting the AV at once. At the time I just wondered for a moment what difference it would make how many directories they were in. Number 2 I didn't consider enough -- now seeing the below I realize it's a huge question. In number 3, the tests I see do specifiy which version and update was used, so that one seems to be taken care of. Number 4, they identify the platform and specs, and apparently aim to use a fairly typical-user setup (for instance using XP these days since it's still about 70% of the machines out there).

Quote

First, the idea in bold up above seems very key to me -- this description of the virus 'running' when it arrives on your computer. I haven't heard viruses described along these lines before, so need to know more about that. Is this all viruses, or some, or a certain number of them, that behave this way? I assume they get activated by the user executing them somehow, in most cases? Could you describe just one good (fairly typical) example of a virus and how it behaves? And I guess it's not quite answered yet whether these tests do have the viruses hit the machine in this activated/running way you describe. You think that's impossible, and you may be right. I'm new at this, so I admit I dont' know enough to answer these questions, and am curious to find out for sure if you're right about that or not.

PropagandaPanda, on Nov 5 2008, 05:50 AM, said:

OK I think I just talked about this a bit -- so it looks like that's a key question that would determine how relevant the test is in a real-world situation. I'd still like to understand a bit better what you mean exactly by how the virus is 'active'. And how that is different from just being a file when it's detected. And maybe another angle to consider is what proportion of viruses behave this way, and if there are some that are just inactive files out there waiting to be downloaded.

Removal? Hoo boy. OK if we note this one for further discussion as soon as we get some closure on this part first? I notice some of the readers' brains are hurting (and that might include mine too now). Tempted to respond about it, but ... resisting.

This post has been edited by Billermo: 06 November 2008 - 03:48 AM

Quote

Viruses, hijackers, trojans, backdoors, rootkits, they all are pieces of code that in order to do anything need to be executed (or run) somehow. Cause and effect. For the payload to affect the system, the "bad" file needs to be run, just like any other program. You have to remember that computers make no difference in what code gets run. As long as said code is valid and functional, if the file is executed in any way, the code will be run. A virus or a trojan, or whatever baddie it doesn't really matter, needs to be executed to affect the system. The file itself, even if identified as a virus, won't do anything until the code is processed by the machine. To use an analogy, because they work and you seem to like them, a bad file would be like a car. The file itself, not active (i.e. not running, just on the HDD for example) would be like a car with the ignition off. Won't go anywhere until you actually hit the ignition (i.e. a starting point, or a loading point).

That's just how the Operating System works. Now, keep in mind that there are more ways to run files than just by double clicking them. Not all files will execute when double clicked. File extensions/file types come into play here as well. You could very well have a file be detected as a virus and it not be anything more than a remnant, or a log file for example. Those, in perspective, wouldn't do anything if you left them on the computer. And they're presence does not mean you are infected. It only means that at one point you may have been. But most AVs don't distinguish between the active executable files and those remnants. The why of that behaviour is a debatable subject, but let's just say they do that in the spirit of completeness. Although I have seen many cases (read: most) where not all files that are brought in by an infection get detected or removed. Propaganda Panda's point of removal as a valid subject in this discussion is also tied in to what we were discussing.

To go back to loading points and examples of running files, there are litterally hundreds of ways a file can be executed, not all of which will be visible to standard tools (i.e. task manager). A lot of the baddies will "hook" themselves in legit, valid, important system files like userinit, or winlogon, or explorer itself. Some masquerade as legit files, with same file name, and different location. Some will overwrite legit system files. Some will change one letter to confuse you into running it or leaving it running. Those are all reasons the average user has a hard time determining what's good versus what's bad. Because there's a lack of knowledge in how the OS works, what it needs and what's not quite right. When a baddie ties itself to an important system file, the standard behaviour of an AV is to attack the file. Some will quarantine it, some will delete it, some will try to "heal" it of the bad code. Neither of those options is 100% fool proof. Remove userinit.exe and you have a non functioning copy of windows. Remove winlogon.exe and you can't login. Remove explorer.exe and things get difficult even though you can still boot up. Those are just examples, but they are used very often by malware coders. Because they make removal difficult. Rootkits are being used more and more for that same reason. Not only is it difficult to remove them, but detecting them is also very difficult without proper tools.

So whether the file being detected as a virus in those tests is actually "running" or installed as it would be during a live infection, is indeed a very very important aspect in the relevancy of those tests. I had a big collection of baddies at one point, when I was more active as a helper on the forums, I gathered a lot of "new" nasties to get them added to AV/AS program detections. I helped analyse/reverse engineer to a certain degree, and kept most of them in a "nasties" folder I setup. Their presence on my HDD was detected by AVs, but none of them were actually running, therefore they weren't a threat to me. Now if someone else had executed them, they would indeed have been a threat. See the difference?

The fact that thousands of infections wouldn't be possible to have actively running, is not something that's really debatable. The OS slows to a crawl when one worm is installed, imagine 10? imagine 100? A thousand? Blue screens/shutdowns/unable to do anything at all is more likely...

As for a typical infection scenario, there really isn't a typical one. There are many ways to get infected. Mostly through security exploits. Either in the OS itself (Windows in this case) or through a exploitable application (i.e. Internet Explorer, Adobe Reader, Java, javascripts, Real Player, Quicktime... the list is endless). Once the original piece of code is run by whatever mean that particular one chose, the infection is active. If an AV catches it in the act, it "may" prevent the payload. I emphasize "may", because more often then not, they don't... By the time the AV acts, usually, the infection already has a hold on the system. So removal becomes very important. Can the running files be stopped? Can they be removed safely? And most importantly, will they stay gone? Most baddies are wise, they watch over themselves, protect their processes. Stop it and another pops up. Worms are a good example. The definition of a worm is a bad executable whose sole purpose is to replicate and infect others. Some of them don't do anything else but bog down the system, use up CPU resources, HDD read and write cycles. Why? To disrupt the functioning of the computer of course. And believe me, they are very efficient at it. Another thing to remember is that most baddies these days don't like to be lonely. They will call their buddies and have a big party in your system. Bringing others into the fray also adds to the payload. Those are called "downloaders" or "droppers". Their sole purpose is to get more stuff in your system.

Malware classification isn't an exact science. Because the lines of definition are becoming blurred more and more. You have viruses with trojan like behaviour. You have worms which also act as backdoors. You have droppers who are hijackers as well. That is the reason antiviruses, anti spywares and anti trojans are targetting a lot of the same things. Because the lines aren't as clear cut as they used to be.

I hope that helps you understand a bit more.

Galadriel, on Nov 7 2008, 12:17 AM, said:

I had an idea this was what you were talking about, though I'm still not clearly getting how it works, or might work. I did already know that virus files need to be active to do anything -- I've been at work at the office in the past when an email virus suddenly hit, which was prompted by someone clicking on the attachment in the bogus email, which sent the virus off to everyone in their Outook addressbook (so, everyone in the company). And since then I learned how viruses need to be active to do their damage. And I do realize the difference between active and activated. I understand (and have experienced) that they're not the same thing, that they can become active without the user doing something to activate them.

But what I don't quite get is exactly how the virus might be active as it enters your computer. I wish we had at least one, if not typical, then maybe one that at least illustrates this real-world scenario, the way it gets on the computer while active.

I guess another part I don't quite understand is why we're assuming the AV would not detect it if it was active but would detect it if it was just an inactive file.

Galadriel, on Nov 7 2008, 12:17 AM, said:

It occurs to me now that we may actually need to think about the separate parts of the AV (or maybe not, not sure) -- typically in AV's there is a system scan and also real time protection. Or are these in effect the same since they both work using the same updates of recognized files? From a user's point of view -- ok just to give a specific example of a situation I've dealt with often -- I use a USB stick save some files off a computer at work (a shared one that others often use USB's on, so it often gets infected)* and bring it home and plug it into my own computer. Just as I plug it in, a warning box pops up from my McAfee telling me a virus has been detected on the USB stick, and do I want to delete it, quarantine it, etc. So in this case at least, the AV seems to be catching the virus before it has a chance to do anything (or so I'm led to believe at least, but who knows). I guess you would probably say that by being plugged in, those files are now 'in my system'. And also that there may be some activity happening between the USB that would make them active -- in fact the worst infection I got this way seemed to behave that way -- merely by plugging in, it became activated. In any case, it was the real-time monitoring that seemed to catch it -- or resident shield or whatever it's called in other AV's.

* I now realize this is absurdly high risk behavior so I don't do it anymore -- much safer to email myself the files. It was always a hidden autorun file that was activating the virus with these.

Galadriel, on Nov 7 2008, 12:17 AM, said:

I see that now. I can now report that I went through the methodology paper by AV-Comapratives and didn't find any details that would be helpful in understanding this aspect of the test. The methodology focused in plenty of detail on other aspects, but not at all on this one.

Galadriel, on Nov 7 2008, 12:17 AM, said:

Yes.

Galadriel, on Nov 7 2008, 12:17 AM, said:

OK but it seems as if we still are presuming that the test involves all the samples hitting the machine at the same time, which we don't know is true.

Quote

This makes me think that for the test to be valid, the samples have to be analyzed to the point where their specific attack strategies are taken into account, and re-enacted realistically with the AV running. To me, Mr New At This, this seems difficult but not impossible. When I see that AV-C is using millions of samples in their 'comparative' test, it does make me think it's very unlikely they're doing a test that's this difficult.

Quote

What comes to mind is that if it's true that there's a huge gap between real-world/active viruses and the inert files they use in these tests, then it makes me think the real-world percentages would probably be much lower than what the tests show, and almost surely can't be higher as long as the methodology is fair and represents a true spectrum of what's out there. If that was true, the degree of those percentages would be misleadingly high, but the relative differences between them, especially if persisting in test after test, would still seem to indicate at least some indication (albeit a rougher one) of their detection rates relative to each other.

Billermo, on Nov 9 2008, 08:01 AM, said:

Ok. I'll try my best to describe a plausible scenario.

Let's say a user is surfing the internet just looking for stuff on google about a song or an artist. Google results will often link you to lyrics sites when searching for songs. That's a good example because most everyone searches for that kind of stuff and will encounter those sites at some point. Most of those sites are riddled with ads. Those ads are the ones people need to be wary of. To display those ads, the webmasters use scripting, because they are often from a database of different ones, and the script will pull a random ad from the list on page load and display it so you wouldn't get the same ads on a reload. Now if the ad itself is coded in flash or shockwave, like most are, they can be coded to do just about anything. Flash has an extremely powerful programming language built in. There's a lot more to flash than cool graphics and effects. Flash is just one example, the same is true for javascript. Javascript is executed in the browser and if the browser has flaws in how it handles the script, you can do anything you want to a machine if it isn't properly patched.
There are several ways to download and execute a file using javascripts, and most of them are completely silent to the user. So keeping this in mind, imagine your user again is on a site that holds a bad script. If javascript is enabled and running, the script gets executed. If the script calls for a file to be downloaded and run on the computer, it'll happen and in most cases, it'll happen silently. Now if the file is a known one, the AV might catch it before it executes, but very often those files change as soon as AVs target them. So a lot of them go undetected. That's the original "exploit file". Once that file runs, it may download other files to install whatever nasty it intends on installing. Everything happens very fast, and most of the time, if the exploit file itself isn't stopped before it executes, the infection grows and is hard to stop. Once the symptoms of infection appear, it's too late to prevent. Removal becomes the priority.

Billermo, on Nov 9 2008, 08:01 AM, said:

Actually we're not assuming it wouldn't detect it if it was active, but it may not detect all of its components.... And in network security and malware removal, if you don't get all components, the likelyhood of reinfection after removal of parts of it, are extremely high.

Billermo, on Nov 9 2008, 08:01 AM, said:

That's a good point. Because yes, those two methods of identification from AVs work differently. One will use definitions and look at specific files on the system (the system scan) and one will look at that in lesser detail but will focus more on behaviour (real time shields).

Billermo, on Nov 9 2008, 08:01 AM, said:

Good example. Mcafee knew the file was bad when it saw it. The reason it saw it, is because when windows explores a folder/or a drive, it will access the files (read them, not execute but read). They will go in memory for a short time and that's what most AVs look at in real time monitoring. The USB infection you speak of, is somewhat different, in the fact that the bad file actually gets run. The autorun process when you insert a drive in the computer is at fault. You can have it point to any executable. Usually good autoruns will just tell the computer to open up the folder for viewing with explorer. But bad files will often cause autorun to execute the main infection file on every insertion. So merely plugging in a USB drive with a bad autorun will infect if the file the autorun points to is present on the drive.

Billermo, on Nov 9 2008, 08:01 AM, said:

That's correct. We have no way of knowing that that is the case or not.

Billermo, on Nov 9 2008, 08:01 AM, said:

Absolutely. You are starting to see why the reluctance in trusting the tests in general.

Billermo, on Nov 9 2008, 08:01 AM, said:

The gap may not be huge, and it may. Therefore we have no way of knowing what kind of consistency there could be in those tests with regards to real in the wild infections and how they behave. The tests might be useful, they might not at all. And that is why I don't put much faith in them myself.

Galadriel, on Nov 10 2008, 02:45 AM, said:

I think we're near the point of going down this road by now. Almost.

Quote

So are you saying that active/inactive is not the issue, but rather the question of catching all components is more the issue? Or some of one and some of the other? I think these components are still part of the category of viruses, they can't really be considered anything else. So it's a question how those tests deal with these, or if they do, and if so, how.

Quote

I'm going to try to get an answer to that question.

Quote

Since development in the area of malware is so fast and varied, it seems as if even if the testing facilities did do a good job of testing the detection rates of the different software (which you don't even agree they do, and I'm still trying to check into), then that should be treated by people who look at those results as something like the performance of a company stock or an athlete over a certain period. I mean, it's a snapshot. The performance may be better or worse in the future, but the tests might still function as a kind of track record. If the AV performed well over a long period of time, chances are pretty good it's going to keep performing pretty well going forward. But things change, so its future performance can't be predicted with certainty.

But it just leads back to what the quality of the test results are. It seems like we're at a dead end on that question unless we can find out more about whether the tests address these issues. You believe they can't possibly address them, I get that. I don't have enough experience in this field to be as sure as you. I'm going to see if I can find something more out from AV-Comparatives.

Thanks for all the input on this. Very much appreciated.

Boy oh Bob what a subject. Only a few inside people will ever be give the exact method of how these test where done and with who. I don't have those connections. I do research on this site and many others. I use this information and my own from the many products I have used and make my own choice which one I want and go out to the store and buy it. I will find out if my new Kaspersky does the job. I have also used Norton and Zone Alarm and I have been safe with all of them.