Jump to content


 


Register a free account to unlock additional features at BleepingComputer.com
Welcome to BleepingComputer, a free community where people like yourself come together to discuss and learn how to use their computers. Using the site is easy and fun. As a guest, you can browse and view the various discussions in the forums, but can not create a new topic or reply to an existing one unless you are logged in. Other benefits of registering an account are subscribing to topics and forums, creating a blog, and having no ads shown anywhere on the site.


Click here to Register a free account now! or read our Welcome Guide to learn how to use this site.

Photo

Test Results


  • Please log in to reply
39 replies to this topic

#1 Billermo

Billermo

  • Members
  • 110 posts
  • OFFLINE
  •  
  • Local time:08:32 AM

Posted 30 October 2008 - 02:13 AM

In other discussions on this forum, I've discovered that considering test results of the different AV programs seems to be a controversial issue here. So far I've mainly seen consensus in favor of the opinion that first-hand experience should be considered more highly than test results. My own opinion is actually opposite. Please let me explain by offering a parallel situation that I think holds key commonalities with the selection of an anti-virus program.

Car safety is a good example of the way I think people should be thinking about their anti-virus software. A person can go his entire life feeling safe driving in a car that has no air-bags or even using seat belts. The illusion of safety is created because a single person, on their own, rarely encounters very dangerous situations. What you want to be prepared for, though, in terms of car safety, is that unusual moment when you are suddenly in danger. I've heard people here say they have been using X product for years with no problems. Well, the same could be said for most car accident victims. Driving with no airbag or seat belt worked perfectly well for years, until the day they were killed. Its an extreme analogy, but the dynamic of the risks is similar.

Just as in car safety tests, where cars are put to the test of the most dangerous situations only, namely impacts, again and again, in order to thoroughly test how well they perform in just those situations, so day-to-day typical use of an antivirus program doesn't seriously test the protection value of the software.

One thing I should mention is my own experience. I was hit by a very persistent, difficult-to-remove virus in this past year. That virus was originally on a computer that was protected by AVG free. It ended up on my own McAfee-protected computer by way of a USB thumb drive. Neither program detected it. I also ran Bitdefender's online scan, and that also did not detect it. For anyone interested in the details, I sought help with this virus on this forum and also on techsupportforum.com -- you can search and find the step by step process of what needed to be done to get rid of it, and how 3 major AV programs totally failed to deal with it or even find it.

So for myself (and I guess it should be the same for anyone), what is most important is finding an AV program that simply has the consistently highest rates of detection and also is best at catching viruses that are not yet in its own database (what they refer to as 'heuristics' -- the ability to spot something that appears to be a virus that hasn't yet made it to their updates of recognized viruses). So for me, listening to people's first hand experience just isn't enough. No program will catch everything, but some surely tend to catch more than others. But by listening to individual users' experience only, there is no serious way to compare them. Individual users can't intensively test their own program vs many viruses, or versus other AV programs. A whole lifetime of a single user's experience is far less intense than a single anti-virus test, where millions of virus samples are thrown at an AV program in one go. A single user doesn't keep track of their results. A single user is just going by gut instinct alone. That's a completely unscientific approach to the subject. You can take the anecdotal evidence of the user who had no problems for years with AVG and put it next to my own, where I got killed using AVG, McAfee and Bitdefender, and what does it total up to? You're in the dark.

I've heard people talk about the fairly useless 'bells and whistles' that come with paid programs -- I tend to agree with that view. The purpose of an AV program is to protect the computer from viruses. If it can't do that well, it's not a good program. The UI is low priority. All the bells and whistles are too. But to find out how well it does its main job, I can't see how anyone can refuse to look at test results. I've been trying to find good information about test results and have only come across av-comparatives.org so far as a good source. There are others but they don't share the details of their test results, which makes them diifficult to appreciate. If someone here would go look at av-comparatives.org 's tests, you can see what I mean. They show the actual % detection rates, the contents of the samples used, and if anything an overwhelming amount of details of their tests. I'd love to see test results like these from other organizations conducting tests. As mentioned by someone, magazine round-ups also are dubious because they tend to give too much credibility to those bells and whistles (like UI or extra features) and downplay the actual detection rates, which is far and away most important. The bottom line is detection rates. The rest is unimportant.

In the interest of thoroughness, I'd be interested to hear from people here who disagree with me, and why they do. I've heard points raised opposing my view, and am ready to listen and consider those. Please be prepared, if you are tempted to make such points, that I will always ask for some kind of convincing corroboration of a point -- basically that a point has to be supported by evidence. If I do that, please don't see it as an attack, and I'll do my best not to phrase it as such, but rather to state it as diplomatically as possible.

Also, I am NOT an expert. I'm just a guy curious about this, about finding the best AV program out there. But I'm also not going to presume that someone else is expert, and that their word on the subject should go unquestioned. Anyone who is an expert should be able to easily back up their contentions with evidence, and since I don't have any idea what someone's credentials are here, or what would make them expert, I'm just going to presume they can pass the expertise test in their statements and with the quality of their evidence.

BC AdBot (Login to Remove)

 


#2 Billermo

Billermo
  • Topic Starter

  • Members
  • 110 posts
  • OFFLINE
  •  
  • Local time:08:32 AM

Posted 30 October 2008 - 03:12 AM

Just came across this list of a test reported last month on Virus Bulletin (tests done by AV-test.org):

http://www.virusbtn.com/news/2008/09_02

34 different programs tested. Details of the tests are better than I've seen from this group before (about 1 million viruses and 90,000 spyware samples used), though they don't give a breakdown of what kinds they used, as AV-comparatives does. These results have a lot of overlap with AV-comparatives' tests. For fans of free AV, the good news is that Antivir performed very well. The bad news is that AVG had mediocre performance.

#3 Billermo

Billermo
  • Topic Starter

  • Members
  • 110 posts
  • OFFLINE
  •  
  • Local time:08:32 AM

Posted 31 October 2008 - 12:32 AM

Hm, it seems as if all people who not long ago were coming from all directions telling me tests were irrelevant, are all strangely silent now.

I thought it would be good to offer them a chance to make their case, and we could see if it stands up to scrutiny.

#4 DSTM

DSTM

    "Bleepin' Aussie Addict"


  • Members
  • 2,649 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:SYDNEY-AUSTRALIA
  • Local time:11:32 AM

Posted 31 October 2008 - 12:43 AM

We have offered our opinions, and don't have to support it, with proof.
As I said before,in a previous post, in another Thread, of yours.
Find one your happy with, that doesn't let any nasties past.Not happy choose another.
You have done your research,make up your own mind. :thumbsup:















#5 Billermo

Billermo
  • Topic Starter

  • Members
  • 110 posts
  • OFFLINE
  •  
  • Local time:08:32 AM

Posted 31 October 2008 - 03:22 AM

Well, in a serious discussion, I thought it was expected that an opinion be supportable if it's to be considered valid. Should these opinions be taken seriously or considered valid? I think this would hold true more so when they're being offered here as expert advice.

Obviously I'm challenging what I see as the conventional wisdom here on this site, which I've seen expressed as advice from the moderator on down. To me, it strikes me as not the best advice (initially at least) -- so I'm just trying to find out what's behind it. If these opinions can be justified, ok let's see it. I'm ready to listen. I guess my first post gives a decent road map of what I might say in reply, so it should be easy to come prepared. If there are some flaws in my thinking, and someone can point them out and show why, I'm ready to admit that if it seems to be correct.

Edited by Billermo, 31 October 2008 - 03:54 AM.


#6 rowal5555

rowal5555

    Just enough info to be armed & dangerous...


  • Members
  • 2,644 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:St Kilda, Dunedin. South Island. NZ
  • Local time:02:32 PM

Posted 31 October 2008 - 05:57 AM

Now, Now. Play nicely.

I hear what Billermo is saying and he is entitled to say it, but seems to belabour the point a bit.

From my point of view, my Sygate, Avast, SuperAntiSpyware Pro, ThreatFire, and WinPatrol have worked satisfactorily for me for years and I have no wish to change, even though test results may show other programs work better. My point is, experiment, find what works for you, and be happy with it.

Cheers

rowal5555 (Rob )                                                             

Avid supporter of Bleeping Computer's
Team 38444

You can help find a cure


 


#7 DSTM

DSTM

    "Bleepin' Aussie Addict"


  • Members
  • 2,649 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:SYDNEY-AUSTRALIA
  • Local time:11:32 AM

Posted 31 October 2008 - 06:13 AM

Now, Now. Play nicely.

I hear what Billermo is saying and he is entitled to say it, but seems to belabour the point a bit.

From my point of view, my Sygate, Avast, SuperAntiSpyware Pro, ThreatFire, and WinPatrol have worked satisfactorily for me for years and I have no wish to change, even though test results may show other programs work better. My point is, experiment, find what works for you, and be happy with it.

Cheers

Of course, everyone is entitled to have their say,Rowal5555.
When a member comes on, and attacks a Moderator,is rude and judgemental of every member's post, and now questioning the advice and integrity of this site,then I say, enough is enough.
IT'S NOT WHAT YOU SAY,IT'S HOW YOU SAY IT.
Billermo,nothing here is offered as expert advice.Techies give their free time and more to help us all.
We help each other here,with what we know,and you dont have to take excellent advice, if you dont wish to. :thumbsup:

Edited by DSTM, 31 October 2008 - 09:59 AM.















#8 Billermo

Billermo
  • Topic Starter

  • Members
  • 110 posts
  • OFFLINE
  •  
  • Local time:08:32 AM

Posted 31 October 2008 - 06:46 AM

Hey let's be gentlemen this time, OK? I'm going to ignore the parts of your post that don't sound very nice, and focus instead on the part that seems to be more serious.

I wouldn't say that I love the test results. I've actually come across some I'd even say look suspect -- such as one website offering test results conducted by someone who worked for Kapersky previously, which all just happen to have Kapersky either alone at the top or at the top along with others. If anything I'd like to further check out how credible these testing outfits really are to see if they're legitimate. But yes, in general I'd say that checking how the programs perform versus one another, and against millions of viruses as opposed to an unknown number, and under similar conditions rather than unknown conditions -- that in general, controlled testing is surely a more intelligent way to determine which ones perform better than others. I'm surprised this seems like a controversial idea to anyone here. We live in a world where the idea of testing products is standard, and where technological advances depend upon it. So it's surprising to me that this idea upsets people here.

The most important point you make is that you don't trust the tests. Can I ask why not? You say they don't have relevance to the real world. I don't understand why you think this. According to the methodology descriptions I'm seeing, the viruses used in the tests are taken from the real world, and the real programs are used, the same ones that exist in the real world. Would you say that about tests for auto safety, testing the safety of seat belts or airbags, that they have no relevance in the real world? Maybe this is all barking up the wrong tree -- maybe you or someone here are aware of some report that has debunked these particular testing facilities. I haven't come across that yet. I have come across lots of references that seem to refer to them as credible. And the AV companies themselves cooperate with them by agreeing to have their products tested (that includes the ones offering free products), and seem to continue to even if their AV doesnt' perform well in a test. Is it the quality of the individual testing facilities that you don't trust, or the very idea of testing? I'm honestly still not clear on what the basic rationale is behind the mistrust of testing.

Anyway, please don't take any of this personally. It's not meant to be. Like you said, you've made some points that I have listened to, investigated, and have been able to corroborate by checking out further -- such as that the core of virus protection offered by AVG and Antivir are mainly the same in their free and paid products. Maybe you can convince me on this point too. I'm still listening.

To Rowal: point taken. I'm long winded. Bear with me and apologies in advance, since I'm sure I won't be able to always avoid the habit in the future.

To DSTM: again, I hope it's ok if we keep to discussing this topic. I feel like so far we're missing out on your potential input into the discussion. Your description of the input here as friendly and polite is certainly interesting, and very tempting to respond to, but how about we behave ourselves. Let's see if you can score any points on the topic at hand instead. I'm ready to listen to reasonable points that get made, and you're welcome to join in and make some if you want.

Edited by Billermo, 31 October 2008 - 06:48 AM.


#9 Grinler

Grinler

    Lawrence Abrams


  • Admin
  • 43,542 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:USA
  • Local time:09:32 PM

Posted 31 October 2008 - 06:47 AM

Everyone, though it is appreciated that you stick up for the site and our mods, calling someone an inappropriate name or being rude to any member is inappropriate to say the least. Regardless of your opinions on this subject, behaviour in that manner is not tolerated.

Billermo, the reason why many of us in the security community disregard the results of AV tests is because there is no structured test environment and methodology that people conduct these tests in. Therefore, different tests/comparitives will all have different results because they all use different methods. This does not allow for results that can be fully trusted.

Examples of items that may be different from test to test:
  • Are the samples lumped into a single directory, though that is not how they would appear in real world examples.
  • Are they installed on the computer as the infection normally would be.
  • Are all of the Anti-malware programs using the latest version and with the latest definitions? You will be surprised by how many comparisons are not using the latest software.
  • Are the computers specifications exactly the same for each anti-malware software test?
Another exampleis that in the last week or so an AV test, a shoddy one at that, was done with supposedly 800K malware samples. Some of the better, but newer, anti-malware programs did not do well on this test. Why? Because these newer programs are protecting you from new malware that is in the wild, not old ones that are no longer active. Therefore, these newer programs do a better job of protecting you from current malware, but fail in the comparison tests because the tests were based around inactive older malware. Now according to this report, you should not use the newer programs, that actually do a better job at protecting you from the latest malware. Does that make sense?

Do you see where I am going with this?


Therefore it is more important for you to go with what works best in a given situation. This type of information is typically found by either testing the software yourself, if your knowledgeable enough, or trusting what your peers have to say. On our forums we are completely unbiased and only offer programs that we trust. Therefore if multiple people who are knowledgeable in security mention a particular program, it is because we believe in it, not because we are promoting it.

Last but not least, Antivir is a good program :thumbsup:

#10 Billermo

Billermo
  • Topic Starter

  • Members
  • 110 posts
  • OFFLINE
  •  
  • Local time:08:32 AM

Posted 31 October 2008 - 06:59 AM

OK this is a lot more like the kind of reply I was hoping to see. And yes I see where you're going.

One thought hits me though, which is that even in science there aren't one set of rules for tests that are conducted. What they reply upon is vindication of experiment results by peer review.

According to the questions you raise, it at least seems possible, even inevitable, that proper tests would eventually be developed. I have to also wonder if we know for certain that some testing facilities aren't already addressing these concerns. The way to check the validity of the tests is to examine the methodology, like you say. I actually have seen tests out there where the methodology is posted, the virus samples list shared -- basically transparency.

I can appreciate that there are some poor tests being run. It doesn't mean that they all are though. And if there are some legitimate tests being run, then it would be wrong to dismiss those because some other facility has run a test that was poorly executed.

So I see where you're going, but I'm not quite with you yet. Which is not to say I think you're wrong, just that I'm not yet there.

Maybe if we focused on one particular facility -- this AV-comparatives that seems to be the most quoted by all the AV companies. Are there specific complaints about that one?

I think I'm coming around to agreeing that Antivir is solid, too.

Edited by Billermo, 31 October 2008 - 09:12 AM.


#11 Billermo

Billermo
  • Topic Starter

  • Members
  • 110 posts
  • OFFLINE
  •  
  • Local time:08:32 AM

Posted 01 November 2008 - 12:14 AM

Billermo, the reason why many of us in the security community disregard the results of AV tests is because there is no structured test environment and methodology that people conduct these tests in. Therefore, different tests/comparitives will all have different results because they all use different methods. This does not allow for results that can be fully trusted.


OK, I hear you saying that there is no structured test environment or methodology in these tests. However, the example I've mentioned is a group called AV Comparatives. This is the one that is most often cited in articles in the media and by the AV companies themselves on their websites and packaging. The tests done there do have a structured test environment and methodology:
http://www.av-comparatives.org/seiten/erge...methodology.pdf
If you are in the security community, then of course I know you must be aware of this group, and may have some other reason besides this particular one to not trust their tests, which I would be interested to hear. Or it's possible you were talking about other tests besides this one -- in which case your general statement doesn't seem quite fair at least toward this one facility. If you generalise about all tests and state as a reason for dismissing them that they don't have methodology -- but the leading one in fact does.. well I guess you can see where I'm going with this. ;-)


Examples of items that may be different from test to test:

  • Are the samples lumped into a single directory, though that is not how they would appear in real world examples.
  • Are they installed on the computer as the infection normally would be.
  • Are all of the Anti-malware programs using the latest version and with the latest definitions? You will be surprised by how many comparisons are not using the latest software.
  • Are the computers specifications exactly the same for each anti-malware software test?

This is very useful information. When I find tests, I'll check such details in their methodology. It is possible to also just send an email to that organization and they may reply with answers to those kinds of questions.

This type of information is typically found by either testing the software yourself, if your knowledgeable enough

Have you tested different AV programs up against one another under serious test conditions?

or trusting what your peers have to say. On our forums we are completely unbiased and only offer programs that we trust. Therefore if multiple people who are knowledgeable in security mention a particular program, it is because we believe in it, not because we are promoting it.

I've never doubted that people here weren't giving their honest opinions, or werer acting as agents for some brand of software. It seems to me that a cool, analytical, detail-oriented approach would be most suited to this topic, which aside from your post, I just haven't seen here. The kinds of replies I've often got here haven't exactly inspired trust, but doubt. But that's a whole other topic, one I have no interest in getting to the bottom of. I trust good evidence, basically.

Edited by Billermo, 01 November 2008 - 12:17 AM.


#12 Guest_fuzzywuzzy6_*

Guest_fuzzywuzzy6_*

  • Guests
  • OFFLINE
  •  

Posted 01 November 2008 - 01:33 AM

:thumbsup: I have been reading this thread with great interest, since i have been experimenting with free firewalls and am looking for an antiviral program. There are two questions that apply to those who are not sophisticated in matters of software or computer language:

(1) How well does the av program work with your other installed programs? These will vary a lot from person to person.

(2) How accessible is the information on how to use this application? Do you always have to go to the help section, leaving the page where the issue arose? Or can you get help easily on that page? Are the info links within the program easy to find? Also, how good are the on-line resources? Is it easy to find info in the self-help section on-line, or do you have to go to the forum for every little thing? If it is very time consuming to find info that is necessary for the novice or unsophisticated user, the antivrus program will not be popular with the general public. I tried one of the firewalls that was recommended here, and found it very difficult to use. Also way too many ads. Am now using another recommended firewall because the info is a lot more accessible and easier to understand. It seems to be working quite well with some of my other security applications.

Computer users are highly idiosyncratic. So are, evidently, computer applications. This site is very helpful for users who want to find a good application match on several levels. The most protective software program in the world will not help you if you do not find it particularly accessible. This site is a great place to get advice for all sorts of thorny problems. They do it for love here, not for money.

#13 DSTM

DSTM

    "Bleepin' Aussie Addict"


  • Members
  • 2,649 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:SYDNEY-AUSTRALIA

Posted 01 November 2008 - 01:50 AM

I have never had a free Antivirus, with an active guard, let any virus into my Computers.
However a Paid, highly rated Antivirus program, let antivirus XP 2008 into one Computer.

Edited by DSTM, 01 November 2008 - 03:20 PM.















#14 Billermo

Billermo
  • Topic Starter

  • Members
  • 110 posts
  • OFFLINE
  •  
  • Local time:11:32 AM

Posted 01 November 2008 - 02:11 AM

I can say based on my own experience that AVG doesn't give online support for their free AV -- if you ask a question at the site they will direct you to the forum. That said, I expect in most cases you would get good replies there. I haven't checked Antivir's. Nor the different free firewalls.

As for this forum, I do understand that the people who come to offer comments here do it for free (as I do). When it comes to offering advice, more than love is needed though, right? I'm just pushing back a bit here because I want to see if this advice is based on truth or on false notions about what is true. And like you said about users with idiosyncracies, well, I've worked with hardcore techies before, and I'd say they're just as prone to them as anyone. I may turn out to be totally wrong in what I suggest here, but if they know they're stuff, they should be able to back up what they say. Giving technical advice is the main purpose of this forum. The advice should be correct, and to be correct it needs to be based on things that are true. Verifying accuracy is a fair part of making sure the advice is good. I won't be surprised if the moderator/administrator comes back with a very impressive argument to support what he said, and if so, great -- it means people here can have that much more faith in advice given here. So everybody wins. If it turns out the position they take about test results is wrong -- that there really are good quality tests out there now, and they are choosing to ignore them based on false notions about them, or because they're not keeping up with developments, well then I would have done a service here by forcing people to re-examine a false idea that had become popular 'wisdom' here. The most unfortunate outcome would be if the question goes ignored and the facts unexamined.

Edited by Billermo, 01 November 2008 - 02:21 AM.


#15 DSTM

DSTM

    "Bleepin' Aussie Addict"


  • Members
  • 2,649 posts
  • OFFLINE
  •  
  • Gender:Male
  • Location:SYDNEY-AUSTRALIA
  • Local time:08:32 AM

Posted 01 November 2008 - 02:21 AM

Hi Billermo,I have read many many of your posts, on other Tech Forums, and get the impression,you know a lot more than your making out. You are never going to get consensus, on a Topic like this, and fail to see the need for it,Frankly.
When you start questioning Tech motives on the site,I think you are stepping over the line.


















0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users