WLO

Hi wildman,
I do not understand your question. I try anyway:
WCL (checkmark) says: "The standard is (in principle) a real-world, industry accepted
test"; this summarizes it I think. The tester decides what will be tested; but if the tester introduces e.g. some too much exotic testing (like testing against unwanted stuff or a absolutly wrong approach) he will maybe be forced in some way to change his testing, as it will maybe not be accepted by the industry.
WCL for example does now also test intensivly against spyware.
I do not know if this answers your question, but I tried to

 

Every product can be tested by VB but this does not qualify the persons that make it as trustworthy. Do you know why they don't get the wildlist or if they did already ask for? Are you in touch with the guys from UNA or why are you so concerned about this? You're not even testing UNA.

Nobody is perfect and there is always room for improvements. Anyway you escaped all my questions in #20 which I was really interested in.

 

I know things, but I do not tell you the sources. I do not test UNA publicly, but they get results. I do not test UNA as I think it would not be that fair atm. VB tests them against a WildCore collection that other companies get and e.g. UNA not, and that is not fair. It would be more fair if none company gets it. Anyway UNA is just an example, who knows how many more companies score bad in ITW tests due this....; It is all kept as a good secret around the WildCore and their decisions.

I did not answer to your post #20, because it seems you do not want to understand it anyway. To answer it simply: if they get from 12 companies the samples and not from the 13th company, they just have to ask the 13th company. I am not a collections shareman, the collections are shared a level higher on me, so they have to discuss with the other companies not with me. I am quite sure you are from a company which does not submit to us samples, and you are sad because you would like to get everything from other comapnies by sending nothing to other companies. That point in the conditions was not introduced by me, but by the companies. If you think a bit more about it, you will see that it is a fair rule. If your company is considered not trustworthy or you feel that your company is not handled fair in our tests, feel free to tell me it, I am not obligated to test you if you anyway think that the test is not fair, even if you know exactly that you have the samples, got the samples, and that it scores what it scores for other reasons, but for sure not due a lack of samples in your lab.
If you prefer that in future no AV company gets any samples from us, propose it in the next mailing-list discussion, instead of remaining anonymous and speaking behind shoulders.

 

If I am understanding this, which I am not sure that I am, there is really no set standard as to what will or will not be tested and it is left up to the whim of the tester. If this is the case, then I don't think that is such a good thing. How is one to know if the list is reliable or not?

What happens when a new malady is reported to a virus protection company?

Also what is the most prevalent type of problems that are being reported?

I hope this is some what clear, as I am very uneducated in this area!

Thanks
Wildman

 

Hi wildman,

some answers you will partially find in the FAQ on the website of the wildlist.
the rest maybe in the paper of bontchev.
the answers to the rest of your questions may remain unanswered (by me, because I do not know the answers).
p.s. i think i am still unable to understand your posts completly (my english is not the best).

 

Now you got exactly the point and I totally agree with you! This is why we got offtopic a little bit.
"So why do you test AV companies that do NOT get your test set while others do" Isn't this the exact problem for your test as well?
I might be wrong but I think you should first look at your own criteria and be fair before you start to complain about others. I think you're doing mistakes on your own but complain about others doing the exact same thing.

In my opinion you should give the samples either to all companies you test or to nobody (which will not help the antivirus industry). Otherwise you are just not objective.

If you're telling that this is up to the companies to ask for the samples than simply give it out to noboy but not only a few selected once (which again will not help the industry itself).

I hope you now understand why I got a bit offtopic. I just wanted to show you that you complain about things that you are practicing yourself. I hope you will agree on this.

 

I would not complain about others if I would not know that how I test and how we proceed is fair. That's why I 'attacked' you a bit, because IMO you think we do not handle fair while we in reality do test as fair as it is possible for us. (no problem, anyone has different opinions)
If you think something is unfair, tell it to the companies which do not share samples. I can only repeat me.
btw, i do not agree I could tell you why your concerns are wrong if I would know who you are, but not here in the forum (to save e.g. the image of your company).

 

I think these are some valid points and I agree. While this test bumps the detection rate of particular AVs(because of given samples) other AVs are left in the dust(lack of samples). IMHO, It’s not fair, it’s unethical and misleading. Give it to all or don’t give it to anybody!

This was my short opinion and I won’t go any deeper about this issue, since I respect IBK and his work.


tD

 

Actually not an opinion it's a fact. You give your samples to some vendors but not to all. The one that don't get it are not treated equally that's it.

Think about the following situation: You borrow mony from the bank and in the contract it's written you have to pay it back in one year. You borrow this money to someone else one day later which is supposed to give it back after 11 months and he does not. After the one year the bank comes to you but not to the guy you borrowed the money. Can you see the similarities?

You're doing the tests so you have to treat each tested vendor equally in any case but you do not. Therefore you're not objective and this is not an opinion this is a fact!

If you would have performed your tests in industrys rather than your own interests you whould have created two different documents after you made the tests. One lists the vendors that get the samples and the other one just shows the vendors who do not get the samples. But then again this will not help the industry itself. So the only solution is to treat each vendor equally and share the same samples.

I am pretty sure it doesn't matter at all who I am. I told the facts above that you give precedence to some companies while others suffer which is not fair and not objective because you're not even telling about this in your tests.

I also think that these are things that can and should be discussed in the public rather than in a private discussion. I hope this fact will open the eyes from other people who read your tests. But it would be even better if you would change your procedures and treat the vendors equally which was actually the intention of all those posts.

 

Thank you Technodrome for some support. I hope others jump in as well and at least post their short opinion and why they think so.

Andreas:
Sorry if I had choosen some rough words. Please take nothing personally as i didn't want to attack you in person but your procedures and how you treat vendors.

 

@Technodrome: seems like 'new' was able to smoke in your eyes, because what he says is wrong. Now you think that companies score what they score because they do not get the samples, well, that's a wrong assumption. AV companies know this and trying to explain it to users is hard as I can not proof it to them. But AV companies with a bit of clue know that it is fair. If a AV company thinks really that it is not fair, than the AV company has to take this nose itself. They were the ones that prohibited me to send out samples to companies which do not want to share their samples with other companies.
I try with a short example:
Company A says test is unfair because they want to get all samples from av-comparatives. av-compparatives was forced from the av-companies to introduce the rule (we wanted to give all samples to any company; to share to no company was rejected by all companies). The companies accepted the rules, as nearly all submit their samples to us. Company A does not want to share his samples with other companies (unfair, isn't it?) but would like to force av-comparatives to send to company A all samples from the other companies. Company A has nothing to complain (not just because they already have most of the samples in their lab and enough to work on) because it has many possibilities open:
a) submit their collection too and they will get all the samples they miss and from other companies (if they do not want to submit to us, they just need to share with other companies with which they do not share atm, and they will get from them the samples)
b) asking us to be not tested
c) adding the xxxxx samples they got from us in past and that are still not detected, if they would work on the files they got already from us, they would be easily reach the next level.

If is not av-comparatives that is acting unfair, if someone is acting unfair, than it is the av-companies which do not share samples (which will increase the security around the world) - same in the WLO case, as the decisions are also in that case made by peoples from av-companies.

 

Let me see if I understand this. IBK, the malware set used for av-compparatives tests are not some personal collection of yours but a combination of all the sets submitted by those companies you test. The rule for any company to get the complete test set from you is that they must submit their set to you. If they do submit then they can get the full test set. If they don't submit they are not allowed to get the test set.

An alternative to submitting to you is that they can contact some other company directly and work out a deal to get their submitted testset privately.

If that is the case, then that does seem fair.

 

Okay then you shouldn't give the samples to anybody and in this case you have fair test conditions. If not you should mention in your report which copanies didn't get some or all samples (and how many). In this case it would be much clearer.

As far as I know some companies exchange samples with some other companies they trust but not everybody. Others may have copmany policies that allows them to only send samples to trustworthy other companies but not testers (which is a good thing in my opinion). If it would be different then everybody would claim to be a tester and ask the companies for samples and they should give it?

Why don't you change your ruls and say: "If you're sharing the samples with me I will distribute it to every product that is tested in my tests".

Would be a brilliant idea - or not?

 

I am now posting questions about this subject on one of the forums that is supposed to specialize in this area. I am still confused, and am getting the impression that these test could be easily rigged to favor one company over an other. Mind you, I am making no allegations, as I am no where near qualified to do so, however I would now like to hear from any one who has knowledge in this area. I agree nothing should be behind closed closed doors as far as this is concerned, and would like to see those who are knowledgeable bring this into the light of scrutiny. I also would like any one who knows to explain in simple English just exactly what comprises a test, and how that is determined, and who determines it.

Thanks
Wildman

 

I agree with ---.

In any case you have to make sure that every or non tested product gets the samples. Don't leave this up to companies as according to some exchange problems, company policies etc. you can not proove it will work. To make an objective test and I suppose this is what you want to do you have to give equal chances to any tested product.

 

The base collection of av-comparatives (which is still the biggest part) is now over 2-3 years old and all companies do already detect that (so no need to send that again). Yes, the rest is now coming mainly from the av-companies, the little rest what does not arrive from av-companies is submitted to them in various ways (like mails, etc., i am usually as BCC so i can see if someone does not get it). the rest is as you say, thats why it is fair.

@new: the av-companies already took their decisions and we agreed together with them about the conditions as they are fair. If now your company thinks it is unfair or due any other problem, you can ask to do not get tested in future. If your company can not be trusted with samples, I can not change it.
But good to read that you agree that only companies which get the WildCore should be included in ITW tests.

The tested products have equal chances.

 

It's up to you to change that. Of course the companies that get the samples and have good results will not complain about this. They would be stupid to do so.

Think about marketing driven companies (Symantec is listed on the stock market). They might say something like "who cares about ethics as long as we get good results and make more money".

I don't think it's about a company that can not be trusted. I think that some companies do simply not exchange with testers but only with some other AV companies.

As you are the tester everything is up to you to provide equal conditions not up to the companies.

 

No smoke in my eyes IBK. The thing that is puzzling me is why an av company would share any samples with you or any tester? Does this sound ethical?

I mean you test Avs so the reader could see how good his or her antivirus is. If they all (antivirus companies) get hands on your samples and add it before the test, wouldn’t they all have a perfect score.?

Correct me if I am wrong but as far as I understand the company A (against sharing but wants to be tested) stands no chance against the company B (sharing) in your test. IMO this concept is not good. Obtaining samples from their competitor is rather subjective (will they get all samples used in your test? Is competitor willing to share with them ,etc).

I think the best thing for us (readers) would be if you didn’t share your samples at all. I as a reader want to see an av improvement, which isn’t happening over the night.


Lastly, I don’t have any doubt that av companies (with good score) would use your test for advertisement purposes. For example, I received an email from one av company mentioning your high score test(its business and its understandable). Sharing virus collection with you would bump their scores and take them to the top but at the same time it would hurt another non-sharing company. Which is pretty much what I don’t like….



tD

 

Sure. Read e.g. the condition of the Check-Mark certification: they ask for the whole sample-sets of the companies in order that they test it. Where do you think other testing institutions get their samples? By all av companies and other sources.

That's what I tought too but seems not . Peoples must understand that the av companies already have the samples. AV-Comparatives does not send samples out in first place because they do not have those samples (they have them already), but in order that they can check that the results are not invented and the test performed correctly. In a perfect world where av companies have lot of employes and where av companies do also care about zoo samples and add correctly every malware they have in their lab, they would score all 99,99%, but as av-companies can not be perfect, the reality is another.

That's why I take care that only those companies which have the samples get tested by me, in order that those that are tested have the same chances. i get the same samples as they get, as i am on the same distribution lists like they are.

When I proposed this at the begin all the companies were unhappy too. They rant in any case, as nearly ALL get additionally all misses from me, I see no problem or need to discuss. On the other hand, I see the need that the WildCore is discussed, as that is really not shared to all companies against it is tested with, but as they keep everything secret, no one from outside has a chance to know this and to critisize it. (next time when i criticise the WLO i will also post as guest, so the topic can not be driven in other direction ).

You may wonder, but I never got any sample from Kaspersky and they are anyway on the top. Why? Because it does in reality not affect the results if the samples are coming from all possible sources and the test-set is nearly complete. If the non-sharing companies (which are not as you may think at the end of the list) think it hurts them, they are free to do not participate. The tested products are already good scanners, those which are higher than 85% are really good, but as you see only the tops, you may maybe think the those at the end are bad :/.

 

We all know that these kinds of institutions are pay per certificate. You pay, you’ll get certified, no pay no certificate. Can we call them independent? AV companies interest is to get certified so they can advertise. Following rules from Check-Mark or ICSA Labs is no problem for them (mutual interest). They wouldn’t really provide anyone with samples. Now would they? Does this mean they are not independent (as they claim) at all? After all they are depending on av companies sample and not on their own.

You on the other hand work independently (not getting paid) and without their influence (or not?) and you should be sending all missed sample to all companies no matter what. Plus, you are the one who should set rules and not them.

Getting virus samples is a very hard and complicated job. It takes a lot time and people to collect a good and clean collection. That’s the thing about testing. AV tester gotta have own virus collection (independently from av companies) if they want to earn peoples trust and respect. That’s why Check-Mark, ICSA Labs or even WLO are not the good av decision-making source.

I hope you know what I mean…

I forgot to mention. I respect you for publicly discussing all this…


tD

 

It seems to me that a good AV company should be working hard everyday adding detection like KAV and some other AVs do and not rely upon some independent tester doing a test a few times a year to provide samples they missed.

It appears kind of odd that some AVs that score poorly on some independent
tests should need to have the samples sent to them instead of obtaining the samples in a timely fashion on their own. I suppose they could lower their cost of doing business by getting those collections instead of incurring the everyday expense of keeping current like KAV and others do.

 

Hi td,

I understand what you mean. Well, they are paid very much for a certification which is IMO not that hard to get (and to test). How much they are paid is a secret, in case I know it I would not tell it publicly .
Now to av-comparatives: we do not get paid. if we get some donations at the end of the year after the tests are done, the donations are invested in computers etc., so we do not make any profit in providing the tests; it is pure devotion to provide independent results and to do tests as good as we can and as independent as possible (otherwise efforts would be senseless).
Anyway, even if other testing institutions get paid very much, their results are still independent. I think tests done by journalists (which are maybe working part-time for an av company, who knows..) or other subjective articles in magazines about av's are more probably to be flawed / not independent, as they could be a hidden publicity for some company (or to make look bad a company).
Not all av companies are trustable, btw. Some condition is also to prevent some things, I could to tell some stories, but it is better if I do not .
p.s.: stan999 is right - he got the point. I would add that some av company should respect the results of some other good products and not try to discredit tests if they product does not score well in tests compared to other products.
p.p.s.: i am now finishing the retrospective test, so i am busy...

 

Thanks all, I am starting to get a bit more clear on this entire subject, but still have quiet a few questions that still go unanswered.

I for a long time have thought that these tests were being used to either promote or discredit virus protection programs. One only needs to see how often AVG has been accused of not being that good at detection. Now I get the impression that this may be as I thought, not that accurate of a statement. I still would like to know just exactly what these tests are looking for, and how that determination is derived.

Thanks
Wildman

 

Of course he is. The point is valid and I am not denying it. But KAV and Mcafee are the only 2 antiviruses which overall score good in any other tests out there. I might be wrong though...


Looking forward to see your new retrospective test.


tD

 

You are making some valid points here but this argument is about should tester give the samples to one av company and not to another. Please read the whole thread...


tD