confused about these AV/Anti-Trojan tests

* * * THIS SUBJECT IS NOT ABOUT ANTI-VIRUS * * *

Hi, perhaps anyone here has read this:

http://www.virus.gr/english/fullxml/default.asp?id=67&mnu=67

I don't know what i have to think about this,
according to these test TDS-3 does not perform well versus a lot of
Virusscanners on TROJAN scanning (not virusses of course) ??

Is this true ? or are these results fake? has anyone payed for the results?

I don't know if the tester(s) are objective,
i'd like to know what you all think of this!

Tuatara

 

I dunno if the site/testers are worth their salt or not, but you ar eincorrect in saying that it's a score of scanning for trojans..

which means all that stuff is in there and trojans are only one small segment of what was tested.

 

i'd say that the poor results of many trojan scanners are a result of the test maker not responding to criticism/suggestions..many people(including me) have told anthony to include only trojan servers in the test. obviously this has not happened, because

by default tds( and many other anti trojans ) only detects the trojan servers, where as an antivirus, like kaspersky detects almost everything in a trojan package, including the not dangerous clients, editservers, etx.
the detection of clients/editservers is optional in tds, tick one box adn you'll see tds' results go up

some anti trojans even detect the help files of trojans as a virus/pest

let's say i scan a set of 1000 unique trojan servers with tds and kaspersky, both will detect close to 100% of them. now if i scan the setup zip files of 1000 unique trojans with both at defeault settings: both will still detect 1000 servers, but kaspersky will detect some 2500 additional files which tds doensn't detect if detection of clients etc is not enabled

i'm 100% sure on the integrity of the tester, Antony cannot be bought/bribed etc. he does take av testing seriously..

the reason of these surprising results is what i described above

 

TDS also performed pretty bad in respect of trojans/backdoors.

But I doubt that this test adequately reflects the performance of TDS under real-world conditions.

 

@illukka Separating trojan clients from trojan servers and creating a proper test archive is an awful lot of work. Most testers prefer to not work that hard but to force AV/AT developers to include servers into their signature database ;-)

 

In addition, I wonder how TDS was tested /w ten thousands of samples.

The mem scanner was most certainly not tested. Moreover, AFAIK TDS does not allow to automatically delete any malware samples found.

Therefore, I guess that the tester relied on the TDS log file? Unfortunately, the log file is not accurate as regards the no. of detected samples (due to double entries for packed samples etc.).

Would be quite interesting to know how the tester solved this problem.

 

most av's are used in corporate environments where detection of clients/editors is quite useful..
companies like kaspersky have a policy to detect everything

 

For reasons known and described one can not compare specific AT with AV/AT scanners, especially not with such a mixed database; it would be the same with specific AV in a test of scripts, worms, trojans, etc.
Or adware/spyware etc.
One should really separe the kind of scanners.

For your remark about TDS not removing automatically any alarms, no, fortunately we operators are kept in the driver's seat ourselves for that, so we investigate more intelligent ourselves and decide after that, so avoiding to delete unnecessary files or trying to get them back from quarantine, etc.
TDS is one of the few able to find malware while it has not infected a system yet by blocking execution completely.

I don't see any reason to presume memory space would not have been tested. Every bit and byte and every function was tested thoroughly before released to public; i know as i was part of the beta testers team myself, and so were quite an amazing amount of others.

In the test description all files were packed in other ways then zip, rar, etc. So that would prevent double counting from the original file and the archive containing the file and the generic and normal scanning.
Carefully configuration of the scan to avoid double scanning does a lot too (logical drives, hard drives, drive name C:\ for example would include 3 times the same scan)

 

Hi Detox: i refered only to the % Trojans /backdoors .

illukka: that seem to me a logical explanation,
and i know that virusscanners, alert for every file that has something to do
with a trojan.
and as i made clear in my post, i didn't know the testers.

The problem is comparing products, was quitte easy in the early days
(said the oldtimer) , because then a firewall was a firewall and a
AV was a AV.
Now the products, have a lot of add-on's like Trojan scanner or Spyware Scanner. etc.

So is Kaspersky (just an example) an AV or... AV + Trojan scanner?
The way that the results are presented in the details (pdf) file, suggest wrong things here, (near my personal opinion) => that you don't need a Trojan Scanner if you have a (certain) AV product.

illukka wrote:

The problem with this report is, that a reader can read this in 2 ways.
1) The way you explained it in the quota here above.
2) He tested with xxx Trojan Servers and the number given is the number the
scanner found.

Thanks for the replies guys, it is good the have some additional info on this.

 

Sorry Jooske, we were typing at the same moment,
as you can see i agree ...

you certainly can do a lot more with TDS-3 then "just" Trojan Scanning

 

KAV is an AV/AT scanner,
TDS is a trojan scanner, detecting ALSO lots of worms, droppers, keyloggers, dialers, adware, spyware, trojan JS script, trojan downloaders, rootkits, etc etc etc
I must leave it to the specialists to add to that list.

There are several former discussions with lots of valid points and details to look into as well. Things depend on settings, are the samples in the wild or zoo samples, etc. and lots more questions.
And afterwards or before it would be nice if all samples were available to all developers to be tested so all get an equal chance in detecting.

 

@Jooske

"I don't see any reason to presume memory space would not have been tested. Every bit and byte and every function was tested thoroughly before released to public; i know as i was part of the beta testers team myself, and so were quite an amazing amount of others."

Whenever I say something re TDS you presume an attack. I did NOT say that DCS did not test the TDS mem scanner.

I said that I presume that virus.gr did not test the TDS mem scanner while performing the test. It would take several weeks to execute 70.000 malware samples in order to test the mem scanner.

As regards auto-delete: I do not say that DCS should implement this feature. Inexperienced users may enable auto-delete and delete important system files. However, the lack of auto-delete makes it very difficult to test TDS.

Last but not least, I believe that AT and AV tests should NOT be separated as regards the detection of trojans. A user wants to know whether the trojan detection rate of an AT is comparable to the trojan detection rate of an AV.

@Illukka

Correct. However, non-corporate users are not interested in the client detection rate. Therefore, the tester should distinguish between server detection rate and client detection rate. But AFAIK there is no tester with a huge test archive who has properly separated servers from clients. Therefore, no tester can provide the user with this important information.

 

You meant the memory space in the test, not by DCS.
That explains.

Let me put it different for the separation between AV/AT and AT or AV:
they should be put in separate columns, and not all together in one long list of %, as explained in first answer in this thread of course with a combined database you get results which can not be compared properly.
Throwing it all in one list is impossible and gives wrong ideas.
One should per class of scanners only take the amount they're supposed to detect and what they might detect extra.
Then you can make valid comparisions per class, and they can score over 100%

 

i do believe that he did not test with trojans servers, i think he used those trojan packages you can download from the net

to demonstrate this i quickly made a set, scanned it with 3 different scanners( tds-3 upd 3.9-04, trojan hunter upd 3.9-04 and kav pp 4.5 upd a minute ago)

tds 51 falarms, 2 heuristics(Positive identification <Adv> (in archive): Binded file (Binder.GoBind) File: editor.exe (In c:\roijia\acidreign.zip)and
Positive identification <Adv> (in archive): Possible keylogger
File: client.exe (In c:\roijia\uboot2abeta.zip)

th 3.9 detected 73 files, some clients(20) included. this is a little tricky situation coz sometimes the server is embedded in the editor and th seems to detect some of them too

kav detected 98 files

the archive contained one unique [recompiled zoo]trojan that was undetected by all. not a very dangerous situation because that trojna exists only on my c drive

every scanner detected the files they really need to detect, the servers!

then i enabled scan for clients editservers in tds's scan control: 137 alarms!

need more proof?

 

Sorry for the misunderstanding, but i don't need proof, because i know
that Kaspersky NOD32 and other AV's will give more alerts on one (Single) Trojan Package ..
i only want to know how the tests were done...

Yes, i know this was not for me, but ...
For the record, this is not an attack, only want to have more info on the subject.

With all that is said in this thread about Trojan Servers and files that go along with the packages of it, will certainly make a diff. on how many 'hits' you have.

After reading all this, it was perhaps better that i had sent this question to the testers:

What is a 100% score in Trojans /backdoors,
is it that All files in all Trojan Packages have to be found (readme's etc )

It would be nice, if there was a 'Anti Trojan' test,
which would compare Trojan Scanners AND AV's etc. in a manner that
you can read the which prog is the best Anti Trojan.

And not which one give you the most alerts on one (1) Trojan (Package).

Personally i use TDS-3 for on demand, BOCLEAN resident, NOD32 resident and on demand, Kaspersky 4.5 on demand-only (and other tools of course)

But to be honest, i would like to see from time to time (by tests etc.), that my choices (of these products), are not out of date yet.
It is always possible that another one will be the best in the future.

(TDS-4 or ....?)

 

Thanks for your kind words illukka. By the way, i have to admit that you are speaking the truth about trojans/backdoors detection rates. I wish i could manually seperate servers\editors\clients but i don't have the time to do so. Thus, i have to include all different files detected by each av used for the making of the virus database of the test, which is not 100% flawless as you describe above. I wish av started detecting each of those files as backdoor.xxx.srv\cl\edt so that my work could get better. But you know and i know this ain't gonna happen, due to (among others) sales issues.

Anyway, i am open to suggestions, and that is why i try to discuss with any openminded person all arounfd the web, no matter what his thoughts of my tests are.

I'd like to thank you all in this forum for your critisism and thoughts, bad or good.

Regards
Antony (a.k.a. VirusP)

 

@Anthony

1.
Server vs Clients

Illukka is correct that corporate users are also interested in the detection of non-dangerous clients. However, it is important to separate the dangerous and the non-dangerous files in order to correctly assess the strengths and weaknesses of a scanner (which should be the purpose of a test).

AFAIK there is no tester with a big test archive (e.g., Clementi or Marx) who has separated servers from clients.

I wonder whether it is more important to create/maintain another super big test archive or to create a smaller, high-quality archive.

Moreover, a tester should anticipate future developments: if the next generation of AVs/ATs features memory scanning or even behaviour-based detection mechanisms which rely upon the execution of a malware sample your entire malware archive will get more or less useless as regards the detection of trojans. This is because the detection of trojans may not be based on a simple file scan anymore and, therefore, the tester may be required to execute the samples (which is not possible with ten thousand malware samples or more).

2.
I would be interested to know how TDS-3 was tested: Did you figure out a way to force TDS-3 deleting any malware samples detected? Or did you simply look at the scan log?

 

@Antony: I apologize for misspelling your name.

 

I just looked at the log file of TDS to calculate the detected samples.