Home      What is a Trojan Horse?      How we reviewed anti-trojans     About us

The 2004 Trojan Test Set

This year we've totally changed the way we measure the trojan detection capabilities of the various anti-trojan programs under review.

In 2002 and 2003  we tested anti-trojans against the very latest crop of advanced trojans.  This gave a good measure of the effectiveness of the anti-trojan products and  also a good indication of  how regularly the vendors were updating their signature databases.

The problem is that average users are not very likely to encounter very many of these new and advanced trojans. So this year we've adopted an new approach which much more closely matches the way folks are likely to encounter trojans in their everyday computer PC work.

One of the most common ways for users to encounter a trojan is through downloading files from P2P file sharing networks like KaZaa or eDonkey.  So we decided to go hunting for trojans in circulation on these networks.

I regret to say that we found lots. We also found viruses, worm, adware and spyware as well.  The statistics are scary with over 38% of the executable files we downloaded being infected with one or more malware products. Most of the infections were viruses but many files had multiple infections presumably as a result of being passed from infected user to infected user. One file has two viruses, one worm, two spyware programs and a trojan.  Ironically the file in question was a copy of McAfee anti-virus!

Now this is not to say that 38% of files on P2P networks are infected. Rather 38% of the files we downloaded were. The files we downloaded were all executable files or archives containing executables. These files were mostly illegal copies of commercial programs or cracks of commercial programs. Of all the different kinds of files on P2P networks these are the most likely to be infected.

To make the anti-trojan evaluation even more relevant to every day needs of users, we use removed all infected files that were picked up by Norton Anti-virus 2004.

Norton detected almost all the viruses and worms and a few spyware products as well. It didn't do so well detecting trojans finding only 57% of the files we knew were infected.

By excluding these Norton-identified files we were able to end with a test set of 41  trojans "in the wild" that were totally missed by Norton AV 2004. This is not an indictment of Norton - most AV scanners have at best, modest trojan detection performance.

Of the 41 trojans, there were a number of files infected with the exactly the same trojan and these were deleted though variants were left in.  A few of the files were also incomplete or the trojan was not working properly due to incorrect configuration of the trojan server.  Excluding these we ending up with 16 unique, fully working infected  programs. Several of the programs contained more than one trojan component.

The 16 trojans we ended up were a mean lot and a tough test for any anti-trojan product. After all, every single one had sailed through Norton totally undetected.

Some of the trojans were old friends such as Litmus, ProRat, Optix Pro and MiniOblivion that managed to avoid detection by sophisticated packing and stealthing techniques. Others were new products or variants of older products that were simply not in the signature files of some products.  Five of the trojans were contained within ZIP or RAR archives. Four contained active elements designed to pull down the user's defenses.

By the end of the tests we got to know each of these trojans intimately. We developed a begrudging respect for the ingenuity of their authors and a realistic appreciation of the difficulty of detecting them.

Some of the social engineering aspects were clever too.  For example quite a few of the trojans were embedded in cracked versions of commercial programs and the cracked versions worked just fine. Any unsuspecting user who installed the program would have been delighted  that had got a commercial program for free without ever realizing that their PC was totally compromised.

One particular example made us smile. It was a serial number for Norton Anti Virus. It had been embedded in a self unpacking zip archive with the trojan embedded in  the unzipping executable. When you double clicked the archive and unpacked it, all you saw was a single text file. When you opened the text file, there was the Norton serial number. To the average user it would all look innocuous - what could be more harmless than a text file?  Of course by that stage the users' computer was already infected.

Test Methodology

This year we used a virtual PC created using VMWare to test our anti-trojan products. This allowed us safely execute each trojan and able to quickly restore the machine to a clean standard state after each test.

To test each anti-trojan we installed it on a virtual machine. We then updated the anti-trojan signature files via the web and saved the machine state. We then used the anti-trojan to scan a folder containing the 16 known trojans and noted what products were detected.

After scanning the trojan folder, we restored the virtual machine to the saved state. We then started up the anti-trojan memory monitor and executed each trojan. After executing each individual trojan the virtual machine was restored back to the standard saved state using the "Revert" feature within VMWare.

This methodology ensured each anti-trojan was tested under the same conditions. No product was favored one way or another.

Test Results

Key:  S-Detected by scanner   M-Detected by monitor  SM-Detected by both

The results are largely consistent with the findings from previous years with TDS-3 and Trojan Hunter topping the results  This year though, the newcomer Ewido came in a very close third, a very creditable performance.

Both The Cleaner and a²  performed well though not quite as well as the top products.  A squared is due for a major upgrade to version 2 "soon" and sounds highly promising if it delivers on its promises.

Tauscan disappointed for the second year in a row and now drops off our recommended list. Its main weakness is the inability to detect polymorphic trojans. Similarly with PestPatrol which only detected two trojans. PestPatrol is a fine general purpose malware detector but it is out of its class when pitted against the 16 lethal modern trojans used in this test.

BOClean had declined in detection performance over the last three years which is a shame as it was once a class leading monitor. Given that BoClean also has no on-demand scanner, it too has been dropped from our recommended products list.

Advertisements:

The Best Backup Software: 18 backup programs reviewed and rated but only one get "Editor's Choice"

Inkjet Printer cartridges: The best places to buy cheap inkjet cartridges. We looked at 47 seven sites but could only recommend eleven.

Anti-trojan Software Reviews Home Page