By Joel Snyder
Network World, 12/20/04
Original Article on Network World Web Site
For the top of the heap of spam products, it's not what is good or bad that sets them apart. It's more a matter of what's different.
For example, if an anti-spam product doesn't allow for SNMP-based monitoring, you will only care if you're already using SNMP.
Our short list, based on the spam catch tests, included three services (Postini, Advascan and Mycom), four appliances (BorderWare, CipherTrust, Barracuda and Messaging Architects), three software packages tested on Unix (from Sophos, Proofpoint and Cloudmark) and two tested on Windows (Symantec and MailFrontier). We let the vendors choose the platform where more than one was supported.
To distinguish between the products, we looked at four key areas: spam-oriented features, per-user features, anti-virus and policy-based filtering, and logging and management.
Let's start with spam
The most important feature in an anti-spam system is how well it catches spam. All of our finalists turned in outstanding false-positive and false-negative scores, but there is considerable variation in how each product lets IT control the spam catch process. Most products offer a cocktail of techniques to catch spam.
The term "cocktail" is used by anti-spam vendors to explain how they make the go/no-go decision on spam. Early spam products had only one technique, such as searching for words in headers of message bodies, or a set of techniques that each could torpedo a message as spam. Modern products mix the results from multiple tests and analyses, combining and weighting them to come up with a final answer for each message. As the SpamAssassin team puts it when describing their anti-spam cocktail, "While any of these tests might by themselves mis-identify a message, their combined score is terribly difficult to fool." To implement the cocktail, each message runs through multiple filters or tests, and receives a set of scores. When enough tests agree (or when a single test gets a high enough score), the message gets its verdict: spam or not spam.
Many vendors sent elaborate white papers explaining how their spam cocktail was mixed to be superior to the competition. In our evaluation, we decided to not go down the path of evaluating the components of the cocktail. The proof of what works well (and what doesn't) comes out of the statistics on false positives and false negatives. In this market, the strategies each vendor uses to classify spam are in rapid flux as they search for better ways to outfox the spammers.
In our tests, products that let one test dominate the score - have only one test - tend to have a high false-positive rate. For example, just having the word Viagra in the subject line of a message does not make it spam. But having Viagra in the subject, in the body two or three times, a Web site URL of an online pharmacy and having the message come from the IP address of a suspected spammer all add up to the message being spam.
If you want to see the rules used to match spam and edit them, then Sophos' PureMessage and Messaging Architects' GWGuardian are your best choices. Both let you dive in and touch every aspect of the spam matching. This is a mixed blessing.
Corporate managers are moving away from tuning systems at this level because it's really not important. If the spam engine is doing its job properly, you don't have to look deep into the innards. However, there will always be exceptions. Sometimes the mail flow at a company can confound the spam engine, and this level of detail will be required.
A more likely requirement will be for coarse control over the factors that go into the spam scoring. Products we looked at range from virtually untouchable (Advascan and Symantec's Brightmail) to the relative openness of CipherTrust's IronMail and Proofpoint's Protection Server.
One critical factor is the ability to balance how well DNS features are incorporated into the spam score. With a notoriously high rate of false positives, DNS blacklists and DNS reverse lookups are dangerous to use in a go/no-go system. However, using DNS features as a component of the larger picture is a great way to filter out spam before it hits the device. BorderWare, Sophos, Proofpoint, CipherTrust, and even service Mycom let you pick which lists to look at, and what weight to give them. Other vendors, such as Postini and Symantec, maintain their own weighted DNS blacklists and whitelists to eliminate false positives that looking at any one list will cause. The ability to adjust these features is critical. For example, service vendor Sublimemail could not turn off DNS features built into its service, which increased its false-positive rate by a factor of 20.
Power to the users
No anti-spam product will have zero false positives. As we discovered in our tests, the better you are at catching spam, the worse your false-positive rate (and vice versa). The problem becomes how to deal with false positives that inevitably happen.
Vendors have taken three approaches. A popular one is to assume that false positives don't exist and to make a few pieces of mail vanish every now and then. These vendors didn't make our final cut.
Another strategy is to tag-and-deliver mail rather than delete it. With tag-and-deliver, some or all of the spam is actually passed onto the corporate mail server, but tagged in such a way that users don't see it unless they specifically look for it.
Tag-and-deliver has a huge problem, though: the volume of spam is so high it dominates Internet message flow. In our test, about 75% of the mail we received was spam. With tag-and-deliver, you would be storing, backing up, indexing and archiving four times the number of messages you really want.
Most products can distinguish between certain spam and mail they think is probably spam. Certain spam can be discarded, or even rejected before it is accepted, while mail with a more uncertain score can be sent to the quarantined, or tagged and sent for a "just in case" review by the user. The only product that doesn't separate spam and maybe-spam is GWGuardian. All other vendors offer the opportunity to separate at least two levels of spam with different actions (Postini doesn't let you tune the thresholds, but every other company does).
The third alternative is per-user quarantines. When a message is identified as spam, it is quarantined instead of delivered. Unlike a normal mailbox, quarantines clean themselves out regularly, and usually don't have to be built on the same kind of highly reliable infrastructure and high-performance servers that corporate mail servers require.
All the products in the top 12 have a quarantine, although it's less common when you consider the entire anti-spam market. By giving each user power over his own questionable spam, and by giving network managers the option to delete the most egregious and obvious unwanted mail, anti-spam products strike a balance between performance, user frustration and wasted effort, and the inevitable false positives.
Not all quarantines are created equal. There are some dark corners, especially with authentication. For example, the Advascan and Mycom services can't use your corporate Lightweight Directory Access Protocol (LDAP) or RADIUS authentication database, which means every user will have to maintain a separate password for his spam quarantine. CipherTrust's quarantine doesn't have any authentication at all - a user clicks on a URL via e-mail, and this acts as his authentication. We also ran into severe design limitations with Barracuda's LDAP authentication and Messaging Architect's SMTP-based authentication. The lesson learned was to dive into the details if you want to use a quarantine, because there are many deal-breakers out there.
We also considered per-user and per-group settings and user control over these settings. While many network managers might not want to let end users play with their spam settings, the argument in favor of empowering them is strong. When users are in control, they are happier, and having some black box filter their e-mail without a way for them to control it doesn't go over well. Several products put an enormous amount of control (perhaps too much) in the hands of the users.
In the top 12 products, we found 12 different group, user and customization strategies. The most flexible were from MailFrontier, Messaging Architects, Mycom and Postini. Each of these has group-level and user-level settings, and gives the network manager the opportunity to expose those settings to users (if desired). If you want to give users control over their own settings, BorderWare, Sophos and Barracuda offer partial or full control. Symantec, CipherTrust and Cloudmark don't really believe in defining per-user settings, while Sophos, Proofpoint and Cloudmark don't believe in per-group or per-domain settings. Cloudmark doesn't believe in any distinction between users - the Zen-like simplicity of its interface allows for only one set of spam settings for the entire server.
What's your policy?
Combining anti-spam tools with anti-virus and policy-based mail controls is a logical evolution and one that 11 out of our top 12 embrace with vigor. Cloudmark alone has turned away from the all-in-one system and focuses entirely on spam filtering. This makes Cloudmark Authority an ideal component for network managers who want to build and control their own mail infrastructure, but it is less useful for someone who wants a fully integrated system.
Looking closely at policy-based controls in each product separates those vendors that have thought about the problem from the "me too" crowd. For most vendors, anti-spam, anti-virus and policy-based controls are silos of completely disparate tools that don't talk to each other and can't affect each other. When the choice was "spam or not" and "virus or not," this approach might have been good enough. But the majority of the products we tested haven't revisited their architecture and tried to create an integrated and simplified approach. For example, we saw products that had separate configuration tools for anti-virus, anti-spam and policy filtering, each with slightly different possible actions. No tool was able to include the results of any other tool, such as "delete this message if it's both spam and virus-infected, but quarantine it otherwise." This is especially true for situations in which a vendor has chosen to act as a value-added reseller and integrate products from other anti-spam and anti-virus vendors.
The clear king of e-mail policy is Sophos' PureMessage. Beautifully integrated with Sophos' virus-scanning tools, PureMessage lets you construct any policy you want. For example, if you want to completely drop some of your incoming virus-infected e-mail but try to clean or quarantine others, you can do it. No other product comes close to this functionality.
The other two very powerful policy products (Messaging Architects GWGuardian and Symantec) are built on similar technology: the Sieve e-mail scripting language. However, while PureMessage is Sieve-driven all the time, Messaging Architects and Symantec use Sieve only at particular points in the product rather than as the general-purpose base for all message filtering. Although writing message filters using Sieve isn't as easy as pointing and clicking in some GUI, the power that Sieve brings to the network manager who might have to implement a complex policy is tremendous. While we focused on tools for fighting spam in this test, if you have a potentially complex policy you want to stick on the same server, these two systems give you a great deal of power. GWGuardian uses Vircom's Modus technology, one of the first commercial Sieve language implementations, while Symantec's Sieve can be nicely hidden under an easy-to-use GUI for the simplest types of policy filters.
If your policy is simpler, such as looking for words in the body of messages, or trying to drop all .EXE attachments or virus-infected messages, you'll be happy with the tools in most of the other products, with two exceptions: Cloudmark and Advascan, which don't have policy-based filtering under the control of a system administrator.
When approaching the policy side of your messaging system, consider whether policies will be uniform across all users, or tied to domains, user groups or even individual users. There was variation on where policies can be applied. For example, Postini gives the option of building deep and complex hierarchies of groups within an organization (or even across organizations), and policies can be applied at any level of the hierarchy. Sophos, Symantec, CipherTrust, MyCom and Messaging Architects also provide some way of applying policies to groups.
It's worth pointing out CipherTrust's Secure Delivery technology. While enforcing e-mail encryption at the gateway is clearly half-baked because it doesn't give true end-to-end encryption or a strong and legally defensible digital signature, the new wave of severe information disclosure and control regulations that have set like concrete around most businesses is making this technology more attractive. In a nutshell, you can have as an action in almost any e-mail policy that the message be "securely delivered." What this means is that CipherTrust will try to encrypt it using standards-based S/MIME, Pretty Good Privacy or Transport Layer Security (TLS) paths. If none of those is available, the message will be stashed on a Web page somewhere, and the recipient will receive an e-mail redirecting him to an encrypted Web page so he can read the message. Look for this kind of security enforcement to appear in future versions of other products.
Security disappoints
In terms of security policy, products ranged from weak to disappointing. Even CipherTrust, with its emphasis on security, had a defective certificate management implementation that kept us from fully testing its code.
Poor security started with management over unencrypted Web sessions. It's unclear why any product designed for more than home use has an unencrypted Web server on it, but BorderWare, Symantec, Barracuda, Cloudmark, MailFrontier, Mycom and Postini all offer full system management capabilities without any encryption requirement. BorderWare and MailFrontier at least give the option of turning off unencrypted management, but it's unclear why they have open Web ports in the first place.
The same security problems extended to features such as encrypted SMTP. BorderWare gets kudos for not only having a clear encrypted SMTP capability, but also the ability to detect man-in-the-middle attacks - alone among our top finishers. Of course, some products get to deflect this criticism by blaming it on the underlying message transfer agent (MTA). For example, Symantec's Windows implementation sits on the Microsoft SMTP MTA, which has no ability to control SMTP encryption. The same is true for Sophos, Proofpoint and Cloudmark, all of which take whatever encryption and control capabilities are built into the MTAs with which they integrate. While you might get Sendmail to do encryption, there is still a policy disconnect because the packages are loosely coupled to both incoming and outgoing mail, simple policy decisions such as "Was this message encrypted incoming?" or "Force this message encrypted outgoing" are not part of these tools.
The strongest security criticism we have for the services from Advascan, Mycom and Postini is that none support encrypted TLS. Postini announced it would be including TLS security features in its products. Postini also gets a slap on the wrist for its user authentication system, based on POP3, which requires every Postini user to constantly send his password in the clear over the Internet to check his quarantine. The alternative is to sacrifice authentication integration with the enterprise, an equally unsavory approach.
While the services should know better, appliance vendors also could do better. Messaging Architects can encrypt management and quarantine traffic, but it doesn't allow for encrypted SMTP.
What's going on?
Most of the top products come with good management interfaces. There are a few blemishes, such as Mycom's Web interface, which obfuscates navigation and context information by hiding options in invisible menus that pop up when you mouse over parts of the screen. Also, Sophos' rule management fills up the screen but never tells you what you want to know. But generally, getting these systems configured once you have them installed will be easy, even for the most harried e-mail administrator.
However, finding out exactly what these boxes are doing is an exercise in confusion. We looked for four features to prove these products were ready to go in a corporate setting: visibility into the message queues, ability to search the logs, basic reports and scheduled reports. Of those four, only the first two are critical - the reports are there to keep the executives well fed, and help in capacity planning. Only two, BorderWare and CipherTrust, gave us an integrated look into our messaging system.
As with security, Sophos, Symantec, Proofpoint, Cloudmark and MailFrontier can blame some of these deficiencies on the underlying platform. With Sendmail as the message delivery system, finding a message in your queue is just a matter of one command-line search command. But that seems like such a 1985 way to do things. Even though these products sit on top of other MTAs, we wanted better integration. Proofpoint tried - you can see the Sendmail logs in the Proofpoint Web GUI.
The other appliance vendors (Barracuda and Messaging Architects) have no excuse for their black-box approach to message management. With no visibility into the messaging queues and no real reporting system, neither administrators nor managers will be happy with the capabilities along these lines.
From the services, we didn't expect much in the way of logs and queue visibility, but we were hoping for some nice reports. Advascan let us schedule reports (a nice plus), while Postini gave us reports on demand.
Picking our favorites
These products have proven themselves capable of doing a great job of filtering spam. It's not a question of better or worse - it's more a question of "What solves your problem best?"
When it comes to roll-your-own software, Sophos' and MailFrontier's offerings impressed us in many ways. But in the world of software-based systems, there are lots of different ways to solve the same problem. For example, if all you want is outstanding spam control, the uncluttered approach of Cloudmark might be your best bet.
On the appliance side, BorderWare was a pretty clear favorite. Although it didn't top other appliance-based anti-spam solutions in every category, it showed excellent design and implementation throughout our testing.
That said, we think Messaging Architects and CipherTrust should also be on your short list. Barracuda's appliance has a fantastic start so early in its life cycle, but issues in management and security kept us from seeing it as an enterprise-class solution today.
If you are looking for a service, Postini gets top billing for the second year in a row. Although Advascan did a great job in filtering mail, our inability to customize it pushed it down on our preference list. With Mycom, the feature set was tremendous for a service, but some consistent delays in performance of the Web GUI and in mail delivery were a concern.