Review of Vamsoft's
|
| Week | DNSBL | Spam Unfiltered |
Spam Filtered |
Spam MX2 |
Spam Total |
Errors | Success Rate* |
|---|---|---|---|---|---|---|---|
| 1 | SpamCop | 177 | 198 | 42 | 375 | 52 | 52.8% |
| 2 | Spamhaus | 213 | 12 | 23 | 226 | 0 | 5.3% |
| 3 | WireHub! | 261 | 0 | 29 | 262 | 7 | 0% |
| 4 | ORDB | 252 | 24 | 26 | 277 | 0 | 13.6% |
| 5 | NJABL | 194 | 103 | 29 | 298 | 34 | 34.6% |
| 6 | Osirusoft | 132 | 168 | 49 | 301 | 0 | 55.8% |
| 7 | Leadmon | 222 | 13 | 33 | 236 | 1 | 5.5% |
| 8 | Blitzed | 267 | 2 | 23 | 270 | 1 | 0.1% |
The table above shows for given weeks the DNSBL used, number of spams that arrived unfiltered, number of filtered spams, spams that circumvented the filter via secondary MX (a subset of unfiltered spam), total spam, number of errors and percentage success rate at spam filtering (computed as strictly as filtered/total, no corrections applied).
In eight weeks I received a whopping 2238 spam emails, or an average of 40 spams per day. Of those spams, 520 were filtered and 1718 were not filtered. The average filter effectiveness over the test period was 23.2%, with the effectiveness of the individual RBL's varying between zero (WireHub!) and 55.8% (Osirusoft Combined). The "errors" column reflects instances where ORF was unable to perform a successful DNS lookup when an email arrived at the Exchange server. While these errors can be indicative of either an overloaded or unavailable DNS server, or of connectivity problems to that server, the time distribution of the errors seem to indicate occasional overloads in the cases of SpamCop and NJABL.
Osirusoft and SpamCop were by a significant margin the two most effective of the eight DNSBLs tested, with Osirusoft winning over SpamCop by just 3%. If one were to factor the 52 errors accessing SpamCop's DNS and the likelihood of those DNS accesses being positive, SpamCop would probably be almost perfectly competitive with Osirusoft. The Blitzed and WireHub! lists were useless and next to useless, respectively.
By correlating information in the SMTP and ORF logs and the headers of the unfiltered spam I noticed several instances where delivery would be attempted through the primary MX several times, clearly being rejected each time, then the originating server would attempt to re-route delivery through the secondary MX. Since in my case the secondary mail exchanger has no filtering, the spam gets through. I also noticed in other cases, delivery was never even attempted through my primary MX! The spam would come straight through my secondary. For a conspicuous example see week #3 where absolutely no email was filtered but 29 came via my secondary. This is no accident - I've NEVER had legitimate email come through my secondary unless the connection to my ISP was down or I was servicing the Exchange box.
Let's look at the secondary MX issue a little. During week six the Osirusoft Combined List helped catch 55.8% of the spam, but 49 of the emails that were not filtered arrived via the mail queue defined in my secondary MX. If those emails had not taken the back door in, the filtering success rate that week could have been as high as 72.1%.
Secondary MX queues are not necessarily the only detour email can make. While further analyzing my results, I discovered that I was receiving email for an address that I abandoned four years ago. Apparently I set that address to forward to my current email back then, and the ISP never turned off that account. Since that email was being forwarded through a legitimate SMTP server, those emails were also not filtered and further skewed the statistics, although not significantly enough to represent here.
I felt it important to note the forwarding issue in spite of its statistical insignificance during testing, since for example this past week I've seen over ten times the traffic (nearly 8% of all spam received) trying to reach that obsolete email address as there was during the eight weeks I tested ORF.
For the long term I settled on the Osirusoft Combined List DNSBL. Discussions on Usenet allege that SpamCop is a substantially more effective tool but my tests showed otherwise. Over the past five months I have been monitoring ORF and have had only a few discernable false positives, and those few instances were actually questionable or even perfectly correct, i.e. mass-mailings from Buy.com being filtered, Yahoo! Groups activity getting blocked due to their non confirmed opt-in signup policy, and a client of mine getting blocked because their SMTP server had more holes in it than O.J.'s alibis.
There was only one problematic episode in January 2003 when some innocent party's server was being exploited by a spammer. Every time ORF blocked the email, the server would try again in seconds. The sending server simply was not taking "no" for an answer!
In retrospect I suspect the problem may have been due to ORF having been configured for a forcible disconnect, but that option never caused trouble with any other servers and I forgot about it entirely. At the time I alleviated the problem by temporarily white-listing the server to allow the email through, and calling the hosting company to tell them to unplug the server and get it fixed so nobody else gets caught in similar hell.
Going back to that client that landed on a DNSBL because of their exploited server, I discovered that once they were verifiably removed from the two DNSBL's that had them listed (ORDB and Osirusoft), ORF was still rejecting their emails. This behavior persisted until I flushed the caches on both my internal (Windows 2000 Server, Active Directory domain controller) DNS and the DNS resolver on the Exchange box. DNS in Win2K caches queries and honors the TTL and expiration values on those DNS records that it queries, values which are often left at large default values in order to minimize redundant DNS traffic. I consider this issue to be poor administrative judgment on the part of the DNSBL operators, rather than a failing of ORF or of my LAN architecture.
Open Relay Filter tolerated perfectly the application of both Win2K SP3 and Exchange2K SP3. In a 90 day period without reboots, there were no detectable anomalies on the server, performance issues or unusual memory growths.
In a two-week follow-up test beginning March 2, 2003, ORF (still configured to use Osirusoft's Combined List) blocked 642 spams but permitted reception of 204. That gives Osirusoft an average success rate of 68.3%. These numbers are not skewed by routing via secondary MX or forwarding from old ISP's.
I created a web page that connects to the Exchange folder where the unfiltered spam is saved - you can view a list of the eight weeks of unfiltered spam here. Give that page a moment or two to load, it's rather large! A similar page for the two-week follow-up test is here. Email that arrived via MX2 is presented in red and email that relayed through my old ISP is presented in fuchsia. I also have a page that parses the ORF logs for that period to show you the spam that was filtered - you can view that list here. It's very interesting to see how obnoxious the junk email can really be.
The aspects of email routing are important for software of this nature. If you collect email addresses like baseball cards and have them all forwarding to single email address, then all those email addresses will not be filterable. This should not be confused with having many domains and having all their respective MX records pointing to a single SMTP server. In that case, the filtering should still work because the email is not being relayed through another legitimate SMTP server.
The problem of secondary (and tertiary, etc.) MX queues are also significant. If those SMTP servers don't use similar filters, the effectiveness of your own filtering will be reduced by as much as 50%. If you are like me, where your secondary MX is provided by your ISP and you have no control over its configuration, you might consider bartering an arrangement with someone else in a similar pickle. That is, you offer to host their secondary MX queue and they host your secondary MX queue. If you both use the same filtering, everyone is happy and the back doors to spam are slammed shut.
Importantly, DNSBLs do have a dark side. There is the potential to unintentionally bounce perfectly legitimate email just because the sender happens to be using a listed server or IP address space, and you're at the mercy of the people managing the DNSBL and/or its data sources with regards to the accuracy of that data. The use of a DNSBL could subject your email users to loss of information and business, although ORF does always send descriptive bounce messages to blocked senders. The politics of DNSBLs are outside the scope of this article but you can read more by searching this web site for relevant terms.
As far as performance, Vamsoft was a little unclear on what sort of performance or capacity I could expect from ORF. Open Relay Filter worked flawlessly for me on a somewhat under-built server and in Windows' Perfmon analysis, ORF imposed no substantial overhead for CPU time, disk I/O, etc. Inbound email delivery times were increased at most a second or two if at all, but any such delays are strictly a factor of DNS lookup latency and I compiled no statistics about delays versus DNSBLs. Outbound email delivery was unaffected.
ORF SE has one particular limitation that I find significant, which is solved by investing in the more expensive Enterprise Edition. You cannot update the list of DNSBLs without waiting for the software to be upgraded. So far, there has been only one such upgrade since I purchased the product in June 2002.
Feature-wise, the only thing I could think of asking for is an "action on block" option for the more militant anti-spam crowd, so that in addition to logging blocked email ORF could fire off optional (user-constructed) commands or scripts to do things like inform the ISP, etc.
The Standard Edition of ORF is only 25 EUR, or about US$27.50 at the time of this writing. The Enterprise Edition is 99 EUR and offers quite a bit more functionality more scalability. Both are available with volume discounts. The price and the value seems absolutely unbeatable for either product.
The trial version of ORF can be downloaded immediately from Vamsoft's web site, where you can also order it on-line quickly and easily. Purchase of the software gets you a customer code (typically within one business day) that gives you access to download an unlimited version of ORF, and facilitates access to special support areas of the web site. Vamsoft also provides support via a dedicated open-access news server.
Product rating: * * * * * (GREAT!)
Enterprise Edition Update - December 2003!
Vamsoft no longer sells the Standard Edition version of Open Relay Filter. Just as well, since a major shortcoming was that the DNSBL information was hard-coded into the software and those resources, especially in light of recent DDOS attacks against them and the FBI's atrocious ignorance of the matter, tend to come and go.
The ORF-EE software still sells for 99 EUR, but owners of the SE product may upgrade for 49 EUR. This makes ORF the deal of the century in my opinion. The decision to upgrade is a no-brainer.
Unfortunately, configuration settings do not carry over from SE to EE. You must de-install SE, then install EE. No reboots were necessary for either step. ORF-EE is now delivered as a un-installable package. ORF-EE installs and configures easily enough, and although you no longer have to restart your SMTP virtual servers or services after ORF configuration changes, you do still have to remember to restart the ORF services. This is done handily from the configuration software.
The Enterprise Edition offers many huge improvements over the SE product. The console software is no longer a Control Panel applet. It sports a similar simple status screen when the software comes up, but the similarity ends there. The tabbed panels have been ditched in favor of a navigation tree on the left side of the screen, from which you may select a very informative statistics display as well as the following options...
- Server bindings
- DNS options
- Logging and events
- Statistics
- Caching
- DNS and IP whitelisting and blacklisting
- Reverse DNS test
- Sender and recipient whitelisting and blacklisting
- Active Directory synchronization
- Miscellaneous options
Most of these options are new. Notable is that you may now choose to use multiple DNS blacklists simultaneously, the list can be updated via XML files supplied by Vamsoft, and you can make your own additions to the list of DNSBLs. You may also whitelist and blacklist by both the IP address of the sending email server and the email addresses, and ORF can import email addresses from your Win2K Active Directory to be used for further filtering. You can export and import the settings for backup purposes or to propagate the settings to other servers. ORF can also now send events to the Windows Event Log service and to SYSLOG daemons, as well as send email notifications. ORF-EE offers very granular control over the conditions to be logged and which conditions should cause how much alarm.
In practice there are some limits to what you should do, versus what you can do. You CAN select every single DNSBL, but you SHOULD only select a few carefully evaluated lists, or else you'll end up bouncing all sorts of legitimate email and substantially delaying the few that make it through. You CAN log to ORF's own log files, the Windows Event Logs and SYSLOGD simultaneously, but you SHOULD choose one mechanism and stick with it.
ORF now has its own DNS cache, which should ease the load on the increasingly popular DNSBLs. The cache is adjustable. ORF ships with the information for about 27 different DNSBL services already programmed in and ready to select.
This year saw the demise of Osirusoft's DNSBL. Joe Jared's "popularity" made his service a high- profile target for DDOS attacks orchestrated by spammers, and he eventually caved. I was not a big fan of his but I'm sad to see him go. In my original evaluation of DNSBLs the Osirusoft service was about on par with SpamCop. This still remains true. Since reconfiguring ORF to use SpamCop I have seen no perceivable change in overall filter effectiveness or fairness.
Easynet also bit the dust this year, though not necessarily as a result of DDOS attacks. The most useful part of Easynet in my opinion - the Dynablocker - has been taken over by SORBS and uses zone 10, a.k.a. DUL-SORBS. ORF-EE leaves this option disabled by default for some reason. I reconfigured ORF to use SORBS instead of Easynet, enabled zone 10, and then almost immediately had to disable zone 6 because I was getting way too many false positives.
I have found that using SpamCop and the Dynablocker together, I could filter more than 90% of the incoming spam with little risk of false positives. With spammers relying increasingly on exploited DSL and Cable Internet customers, the Dynablocker - a compendium of "dynamic" ISP customer IP address ranges - has been getting more and more effective, often exceeding 95% effectiveness.
Open Relay Filter Enterprise Edition is efficient, effective, pretty much as configurable as it could possibly be, is inexpensive, simple to install and set up, and generally trouble-free. I couldn't possibly be more delighted with this software.
Handy Stuff
Here is a Win2K command file that helps tally the output from ORF's logs:
@ECHO OFF :: ORFTALLY.CMD :: SET LOGDIR TO YOUR LOG PATH :: ASSUMES ALL LOGGING OPTIONS ENABLED SETLOCAL SET BLOCKS=0&SET ALLOWS=0 SET TRAFFIC=0&SET DAYS=0 SET LOGDIR=%WINDIR%\SYSTEM32\LOGFILES\ORF FOR %%f IN (%LOGDIR%\*.LOG) DO CALL :GETDATA %%f ECHO Activity for %DAYS% days in Open Relay Filter: ECHO Email Blocked: %BLOCKS% ECHO Email Allowed: %ALLOWS% ECHO Total Traffic: %TRAFFIC% GOTO :EOF :GETDATA ECHO Checking %1... SET /A DAYS=DAYS+1 FOR /F "skip=2 tokens=4,5*" %%k IN (%1) DO ( IF "%%k"=="ALLOW" ( SET /A TRAFFIC=TRAFFIC+1 SET /A ALLOWS=ALLOWS+1 ) IF "%%k"=="BLOCK" ( SET /A TRAFFIC=TRAFFIC+1 SET /A BLOCKS=BLOCKS+1 FOR /F "tokens=3,5 delims=() " %%n IN ("%%m") DO ( ECHO FROM %%n TO %%o ) ) GOTO :EOFHere is an ASP (Active Server Page) script to display the spam collection:
' DisplaySpam.asp intMailCount = 0 intSubTotal = 0 intDayIndex = 0 intWeekNum = 1 intDayCount = 0 intMX2count = 0 intCOLcount = 0 strFontFuschia = "<font color='#FF00FF'>" strFontRed = "<font color='#FF0000'>" Set objRecord = Server.CreateObject("ADODB.Record") Set objRecordSet = Server.CreateObject("ADODB.Recordset") Set objConnection = Server.CreateObject("ADODB.Connection") strURLMailbox = _ "file://./backofficestorage/mydomain.tld/public folders" objConnection.Provider = "ExOLEDB.DataSource" objConnection.Open strURLMailbox objRecord.Open strURLMailbox, objConnection, adModeRead ' folder below must have rights set to Default = Reviewer strURLInbox = _ objRecord.Fields.Item("urn:schemas:httpmail:inbox").Value _ & "/folder/subfolder" objRecord.Close objRecord.Open strURLInbox, objConnection, adModeRead strSQL = "select * " _ & "from scope ('shallow traversal of """ _ & strURLInbox & """') " _ & "where ""DAV:ishidden"" = False " _ & "order by ""urn:schemas:httpmail:datereceived""" objConnection.BeginTrans objRecordSet.Open strSQL, objConnection While Not objRecordSet.EOF With objRecordSet.Fields intMailCount = intMailCount + 1 intSubTotal = intSubTotal + 1 ' Get date/time, compensate for time zone... MsgDateTime = _ DateAdd("h",-4,.Item("urn:schemas:httpmail:datereceived").Value) sep = Instr(MsgDateTime," ") ItemDate = Left(MsgDateTime,sep-1) ItemTime = Mid(MsgDateTime,sep+1) If ItemDate <> PrevItemDate Then intDayCount = intDayCount + 1 intDayIndex = intDayIndex + 1 If intDayIndex = 8 Then Response.Write "Week #" & intWeekNum & " subtotal: " & intSubTotal intSubTotal = 0 intDayIndex = 1 intWeekNum = intWeekNum + 1 End If End If MsgSubject = .Item("urn:schemas:httpmail:subject").Value MsgFrom = .Item("urn:schemas:httpmail:from").Value MsgHeader = .Item("urn:schemas:mailheader:received").Value If Instr(MsgHeader, "otherisp.tld") Then COL = vbTrue intCOLcount = intCOLcount + 1 Else COL = vbFalse End If If Instr(MsgHeader, "mx2.ourisp.tld") Then MX2 = vbTrue intMX2count = intMX2count + 1 Else MX2 = vbFalse End If ' Server.htmlencode barfs on nulls... If VarType(MsgSubject) = vbNull Then MsgSubject = "" ' Mangle email addresses to avoid harvesting... AddrFound = Instr(MsgSubject,"@mydomain.tld") If AddrFound > 0 Then MsgSubject = Left(MsgSubject,AddrFound) & _ ":JUMBLED:" & Mid(MsgSubject,AddrFound+1) End If If MX2 Then Response.Write(strFontRed) If COL Then Response.Write(strFontFuschia) Response.Write MsgFrom & " " Response.Write Server.HTMLencode(MsgSubject) Response.Write MsgDateTime If MX2 Or COL Then Response.Write("</font>") PrevItemDate = ItemDate End With objRecordSet.MoveNext Wend objConnection.CommitTrans objRecord.Close objConnection.Close Response.Write "Week #" & intWeekNum & " subtotal: " & intSubTotal Response.Write "Total in " & intDayCount & " days: " & intMailCount
|
|