taint.org: Justin Mason’s Weblog

I have a Google News notification set up for mentions of “SpamAssassin”, which is how I came across this press release on PRNewsWire:

Study: Challenge-Response Surpasses Other Anti-Spam Technologies in Performance, User Satisfaction and Reliability; Worst Performing are Filter-based ISP Solutions

NORTHBOROUGH, Mass., July 17 /PRNewswire/ — Brockmann & Company, a research and consulting firm, today released findings from its independent, self-funded “Spam Index Report– Comparing Real-World Performance of Anti-Spam Technologies.”

The study evaluated eight anti-spam technologies from the three main technology classes — filters, real-time black list services and challenge- response servers. The technologies were evaluated using the Spam Index, a new method in anti-spam performance measurement that leverages users’ real-world experiences.

[...] The report finds that the best performing anti-spam technology is challenge-response, based on that technology’s lowest average Spam Index score of 160.

[...] Filter – Open Source software-(Spam Index: 388): This technology is frequently configured to work in conjunction with PC email client filters. The server adds * * SPAM * * to the subject line so that the client filter can move the message into the junk folder. This class of software includes projects such as ASSP, Mail Washer and SpamAssassin, among others.

The “Spam Index” is a proprietary measurement of spam filtering, created by Brockmann and Company. A lower “Spam Index” score is better, apparently, so C/R wins! (Funny that. The author, Peter Brockmann, seems to have some kind of relationship with C/R vendor Sendio, being quoted in Sendio press releases like this one and this one, and providing a testimonial on the Sendio.com front page.)

However — there’s a fundamental flaw with that “Spam Index” measurement, though; it’s designed to make C/R look good. Here’s how it’s supposed to work. Take these four measurements:

• Average number of spam messages each day x 20 (to get approximate number per work-month)
• Average minutes spent dealing with spam each day x 20 (to get approximate minutes per work-month)
• Number of resend requests last month
• Number of trapped messages last month

Then sum them, and that gives you a “Spam Index”.

First off, let’s translate that into conventional spam filter accuracy terms. The ‘minutes spent dealing with spam each day’ measures false negatives, since having to ‘deal with’ (ie delete) spam means that the spam got past the filter into the user’s inbox. The ‘number of trapped messages’ means, presumably, both true positives — spam marked correctly as spam — and false positives – nonspam marked incorrectly as spam. The ‘number of resend requests last month’ also measures false positives, although it will vastly underestimate them.

Now, here’s the first problem. The “Spam Index” therefore considers a false negative as about as important as a false positive. However, in real terms, if a user’s legit mail is lost by a spam filter, that’s a much bigger failure than letting some more spam through. When measuring filters, you have to consider false positives as much more serious! (In fact, when we test SpamAssassin, we consider FPs to be 50 times more costly than a false negative.)

Here’s the second problem. Spam is sent using forged sender info, so if a spammer’s mail is challenged by a Challenge/Response filter, the challenge will be sent to one of:

• (a) an address that doesn’t exist, and be discarded (this is fine); or
• (b) to an invalid address on an innocent third-party system (wasting that system’s resources); or
• (c) to an innocent third-party user on an innocent third-party system (wasting that system’s resources and, worst of all, the user’s time).

The “Spam Index” doesn’t measure the latter two failure cases in any way, so C/R isn’t penalised for that kind of abusive traffic it generates.

Also, if a good, nonspam mail is challenged, either

• (a) the sender will receive the challenge and take the time to jump through the necessary hoops to get their mail delivered (”visit this web page, type in this CAPTCHA, click on this button” etc.); or
• (b) they’ll receive the challenge, and not bother jumping through hoops (maybe they don’t consider the mail that important); or
• (c) they’ll not be able to act on the challenge at all (for example, if an automated mail is challenged).

Again, the “Spam Index” doesn’t measure the latter two failure cases.

In other words, the situations where C/R fails are ignored. Is it any wonder C/R wins when the criteria are skewed to make that happen?

Tags: anti-spam, challenge-response, false-positives, measurement, press-releases, sendio, spamassassin

As all right-thinking people know by now, Challenge-response spam filtering is broken and abusive, since it simply shifts the work of filtering spam out of your email, onto innocent third-parties — either your legitimate correspondents, people on mailing lists you read, or even random people you have never heard of (due to spam blowback).

I’ve ranted about this in the past, but I’m not alone in this opinion — and frequently find myself explaining it. To avoid repeating myself, here’s a canonical collection of postings from around the web on this topic.

• Spamcop FAQ: Why are auto responders bad?:

Description: This “selfish” method of spam filtering replies to all email with a “challenge” – a message only a living person can (theoretically) respond to. There are several problems with this method which have been well known for many years.

1. Does not scale: If everyone used this method, nobody would ever get any mail.
2. Annoying: Many users refuse to reply to the challenge emails, don’t know what they are or don’t trust them.
3. Ineffective: Because of confusion about these emails, many of them are confirmed by people who did not trigger them. This results in the original malicious email being delivered.
4. Selfish: This is the problem we are mainly concerned with. By using challenge/response filtering, you are asking innumerable third parties to receive your challenge emails just so that a relatively few legitimate ones get through to the intended recipient.

• Karsten M. Self: Challenge-Response Anti-Spam Systems Considered Harmful:

C-R systems in practice achieve an unacceptably high false-positive rate (non-spam treated as spam), and may in fact be highly susceptible to false-negatives (spam treated as non-spam) via spoofing.

Effective spam management tools should place the burden either on the spammer, or, at the very least, on the person receiving the benefits of the filtering (the mail recipient). Instead, challenge-response puts the burden on, at best, a person not directly benefitting, and quite likely (read on) a completely innocent party. The one party who should be inconvenienced by spam consequences ¿ the spammer ¿ isn’t affected at all.

Worse: C-R may place the burden on third parties either inadvertantly (via spoofed sender spam or virus mail), or deliberately (see Joe Job, below). Such intrusions may even result in subversion of the C-R system out of annoyance. Many recent e-mail viruses spoof the e-mail sender, including Klez, Sobig variants, and others.

• John Levine: Challenge-response systems are as harmful as spam:

The collateral damage from widely used C/R systems, even with implementations that avoid the stupid bugs, will destroy usable e-mail. [jm: in fairness, this was written in 2003.]

Challenge systems have effects a lot like spam. In both cases, if only a few people use them they’re annoying because they unfairly offload the perpetrator’s costs on other people, but in small quantities it’s not a big hassle to deal with. As the amount of each goes up, the hassle factor rapidly escalates and it becomes harder and harder for everyone else to use e-mail at all.

• Ed Felten: A Challenging Response to Challenge-Response:

I’m skeptical of CR as a response to email. If you’re the first on your block to adopt CR, and if nobody else uses anti-spam technology, then CR might provide you some modest benefit. But it¿s hard to see how CR can be widely successful in a world where most people use some kind of spam defense.

• Jeremy Zawodny: TMDA Users Can Blow Me (heh):

If these systems are so brain-dead as to not bother adding my address to the whitelist when the user sends me e-mail, I have serious trouble understanding why anyone is using them.

Is it just me? Is this too hard to figure out?

Anyway, there’s another 5 minutes I’ll never get back. It’s too bad there’s no mail header to warn me that “this message is from a TDMA user”, because then I’d be able to procmail ‘em right to /dev/null where they belong.

Ugh.

This bullshit is not going to “solve” the spam problem, people. If that’s your solution, please let me opt out. Forever.

• Michele Neylon: Why C/R is not a good idea:

C/R slows down and impedes communication by placing unwanted barriers between you and your clients/suppliers.

If you must insist on using some form of C/R please make sure that you whitelist my address before you contact me as I will not reply to challenges.

• TidBITS policy on Challenge-Response:

We will not answer any challenges generated in response to our mailing list postings. Thus, if you’re using a challenge-response system and not receiving TidBITS, you’ll need to figure that out on your own. Also, if you send us a personal note and we receive a challenge to our reply, we may or may not respond to it, depending on our workload at the time.

• Fedora Project policy on UOL — a Brazilian ISP that uses C-R extensively:

uol.com.br uses a very broken method of anti-spam. Everytime someone sends an email message to one of their members, they send back a verification message, asking the original sender to click a link before they will allow the message through. These messages are themselves a form of spam, and the resulting back-scatter of these messages is altogether bad for the Internet, the UOL member, and all of the UOL member’s contacts. UOL is aware of the complaints against them, and they refuse to correct the issue, claiming that their members love the service.

• Matt Sergeant: C-R spam solutions:

I hate C/R systems. With a passion. I absolutely will not respond to them. They go in the trash. I don’t get them very often but I get them more and more. I think they have the potential to seriously damage email communication as we know it. And I’m not alone in this opinion.

• Richi Jennings: ‘Challenge/Response makes you a spammer.’

• BusinessWeek: Stephen Wildstrom: A Spam-Fighter More Noxious Than Spam: ‘Challenge-response filtering systems are likely to wipe out e-mails you want, too.’

• and lots more at Spamlinks.net

Phew.

Tags: abuse, anti-spam, backscatter, blowback, bounces, c-r, challenge-response, email, filtering, rants, smtp

Yay! Kudos to Richi Jennings, who’s been trumpeting the dangers of backscatter to InformationWeek recently. It’s a great article. I particularly like how it digs up this impressively off-the-mark quote:

Tal Golan, CTO, president, and founder of Sendio, maker of a challenge/response e-mail appliance used by more than 150 enterprise consumers, disagrees strongly with Jennings’s assertion that challenge-based filtering has problems. “Without question, the benefit to the whole community at large drastically outweighs that FUD [fear, uncertainty, and doubt] that’s out there in the marketplace that somehow challenge/response makes the problem worse,” he says. “The real issue is that filters don’t work. From our perspective, challenge/response is the only solution. This whole concept of backscatter is just not true. Very, very rarely do spammers forge the e-mail addresses of legitimate companies anymore.”

hahahaha. Well, since last Thursday, “very very rarely” translates as “214 MB of backscatter in my inbox”. The facts aren’t on Tal Golan’s side here…

(PS: SpamAssassin 3.2.0 will include backscatter detection.)

Tags: anti-spam, backscatter, bounces, challenge-response, filtering, funny, richi-jennings, sendio, spam

I get quite a lot of spam. For one random day last month (Aug 21st), I got 48 low-scoring spam mails (between 5 and 10 points according to SpamAssassin), and 955 high-scorers (anything over 10). I don’t know how much malware I get, since my virus filter blocks them outright, instead of delivering to a folder.

That’s all well and good, because spam and viruses are now relatively easy to filter — and if I recall correctly, they were all correctly filed, no FPs or FNs (well, I’m not sure about the malware, but fingers crossed ;).

The hard part is now ‘bogus bounces’ — the bounces from ‘good’ mail systems, responding to the forged use of my addresses as the sender of malware/spam mails. There were 306 of those, that day.

Bogus bounces are hard to filter as spam, because they’re not spam — they’re ‘bad’ traffic originating from ‘good’, but misguided, email systems. They’re not malware, either. They’re a whole new category of abusive mail traffic.

I say ‘misguided’, because a well-designed mail system shouldn’t produce these. By only performing bounce rejection with a 4xx or 5xx response as part of the SMTP transaction, when the TCP/IP connection is open between the originator and the receiving MX MTA, you avoid most of the danger of ’spamming’ a forged sender address. However, many mail systems were designed before spammers and malware writers started forging on a massive scale, and therefore haven’t fixed this yet.

I’ve been filtering these for a while using this SpamAssassin ruleset; it works reasonably well at filtering bounces in general, catching almost all of the bounces. (There is a downside, though, which is that it catches more than just bogus bounces — it also catches real bounces, those in response to mails I sent. At this stage, though, I consider that to be functionality I’m willing to lose.)

The big remaining problem is challenge-response messages.

C-R is initially attractive. If you install it, your spam load will dwindle to zero (or virtually zero) immediately — it’ll appear to be working great. What you won’t see, however, is what’s happening behind the scenes:

• your legitimate correspondents are getting challenges, will become annoyed (or confused), and may be unwilling or unable to get themselves whitelisted;

• spam that fakes other, innocent third party addresses as the sender, will be causing C-R challenges to be sent to innocent, uninvolved parties.

The latter is the killer. In effect, you’re creating spam, as part of your attempts to reduce your own spam load. C-R shifts the cost of spam-filtering from the recipient and their systems, to pretty much everyone else, and generates spam in the process. I’m not alone in this opinion.

That’s all just background — just establishing that we already know that C-R is abusive. But now, it’s time for the next step for me — I’ve had enough.

I initially didn’t mind the bogus-bounce C-R challenges too much, but the levels have increased. Each day, I’m now getting a good 10 or so C-R challenges in response to mails I didn’t send. Worse, these are the ones that get past the SpamAssassin ruleset I’ve written to block them, since they don’t include an easy-to-filter signature signifying that they’re C-R messages, such as Earthlink’s ’spamblocker-challenge’ SMTP sender address or UOL’s ‘AntiSpam UOL’ From address. There seems to be hundreds of half-assed homegrown C-R filters out there!

So now, when I get challenge-response messages in response to spam which forges one of my addresses as the ‘From’ address, and it doesn’t get blocked by the ruleset, I’m going to jump through their hoops so the spam is delivered to the C-R-protected recipient. Consider it a form of protest; creating spam, in order to keep youself spam-free, is simply not acceptable, and I’ve had enough.

And if you’re using one of these C-R filters — get a real spam filter. Sure they cost a bit of CPU time — but they work, without pestering innocent third parties in the process.

Tags: anti-spam, challenge-response, spam, spamateurs