taint.org: Justin Mason’s Weblog

Last year, I blogged about Full-Text RSS, a utility to convert those useless “partial-text” RSS/Atom feeds into the real, full-story-inline deal.

The only downside is that the author felt it necessary to withhold the source, saying:

Still, I wouldn’t want to offer a feature that middlemen can resell at the expense of bloggers. So while I do want to open this up, I don’t want to make things easy for the unscrupulous.

However, recently Keyvan Minoukadeh from the Five Filters project got in touch to say:

I recently created a similar service (along with a bookmarklet for it). [...] It’s a free software (open source) project so code is also available.

Here it is:

fivefilters.org: Create Full-Text Feeds

I’ve tried it out and it works great, and the source is indeed downloadable under the AGPL.

Five Filters — its overarching project — looks interesting, too:

Edward Herman and Noam Chomsky describe the media as businesses which sell a product (readers) to other businesses (advertisers). In their propaganda model of the media they point to five ‘filters’ which determine what we read in the newspapers and see on the television. These filters produce a very narrow view of the world that is in line with government policy and business interests.

In this project we try to encourage readers to explore the world of non-corporate online news, websites which avoid the five filters of the propaganda model. We also try to make these sources of news more accessible by allowing users to print the stories found on these alternative news sites in the format of a newspaper.

Tags: atom, conversion, feeds, filtering, filters, full-text, partial-text, rss, text

• Vint Cerf interviewed on spam, malware etc. : pretty much the EFF party line, I think: “every man for himself”. also talks about net neutrality
  (tags: vint-cerf internet filtering spam malware abuse network-neutrality anti-spam eff)

• video of a fake e-Passport being accepted by airport security reader : an e-Passport for “Elvis Aaron Presley”, no less, happily scanned by an Amsterdam passport security station. hahahaha!
  (tags: elvis funny security e-passports video via:slashdot rfid)

• Facebook adds Ireland as a Friend : ‘Dublin will be the centre for Facebook’s international operations and will provide a range of online technical, sales and operations support to Facebook’s users and customers across EMEA region.’ good news
  (tags: facebook dublin ireland web2.0 emea)

• RFC-5321 (Obsoletes: 2821) : The newest rev to the Simple Mail Transfer Protocol (via fanf)
  (tags: rfcs rfc-2821 standards internet smtp email rfc-5321)

• RFC-5322 (Obsoletes: 2822) : the newest rev to the Internet Message Format for email (via fanf)
  (tags: via:fanf rfc rfc-2822 rfc-5322 standards email internet)

Tags: abuse, anti-spam, dublin, e-passports, eff, elvis, email, emea, facebook, filtering, funny, internet, ireland, malware, network-neutrality, rfc, rfc-2821, rfc-2822, rfc-5321, rfc-5322, rfcs, rfid, security, smtp, spam, standards, via:fanf, via:slashdot, video, vint-cerf, web2.0

• Commtouch Plug-in for SpamAssassin : SA plugin to add the proprietary Commtouch filter to an existing SpamAssassin system; nifty
  (tags: commtouch spamassassin anti-spam filtering plugins)

Tags: anti-spam, commtouch, filtering, plugins, spamassassin

GoDaddy is rejecting mail with URLs that appear in the Spamhaus PBL. As this thread on the Amazon EC2 forum notes, this is creating false positives, causing nonspam mail to be rejected. Here’s what GoDaddy reportedly said about this policy:

Unfortunately, our system is set to reject mails sent from or including links listed in the SBL, PBL or XBL. Because the IP address associated to [REMOVED] is listed in the PBL, any emails containing a link to this site will be rejected. This includes plain-text emails including this information.

If this is true, it’s utterly broken.

Spamhaus explicitly warn that this is not to be done, on the PBL page:

Do not use PBL in filters that do any ‘deep parsing’ of Received headers, or for other than checking IP addresses that hand off to your mailservers.

And more explicitly in the Spamhaus PBL FAQ:

PBL should not be used for URI-based blocking! Consider the false positive potential: legitimate webservers hosted with services such as dyndns.com or ath.cx! Or consider that ISPs and other networks are encouraged to list any IP ranges which should not send mail, and that could include web servers! Use SBL or XBL (or sbl-xbl.spamhaus.org) for URI blocking as described in our Effective Spam Filtering section. Use PBL only for SMTP (mail).

Critically, the PBL now lists all Amazon EC2 space, since Spamhaus interpret Amazon’s policy as forbidding email to be delivered via direct SMTP from there. (Note — email, not HTTP.)

With this filter in place at GoDaddy, that now means that if you mail a URL of any page on any site hosted at EC2 to a user of GoDaddy, your mail won’t get through.

Note: this is much worse than blocks of SMTP traffic from EC2. In that case, an EC2 user can relay their legit SMTP traffic via an off-EC2 host. In this case, there is no similar option in HTTP that isn’t insufferably kludgy. :(

Tags: anti-spam, broken, ec2, filtering, godaddy, pbl, spamhaus, sucks

• Cycle Helmets and Other Religious Symbols : there appears to be a lack of published research suggesting that bike helmets help avoid serious injury and death — in fact, research seems to suggest the _opposite_.
  (tags: cycling helmets safety research bikes equipment)

• Clever method of near duplicate detection : ‘SIGIR 2008 paper, “SpotSigs: Robust and Efficient Near Duplicate Detection in Large Web Collections”‘. may be useful, although we’ve pretty much stopped deduping in SpamAssassin nowadays
  (tags: corpora dupes duplicates spotsigs collections sigir papers via:jzawodny)

• HadoopStreaming : ‘Using the streaming system you can develop working hadoop jobs with extremely limited knowldge of Java. [..] Hadoop basically becomes a system for making pipes from shell-scripting work (with some fudging) on a cluster.’
  (tags: hadoop perl streams unix distcomp clusters mapreduce)

• political zealot using GMail’s “this is spam” button to deliberately cause spamfilter problems for Obama’s campaign : ‘Tablemate at benihana confided how he subscribes to Obama’s mailing list and marks it all as spam to train Gmail. Urge to kill rising.’ – Kevin Fox on Twitter
  (tags: twitter kevin-fox foaf-story benihana obama anti-spam filtering this-is-spam us-politics moveon)

• “Jake Leg” : ‘large numbers of [adulterated Prohibition-era alcohol, Jamaican Ginger Extract] users began to lose use of their hands and feet. Some victims could walk, but they had no control over the muscles which would normally have enabled them to point their toes upward. Therefore, they would raise their feet high with the toes flopping downward, which would touch the pavement first followed by their heels. The toe first, heel second pattern made a distinctive “tap-click, tap-click” sound as they walked. This very peculiar gait became known as the jake walk and those afflicted were said to have jake leg’
  (tags: jake-leg walking history prohibition alcohol odd bizarre adulteration poison 1930s)

Tags: 1930s, adulteration, alcohol, anti-spam, benihana, bikes, bizarre, clusters, collections, corpora, cycling, distcomp, dupes, duplicates, equipment, filtering, foaf-story, hadoop, helmets, history, jake-leg, kevin-fox, mapreduce, moveon, obama, odd, papers, perl, poison, prohibition, research, safety, sigir, spotsigs, streams, this-is-spam, twitter, unix, us-politics, via:jzawodny, walking

As I noted on Monday, the Irish branches of several major record companies have brought a case against Eircom, demanding in part that the ISP install Audible Magic’s Copysense anti-filesharing appliances on their network infrastructure.

I thought I’d do a quick bit of research online into how they do their filtering. Here’s what the EFF had to say:

Audible Magic’s technology can easily be defeated by using one-time session key encryption (e.g., SSL) or by modifying the behavior of the network stack to ignore RST packets.

It’s interesting to see that they used RST packets — this is the same mechanism used by the “Great Firewall of China” to censor the internet:

the keyword detection is not actually being done in large routers on the borders of the Chinese networks, but in nearby subsidiary machines. When these machines detect the keyword, they do not actually prevent the packet containing the keyword from passing through the main router (this would be horribly complicated to achieve and still allow the router to run at the necessary speed). Instead, these subsiduary machines generate a series of TCP reset packets, which are sent to each end of the connection. When the resets arrive, the end-points assume they are genuine requests from the other end to close the connection — and obey. Hence the censorship occurs.

But there’s a very easy way to avoid this, according to that blog post:

However, because the original packets are passed through the firewall unscathed, if both of the endpoints were to completely ignore the firewall’s reset packets, then the connection will proceed unhindered! We’ve done some real experiments on this — and it works just fine!! Think of it as the Harry Potter approach to the Great Firewall — just shut your eyes and walk onto Platform 9¾.

Clayton, Murdoch, and Watson’s paper on this technique provides the Linux and FreeBSD firewall commands they used to do this. Here’s Linux:

   iptables -A INPUT -p tcp --tcp-flags RST RST -j DROP

For FreeBSD, the command is:

   ipfw add 1000 drop tcp from any to me tcpflags rst in

So assuming Copysense haven’t changed their approach yet, it’s trivial to block Copysense’s filtering, if both ends are running Linux or BSD. I predict if Copysense becomes widespread, someone will patch Windows TCP to do the same.

I love Audible Magic’s response:

The current appliance happens to use the TCP Reset to accomplish this today. There are many other technical methods of blocking transfers. Again, we have strategies to deal with them should they ever prove necessary. This is why we recommend our customers purchase a software support agreement which provides for these enhancements that keep their purchase up-to-date and protect their investment.

in other words, “hey customers! if you don’t have a support contract, you’re shit out of luck when the p2p guys get around our filters!” Nice. ;)

Tags: audible-magic, censorship, copysense, eircom, filtering, internet, ireland, p2p, rst, tcp

Trend Micro are demanding that Barracuda Networks pay licensing fees, alleging that they infringe U.S. Patent No. 5,623,600 with their use of the open-source anti-virus tool ClamAV. Here’s a Barracuda press release, and here’s some details from Barracuda:

Trend Micro alleges that Barracuda Networks and ClamAV infringe on Trend Micro’s U.S. Patent No. 5,623,600. Barracuda Networks believes that the patent is invalid due to prior art and further believes that neither its products nor the ClamAV software infringe the patent.

On Sept. 21, 2006, Trend Micro sent Barracuda Networks a letter regarding a license to Trend Micro’s ‘600 patent. After several discussions on paying a license for the patent, Trend Micro demanded Barracuda Networks either remove ClamAV from its products or pay a patent license fee. Barracuda Networks felt it had no choice other than to file for a declaratory judgment in early 2007 in U.S. Federal Court to invalidate Trend Micro’s ‘600 patent and end continued legal threats against Barracuda Networks for use of the free and open source ClamAV software.

Trend Micro subsequently responded to that declaratory action and more recently, Trend Micro filed a claim with the International Trade Commission (ITC). The ITC voted to investigate the claim in December 2007. Trend Micro’s ITC claim alleges that Barracuda Networks infringes on Trend Micro’s ‘600 patent, but effectively implies that anyone using the free and open source ClamAV software at the gateway infringes the patent.

The interesting aspects of this case, from my point of view, are twofold — the patent is a classic bad software patent, very broad and totally obvious both now and at the time it was issued; and it hinges on Barracuda’s use of the free software antivirus product, ClamAV. Given Apache SpamAssassin’s prevalence in many anti-spam mail filtering appliances (including Barracuda!), this is a very worrying precedent for us — our product could be next, for some other patent troll company’s extortion scheme.

For what it’s worth, it appears this patent has long been a licensing moneyspinner for Trend. In 1997, once the patent was issued, Trend went on a spree; McAfee, Symantec and Integralis were sued, eventually buying licenses, as did Electric Mail Company. 2 years ago, Fortinet were sued and settled in their case.

I happily gave Barracuda a quote for their press release on this:

“Trend Micro’s actions are clearly an attack on free and open source software and its users, as well as on Barracuda Networks. The ‘600 patent covers a trivial method, one which was obvious to anyone skilled in the art at the time the patent was written, and should be rendered invalid as soon as possible. I hope that Barracuda Networks is successful in its attempts to defend all users from this patent shakedown.”

If you know of prior art for this patent, please head over to Barracuda’s site and provide details — helping to fend off this protection racket would be good for all of us. Barracuda say:

People should look for art dated prior to Trend Micro’s filing date of September 26, 1995. The ‘600 patent is entitled “Virus Detection And Removal Apparatus For Computer Networks.” We are interested in all material, including software, code, publications or papers, patents, communications, other media or Web sites that relate to the technology described prior to the filing date.

In particular, this prior art should show antivirus scanning on a firewall or gateway. However, many of the claims do not require virus detection at a gateway. So any material that illustrates virus scanning on a file server is also of interest.

We also believe that a product called MIMESweeper 1.0 from a company called Clearswift, Authentium, or Integralis anticipates several claims of the ‘600 patent. We have yet to locate a copy of this product and would appreciate anyone who has a copy sending it our way.

Some more coverage:

• Don Marti at LinuxWorld: ‘Regardless of the decision in this case, software patent trolls will continue to be a problem for all software companies, Eben Moglen says. “Getting them to [not operate] in your neighborhood is the best you can do.”‘

• Matt Asay at C|Net: ‘Antivirus and antispam innovation has tended to come from open source, not the large proprietary vendors. Trend Micro’s lawsuit is designed to put cash in its pocket but will end up hurting the consumer.’ (Matt led with my quote ;)

• GrokLaw: ‘Anyone using ClamAV, should Trend Micro be successful, is potentially a target.’

• Ars Technica: ‘The patent is very clearly without merit, but that hasn’t stopped Trend Micro from using it to threaten ClamAV and extort money from several companies. Situations like this demonstrate a very urgent need for patent reform and illuminate the risks posed by broad software patents, particularly in the area of security.’

Tags: antivirus, barracuda-networks, filtering, gateways, patents, prior-art, scanning, swpats, trend-micro

According to this Wired story, Google reckons spammers are giving up on spam:

a remarkable trend is underfoot, according to Brad Taylor, a staff software engineer at Google: The number of spam attempts — that is, the number of junk messages sent out by spammers — is flat, and may even be declining for the first time in years.

Actually, this is a wilful misunderstanding of what the Googler in question really said, which was that ‘attempts to spam Gmail users have been leveling off over the last year and more recently, even declining slightly’. In other words, they didn’t make an observation about the state of the spam problem on an internet-wide basis — just about the “local” situation as it pertains to Gmail. Bad reporting there, Wired.

But, in passing…

David Berlind at ZDNet recently blogged a rather grumpy response to InfoWorld coverage of CEAS 2007. He raised a very important point:

If I could say something to the author of that story, it would be that so long as any anti-spam solution is not deployed universally throughout the Internet’s e-mail system (in other words, so long as some anti-spam tech is not a standard), that anti-spam solution actually makes the spam problem worse. You read that right. Worse. Proprietary anti-spam solutions make the global spam problem worse. They are digging us deeper into the hole that the Internet is already in because everyone who makes those solutions is under the false belief that “s/he who is finally successful at filtering out all spam while allowing the legitimate mail in wins.”

Google’s blog post is a case in point: ‘we’re keeping more spam out of your inbox than ever before, so more and more, you can use Gmail for things you enjoy without even realizing that the spam filter is there most of the time.’

That’s great — but it doesn’t help anyone except Gmail. It’s a myopic view of the spam problem, and David’s point stands.

(I disagree with his later conclusion that the only way forward is for Google, MS, AOL and Yahoo! to get together and ‘commit to jointly supporting the same technical solutions’ — when the usual BigCos get together, they tend to focus on their own priorities. Take what happened back in 2005 with nofollow for blog-spam — while it helped the search giants with their own overriding priority, which was to tweak their algorithms to filter out the spam on the search results page, it did nothing to slow the spam flood itself, which has continued unabated.)

We need more open-source, and open-data, anti-spam work.

Tags: anti-spam, aol, ceas, david-berlind, filtering, google, microsoft, nofollow, spam, wired, yahoo

whoa, didn’t see that coming. Quoting Jonathan Zdziarski via jgc’s newsletter:

…The [DSPAM] project had grown to a point where it would take others – with enough free time – to bring DSPAM to the next level as a widely accepted enterprise-class solution, and [I] decided that it would be in the best interest of the project to entrust it to someone with the technical knowhow and dedication to reach these goals. Many of you are aware of my work in the past with Sensory Networks in developing a hardware-accelerated version of DSPAM (capable of supporting multi-megabit speeds in large carrier environments). I’ve spent a considerable amount of time with SN’s team over the past several years and when we initially discussed working together, they had shown to be very excited and motivated about the project.

After careful consideration and many discussions at length, I decided to allow Sensory Networks to acquire the rights to the project, and continue development on it with their own team. SN has displayed a strong commitment to the open source community and has been working closely with other leading projects such as Snort, Clam Antivirus, and SpamAssassin. They assured me that the project will remain open-source and available to all, and at the same time the project will receive exposure in commercial environments it has not seen before, as many of you have been asking for. We’ve now completed the acquisition for the project, and I’d like to encourage you to support them in helping them move forward as it grows into new areas.

More details at zdziarski.com.

Tags: anti-spam, dspam, filtering, filters, sensory-networks

Both Jeremy Zawodny and Dale Dougherty at O’Reilly Radar are expressing some pretty serious frustration with the current state of SMTP. I have to say, I’ve been feeling it too.

A couple of months back, our little server came under massive load; this had happened before, and normally in those situations it was a joe-job attack. Switching off all filtering and just collecting the targeted domain’s mail in a buffer for later processing would work to ameliorate the problem, by allowing the load to “drain”. Not this time, though.

Instead, when I turned off the filtering, the load was still too high — the massive volume of spam (and spam blowback / backscatter) was simply too much for the Postfix MTA. The MTA could not handle all the connections and SMTP traffic in time to simply collect all the data and store it in a file!

Looking into the “attack” afterwards, once the load was back under control, it looked likely that it wasn’t really an attack — it was just a volume spike. Massive SMTP load, caused by spammers increasing the volume of their output for no apparent reason. (Since then, spam volumes have been increasing still further on a nearly weekly basis.)

This is the effect of botnets — the amount of compromised hosts is now big enough to amplify spam attacks to server-swamping levels. Our server is not a big one, but it serves less than 50 users’ email I’d say; the user-to-CPU-power ratio is pretty good compared to most ISPs’ servers.

So here’s the thing. New SMTP-based methods of delivering nonspam email — whether based on DKIM, SPF, webs of trusted servers, or whatever — will not be able to operate if they have to compete for TCP connection slots with spammers, since spammers can now swamp the SMTP listener for port 25 with connections. In effect, spam will DDoS legitimate email, no matter what authentication system that legit mail uses to authenticate itself.

This, in my opinion, is a big problem.

What’s the fix? A “new SMTP” on a whole different port, where only authed email is permitted? How do you make that DoS-resistant? Ideas?

(Obviously, counting on spammers to notice or care is not a good approach.)

Tags: anti-spam, botnets, dos, email, filtering, security, smtp, spam

Via Steve Champeon’s daily links, the following spam-in-the-news stories illustrate a rising trend:

• Spam causing email delays: NZ Telecom

Huge amounts of spam are said to be responsible for delays in the email network of NZ ISP Xtra.

Several customers have vented their frustrations on an Xtra website message board saying some emails were days late, The New Zealand Herald reports.

… Record volumes of spam meant such problems would be “an unfortunate and on-going reality of the internet not specific to any provider”, he said.

Mr Bowler said Telecom had invested “tens of millions of dollars” in email and anti-spam software and worked closely with two of the world’s leading anti-spam vendors.

• Holiday spam turns e-mail to snail mail for states schools

Holiday spam e-mails are to blame for slowing message delivery to faculty and staff in schools across Kentucky …

• Channel Register: 123-Reg says sorry

“Some 123-reg customers may have experienced intermittent delays in their emails in the last two weeks. We had received a particularly high level of image-based spam attacks over a short period of time,” the Pipex subsidiary said.

• Xtra faces lawsuits over email delays

Small businesses are threatening legal action over continuing glitches with Xtra’s email service and the Consumers’ Institute says they may have a case.

Several people have contacted the Herald complaining that delays and non-deliveries of emails over the past three weeks on the Xtra network are severely affecting their businesses. …

The institute’s David Russell said home users could claim compensation for email delays if they had suffered “a real measurable loss”.

Non-commercial customers were covered by the Consumer Guarantees Act and services they paid for had to be of a “reasonable quality”.

Although it might be more difficult for small business owners, they could also have a case, Mr Russell said. “If there has been a considerable amount of money, they could consider legal action or, if the amount was smaller, they could go through the disputes tribunal.”

In other words, the DDOS-like elements of the spam problem are becoming an increasing worry; even with working spam filtering in place, the record size of zombie botnets means that spammers can now destroy organisations’ computing infrastructure, almost accidentally.

Spammers don’t care if an organisation’s infrastructure collapses while they’re sending their spam to it — they just want to maximise exposure of their spam, by any means necessary. If that requires knocking a company off the air entirely for a while, so be it.

I’m not sure what can be done about this, in terms of filtering. It may finally be time to fall back to a “side channel” of trusted, authenticated SMTP peers, and leave the spam-filled world of random email from people and organisations you don’t know to one side, as a lower-priority system which can (and will, frequently) collapse, without affecting the ‘important’ stuff. What a mess. :(

Alternatively, maybe it’s time for governments to start putting serious money into botnet-spam-related arrests and prosecution.

This has additional issues for ISPs, too, btw — I wonder if Earthlink are taking note of that Xtra lawsuit story above….

Tags: anti-spam, botnets, ddos, delays, filtering, security, spam

As all right-thinking people know by now, Challenge-response spam filtering is broken and abusive, since it simply shifts the work of filtering spam out of your email, onto innocent third-parties — either your legitimate correspondents, people on mailing lists you read, or even random people you have never heard of (due to spam blowback).

I’ve ranted about this in the past, but I’m not alone in this opinion — and frequently find myself explaining it. To avoid repeating myself, here’s a canonical collection of postings from around the web on this topic.

• Spamcop FAQ: Why are auto responders bad?:

Description: This “selfish” method of spam filtering replies to all email with a “challenge” – a message only a living person can (theoretically) respond to. There are several problems with this method which have been well known for many years.

1. Does not scale: If everyone used this method, nobody would ever get any mail.
2. Annoying: Many users refuse to reply to the challenge emails, don’t know what they are or don’t trust them.
3. Ineffective: Because of confusion about these emails, many of them are confirmed by people who did not trigger them. This results in the original malicious email being delivered.
4. Selfish: This is the problem we are mainly concerned with. By using challenge/response filtering, you are asking innumerable third parties to receive your challenge emails just so that a relatively few legitimate ones get through to the intended recipient.

• Karsten M. Self: Challenge-Response Anti-Spam Systems Considered Harmful:

C-R systems in practice achieve an unacceptably high false-positive rate (non-spam treated as spam), and may in fact be highly susceptible to false-negatives (spam treated as non-spam) via spoofing.

Effective spam management tools should place the burden either on the spammer, or, at the very least, on the person receiving the benefits of the filtering (the mail recipient). Instead, challenge-response puts the burden on, at best, a person not directly benefitting, and quite likely (read on) a completely innocent party. The one party who should be inconvenienced by spam consequences ¿ the spammer ¿ isn’t affected at all.

Worse: C-R may place the burden on third parties either inadvertantly (via spoofed sender spam or virus mail), or deliberately (see Joe Job, below). Such intrusions may even result in subversion of the C-R system out of annoyance. Many recent e-mail viruses spoof the e-mail sender, including Klez, Sobig variants, and others.

• John Levine: Challenge-response systems are as harmful as spam:

The collateral damage from widely used C/R systems, even with implementations that avoid the stupid bugs, will destroy usable e-mail. [jm: in fairness, this was written in 2003.]

Challenge systems have effects a lot like spam. In both cases, if only a few people use them they’re annoying because they unfairly offload the perpetrator’s costs on other people, but in small quantities it’s not a big hassle to deal with. As the amount of each goes up, the hassle factor rapidly escalates and it becomes harder and harder for everyone else to use e-mail at all.

• Ed Felten: A Challenging Response to Challenge-Response:

I’m skeptical of CR as a response to email. If you’re the first on your block to adopt CR, and if nobody else uses anti-spam technology, then CR might provide you some modest benefit. But it¿s hard to see how CR can be widely successful in a world where most people use some kind of spam defense.

• Jeremy Zawodny: TMDA Users Can Blow Me (heh):

If these systems are so brain-dead as to not bother adding my address to the whitelist when the user sends me e-mail, I have serious trouble understanding why anyone is using them.

Is it just me? Is this too hard to figure out?

Anyway, there’s another 5 minutes I’ll never get back. It’s too bad there’s no mail header to warn me that “this message is from a TDMA user”, because then I’d be able to procmail ‘em right to /dev/null where they belong.

Ugh.

This bullshit is not going to “solve” the spam problem, people. If that’s your solution, please let me opt out. Forever.

• Michele Neylon: Why C/R is not a good idea:

C/R slows down and impedes communication by placing unwanted barriers between you and your clients/suppliers.

If you must insist on using some form of C/R please make sure that you whitelist my address before you contact me as I will not reply to challenges.

• TidBITS policy on Challenge-Response:

We will not answer any challenges generated in response to our mailing list postings. Thus, if you’re using a challenge-response system and not receiving TidBITS, you’ll need to figure that out on your own. Also, if you send us a personal note and we receive a challenge to our reply, we may or may not respond to it, depending on our workload at the time.

• Fedora Project policy on UOL — a Brazilian ISP that uses C-R extensively:

uol.com.br uses a very broken method of anti-spam. Everytime someone sends an email message to one of their members, they send back a verification message, asking the original sender to click a link before they will allow the message through. These messages are themselves a form of spam, and the resulting back-scatter of these messages is altogether bad for the Internet, the UOL member, and all of the UOL member’s contacts. UOL is aware of the complaints against them, and they refuse to correct the issue, claiming that their members love the service.

• Matt Sergeant: C-R spam solutions:

I hate C/R systems. With a passion. I absolutely will not respond to them. They go in the trash. I don’t get them very often but I get them more and more. I think they have the potential to seriously damage email communication as we know it. And I’m not alone in this opinion.

• Richi Jennings: ‘Challenge/Response makes you a spammer.’

• BusinessWeek: Stephen Wildstrom: A Spam-Fighter More Noxious Than Spam: ‘Challenge-response filtering systems are likely to wipe out e-mails you want, too.’

• and lots more at Spamlinks.net

Phew.

Tags: abuse, anti-spam, backscatter, blowback, bounces, c-r, challenge-response, email, filtering, rants, smtp

Yay! Kudos to Richi Jennings, who’s been trumpeting the dangers of backscatter to InformationWeek recently. It’s a great article. I particularly like how it digs up this impressively off-the-mark quote:

Tal Golan, CTO, president, and founder of Sendio, maker of a challenge/response e-mail appliance used by more than 150 enterprise consumers, disagrees strongly with Jennings’s assertion that challenge-based filtering has problems. “Without question, the benefit to the whole community at large drastically outweighs that FUD [fear, uncertainty, and doubt] that’s out there in the marketplace that somehow challenge/response makes the problem worse,” he says. “The real issue is that filters don’t work. From our perspective, challenge/response is the only solution. This whole concept of backscatter is just not true. Very, very rarely do spammers forge the e-mail addresses of legitimate companies anymore.”

hahahaha. Well, since last Thursday, “very very rarely” translates as “214 MB of backscatter in my inbox”. The facts aren’t on Tal Golan’s side here…

(PS: SpamAssassin 3.2.0 will include backscatter detection.)

Tags: anti-spam, backscatter, bounces, challenge-response, filtering, funny, richi-jennings, sendio, spam

Donncha asks, is spam self-defeating?

has anyone else noticed that the new generation of gif based stock-trading spams are getting really hard to read? In the last one I had to squint and look really carefully to find out what stock was hot and a sure-buy today!

I’ve been wondering about this, too. We continually push spammers further and further from comprehensibility, since comprehensible spam is easily-filtered spam, but the spam flood doesn’t stop. In fact, spam volumes have shot up higher than ever.

My theory is that it’s a symptom of the spam side of things being a market in itself (and an inefficient, scam-heavy one at that).

IMO, the people providing the underlying products advertised in “high-end” spam – the pill-peddlers and stock pumpers — no longer control the technical details of how or where the spam is sent. Instead, they are the customers of professional spam gangs who do that, and take care of the obfuscation, filter-evasion, etc.

In other words, the pill-peddlers and scam operators are getting ripped off, too. They think their products or scams will be advertised in a comprehensible manner, in readable emails; but instead, odd, opaque 3-word messages with “cut and paste this” lines, hidden inside filter-evasion text and bits of Project Gutenberg, are what gets delivered to the victims.

I can’t imagine the clickthrough rates are exactly stellar on that. So I’d guess the spammers are responding by pushing up volumes to attempt to increase clickthrough/sales volumes. Wonder if it’s working or not?

Tags: anti-spam, filtering, obfuscation, spam

Spam: /.: New Method of Spam Filtering: ‘A simple and easily implemented scheme for combating e-mail spam has been devised by two researchers in the United States. P. Oscar Boykin and Vwani Roychowdhury of the University of California, Los Angeles use their method to exploit the structure of social networks to quickly determine whether a given message comes from a friend or a spammer. The method works for only about half of all e-mails received – but in all of those cases, it sorts the mail into the right category.’

Abstract here. It appears it classifies 53% of the emails and leaves the other 47% as undiagnosed.

The problem with this scheme is that it relies on the data in the To, From, and CC fields being accurate. Currently, there’s no means to stop spammers faking those addresses.

A trivial way to get around this filter, similarly to the other filters that trust the From address, is for a spammer to send a message using your address in both the From and To fields. Most people would include themselves in their web of trust, hence the spam would get through.

A more resilient method uses IP addresses from the Received headers in conjunction with the From address. Once you do this, you can no longer use To and CC data — and the scheme becomes pretty much similar to SpamAssassin’s auto-whitelist.

Tags: address, filtering, message, method, new, scheme, spam, spammer, trust, use