taint.org: Justin Mason’s Weblog

• MailChannels’ Ken Simpson on the new email RFCs : diffed. the explicit statement that there is no length limit to mail headers is potentially tricky
  (tags: rfcs rfc-5321 rfc-5322 email standards internet smtp)

Tags: email, internet, rfc-5321, rfc-5322, rfcs, smtp, standards

• Vint Cerf interviewed on spam, malware etc. : pretty much the EFF party line, I think: “every man for himself”. also talks about net neutrality
  (tags: vint-cerf internet filtering spam malware abuse network-neutrality anti-spam eff)

• video of a fake e-Passport being accepted by airport security reader : an e-Passport for “Elvis Aaron Presley”, no less, happily scanned by an Amsterdam passport security station. hahahaha!
  (tags: elvis funny security e-passports video via:slashdot rfid)

• Facebook adds Ireland as a Friend : ‘Dublin will be the centre for Facebook’s international operations and will provide a range of online technical, sales and operations support to Facebook’s users and customers across EMEA region.’ good news
  (tags: facebook dublin ireland web2.0 emea)

• RFC-5321 (Obsoletes: 2821) : The newest rev to the Simple Mail Transfer Protocol (via fanf)
  (tags: rfcs rfc-2821 standards internet smtp email rfc-5321)

• RFC-5322 (Obsoletes: 2822) : the newest rev to the Internet Message Format for email (via fanf)
  (tags: via:fanf rfc rfc-2822 rfc-5322 standards email internet)

Tags: abuse, anti-spam, dublin, e-passports, eff, elvis, email, emea, facebook, filtering, funny, internet, ireland, malware, network-neutrality, rfc, rfc-2821, rfc-2822, rfc-5321, rfc-5322, rfcs, rfid, security, smtp, spam, standards, via:fanf, via:slashdot, video, vint-cerf, web2.0

Del.icio.us 2.0 goes live yay! I’ve been waiting for this for yonks

10 years of Boards.ie massive ~50GB RDF/XML dump, for open crunching, to generate interesting “SIOC Semantic Web” apps

Postmaster.comcast.net how to get mail delivered successfully to Comcast, the usual stuff

Why we’ll never replace SMTP ‘The reason that e-mail is uniquely useful is that you can exchange mail with people you don’t already know. The reason that spam exists is that you can exchange mail with people you don’t already know.’ +1

“Bikes-for-Billboards” scheme exposes major planning flaws ‘what was initially hailed as “free bikes” has become one of the biggest planning controversies to hit Dublin in years.’ No shit. 70% of sites are on the Northside, rather than the richer Southside; and each bike will cost over EUR300k in ad revenue!

Rob Enderle’s page on Wikipedia detailing this analyst’s hilariously wrong pro-SCO, anti-Apple/Linux predictions over the years. John Gruber: ‘the only way it would be worthwhile for reporters to [quote him] would be if they were willing to describe him as “almost always utterly wrong”‘

Tags: ads, advertising, analysts, anti-spam, apple, bikes, billboards, blogging, boards.ie, comcast, cycling, del.icio.us, dublin, email, enderle-group, funny, fussp, guidelines, history, internet, ireland, isps, jc-decaux, john-gruber, Links, linux, mail, microsoft, oh-dear, planning, postmaster, releases, rob-enderle, scandals, sco, semantic-web, sioc, smtp, spam, tech, urban, via:archiseek, wikipedia

I’m on the technical advisory board for MailChannels, a company who make a commercial traffic-shaping antispam product, Traffic Control. Basically, you put it in front of your real MTA, and it applies “the easy stuff” — greet-pause, early-talker disconnection, lookup against front-line DNSBLs, etc. — in a massively scalable, event-driven fashion, handling thousands of SMTP connections in a single process. By taking care of 80% of the bad stuff upfront, it takes a massive load off of your backend — and, key point, off your SpamAssassin setup. ;)

Until recently, the product was for-pay and (relatively) hard to get your hands on, but as of today, they’re making it available as a download at http://mailchannels.com/download/. Apparently: “it’s free for low-volume use, but high volume users will need a license key.”

Anyway, take a look, if you’re interested. I think it’s pretty cool. (And I’m not just saying that because I’m on their tech advisory board. ;)

Tags: anti-spam, internet, mail, mailchannels, networking, plug, smtp, tcp, traffic-control, traffic-shaping

Recently, more and more people have been complaining about backscatter; its levels seem to have increased over the past few weeks.

If you’re unfamiliar with the terminology — backscatter is mail you didn’t ask to receive, generated by legitimate, non-spam-sending systems in response to spam. Here are some examples, courtesy of Al Iverson:

• Misdirected bounces from spam runs, from mail servers who “accept then bounce” instead of rejecting mail during the SMTP transaction.
• Misdirected virus/worm “OMG your mail was infected!” email notifications from virus scanners.
• Misdirected “please confirm your subscription” requests from mailing lists that allow email-based signup requests.
• Out of office or vacation autoreplies and autoresponders.
• Challenge requests from “Challenge/Response” anti-spam software. Maybe C/R software works great for you, but it generates significant backscatter to people you don’t know.

It used to be OK to send some of these types of mail — but no longer. Nowadays, due to the rise in backscatter caused by spammer/malware abuse, it is no longer considered good practice to “accept then bounce” mail from an SMTP session, or in any other way respond by mail to an unauthorized address of the mail’s senders.

Backscatter as spam delivery mechanism

I would hazard a guess that this rise is due to one of the major spam-sending botnets adopting the use of “real” sender addresses rather than randomly-generated fake ones, probably in order to evade broken-by-design Sender-Address Verification filters.

There’s an alternate theory that spammers use backscatter as a means of spam delivery — intending for the mails to bounce, in effect using the bounce as the spam delivery mechanism. Symantec’s most recent “State of Spam” report in particular highlights this.

I don’t buy it, however. Compare their own example message — here’s what the mail originally sent by the spammer to the bouncer, rendered:

And here’s what it looks like once it passes through the bouncer’s mail system:

That’s simply unreadable. There’s absolutely no way for a targeted end user to read the “payload” there…

Getting rid of it

I haven’t run into this recent spike in backscatter at all, myself, since I have a working setup that deals with it. This blog post describes it. If you’re using Postfix and SpamAssassin, it would be well worth taking a look; if you’re just using SpamAssassin and not Postfix, you should still try using the Virus Bounce Ruleset to rid yourself of various forms of unwanted bounce message.

Note that you need to set the ‘whitelist_bounce_relays’ setting to use the ruleset, otherwise its rules will not fire.

SPF

There’s a theory that setting SPF records (or other sender-auth mechanisms like DomainKeys or DKIM) on your domains, will reduce the amount of backscatter sent to your domains. Again, I doubt it.

Backscatter is being sent by old, legacy mail systems. These systems aren’t configured to take SPF into account either. When they’re eventually updated, it’s likely they’ll be fixed to simply not send “accept then bounce” responses after the SMTP transaction has completed. It’s unlikely that a system will be fixed to take SPF into account, but not fixed to stop sending backscatter noise.

It’s good advice to use these records anyway, but don’t do it because you want to stop backscatter.

What about my own bounces?

You might be worried that the SpamAssassin VBounce ruleset will block bounces sent in response to your own mail. As long as the error conditions are flagged during the SMTP transaction (as they should be nowadays), and you’ve specified your own mailserver(s) in ‘whitelist_bounce_relays’, you’re fine.

Tags: anti-spam, backscatter, email, smtp, spam, spamassassin

Back in January, I wrote about how I deal with email backscatter nowadays. Since then, I’ve made a notable tweak.

This is that I no longer reject “null-sender” traffic during the SMTP transaction. It turned out that it broke Exim’s implementation of Sender Address Verification, which performs the SAV check using a MAIL FROM of <>, rendering it indistinguishable from a bounce during the SMTP transaction.

Now, I’ve complained about SAV, but I have to be pragmatic anyway (Postel’s law and all that!) — so it was better to just allow other sites to perform SAV lookups against our server, and fix the anti-bounce stuff some other way.

The new method (below) does this, by allowing null-sender SMTP traffic just fine; it detects bounces in Postfix if they arrive via SMTP in RFC-3464 format, and bounces that slip past are then dealt with in a more CPU-intensive manner using the SpamAssassin “VBounce” ruleset (which is part of the now-released SpamAssassin 3.2.0, btw).

This increases the load, since some bounces cannot be rejected at MAIL FROM time now, and instead we have to wait ’til DATA — but CPU hasn’t been a problem recently, so this is ok.

Here are the updated instructions:

In Postfix

In my Postfix configuration, on the machine that acts as MX for my domains – edit ‘/etc/postfix/header_checks’, and add these lines:

/^Content-Type: multipart\/report; report-type=delivery-status\;/  REJECT no third-party DSNs
/^Content-Type: message\/delivery-status; /     REJECT no third-party DSNs

Edit ‘/etc/postfix/main.cf’, and ensure it contains:

header_checks = regexp:/etc/postfix/header_checks

Then run:

sudo /etc/init.d/postfix restart

This catches most of the bounces — RFC-3464-format Delivery-Status-Notification messages from other mail servers.

In SpamAssassin

As before, install the Virus-bounce ruleset and set it up. This will catch challenge-response mails, “out of office” noise, “virus scanner detected blah” crap, and bounce mails generated by really broken groupware MTAs — the stuff that gets past the Postfix front-line.

Tags: anti-spam, backscatter, bounces, email, howto, smtp, spamassassin, vbounce, virus-bounces, viruses

Like many anti-spam systems these days, SpamAssassin operates a network of spamtraps. One set of these run off traps.SpamAssassin.org, a server kindly donated by ISP Sonic.net.

Large-scale spam-trapping systems like this are generally run in quite a secretive manner, but we’re an open source project — so it may be interesting if I give some details of our setup. Here’s a potted history of how this spamtrap server has run over the years…

The beginning

The architecture was initially very simple. The MX was Postfix, delivering to the “trapper” user, which in turn ran procmail, which directly ran a perl script. This perl script then performed the trap actions, namely: DoS prevention, discarding viruses and malware, discarding backscatter bounces, extraction and cleanup of the incoming mails, then onward reporting, archival, and further distribution.

Given that this was a target for spam — and we want as much spam as possible here! — this would predictably run into load issues. Right at the beginning, back in around 2001/2002, I ran this on our shared server, where it pretty quickly caused trouble for delivery of other, more useful mail. It was around this time that Sonic kindly donated the server.

With dedicated hardware, we weren’t seeing much trouble — it was enough to just wait for the few hours for a traffic spike to pass, and the Postfix queue would then clear.

Clearing the queues

After a few months, though, this wasn’t enough — the queue would get consistently clogged, and the backlog became enough to result in the incoming spam being delayed for days before it made it from the MX to the trap archives. For a spamtrap, you want fresh spam, but not necessarily all spam — so I installed a cron job to simply clear the queue on a nightly basis. (I also had to restart the Postfix server, too, since it’d occasionally get hung and stop accepting connections on port 25, presumably due to load issues.)

IPC::DirQueue

The next level was an inability of the procmail/perl script end to process the mail fast enough for the MTA to keep up with the incoming connections, and follow-on problems, caused by load generated by the perl script impacting the MX’s activity. To work around these, I designed a new queueing backend, based around IPC::DirQueue. This allowed a new split architecture; the procmail-run perl script was extremely lightweight, delivering all inbound mail to a dirqueue and exiting quickly, allowing the MX to get back to the next inbound spam message, and the trap processing script was then split into a web of dirqueues, allowing each individual part of the trap backend pipeline to operate independently.

There were several benefits to this:

1. Since dirqueues operate as a batch-processing model, load spikes become irrelevant; the load incurred is limited by how many dequeuer processes are run.
2. The time taken in backend tasks becomes irrelevant to the MX throughput, since that is bottlenecked only by the lightweight perl script and its write speed to the “incoming” dirqueue.
3. By splitting the backend work into multiple queues, outages in the spam-reporting systems or onward forwardings become much less of a problem, since they won’t affect inbound spam, archival, outbound delivery to other reporting systems, forwards, etc.

Again, the dirqueues were cleared on a frequent basis, to discard the “spiky” traffic and ensure we were just seeing samples of the freshest spam. The dirqueues use a tmpfs as the backing storage directory, so it never hits the disk at all.

This worked pretty well for several years — from 80 megabytes of spam per day to the current level, which is around 130MB per day. However, we still occasionally saw problems from load spikes, where high load caused the traps to refuse incoming SMTP connections — purely because the load of inbound connections is too high for the Postfix MX to accept them all in a timely fashion.

qpsmtpd

Last weekend, I had a go at a project I’d been thinking of trying out for a long time — switching from Postfix to qpsmtpd. A while back, Matt Sergeant rewrote qpsmtpd to use Danga::Socket, Danga Interactive / Six Apart’s insanely scalable event-driven asynchronous socket class, as used in mogilefsd, perlbal and djabberd. This article notes that ‘two large antispam companies’ high-traffic spam traps have used this effectively since the second quarter of 2005, delivering concurrency as high as 10,000 on some occasions’, so it seemed likely to work ;)

Sure enough, results have been great… we now have a pure-perl system handling heavy volumes without breaking a sweat, certainly compared to the previous system. qpsmtpd’s plugin system was elegant, allowing me to annotate inbound spam with more details of the SMTP transaction, write plugins to deliver mail to a dirqueue directly instead of to an MTA, and do some conditional code (ie. basic “deliver this RCPT TO to this queue”) where needed.

Full details are over on the QpsmtpdSpamtrap page on the taint.org wiki, for the curious.

Tags: anti-spam, ipc-dirqueue, mx, postfix, qpsmtpd, smtp, spamassassin, spamtraps, traps.spamassassin.org

(as an anti-spam technique, at least.)

Sender-address verification, also known as callback verification, is a technique to verify that mail is being sent with a valid envelope-sender return address. It is supported by Exim and Postfix, among others.

Some view this as a useful anti-spam technique. In my opinion, it’s not.

Spam/anti-spam is an adversarial “game”. Whenever you’re considering anti-spam techniques, it’s important to bear in mind game theory, and the possible countermeasures that spammers will respond with. Before SAV became prevalent, spam was often sent using entirely fake sender data; hence the initial attractiveness of SAV. Once SAV became worth evading, the spammers needed to find “real” sender addresses to evade it. And where’s the obvious place to find real addresses? On the list of target addresses they’re spamming!

Since the spam is now sent using forged sender addresses of “real” people, when a spam bounces (as much of it does), the bounce will be sent back not to an entirely fake address, but to a spam recipient’s address.

Hence, the spam recipients now get twice as much mail from each spam run – spam aimed at them, and bounce blowback from hundreds of spams aimed at others, forged to appear to be from them.

This is the obvious “next move” in response to SAV, which is one reason why we never implemented something like it in SpamAssassin.

On top of this — it doesn’t work well enough anymore. Verizon use SAV. Have you ever heard anyone talk about how great Verizon’s spam filtering is? Didn’t think so.

(This post is a little late, given that SAV has been used for years now, but better late than never ;)

By the way, it’s worth noting that it’s still marginally acceptable to use SAV as a general email acceptance policy for your site — ie. as a way to assert that you’re not going to accept mail from people who won’t accept mail to the envelope sender address used to deliver it. Just don’t be fooled into thinking it’s helping the spam problem, or is helping anyone else but yourself.

Finally, this Sender Address Verification is different from what Sendio calls Sender Address Verification. That’s just challenge-response, which is crap for an entirely different, and much worse, set of reasons.

Tags: anti-spam, exim, game-theory, mail, mtas, postfix, sav, sender-address-verification, smtp

Both Jeremy Zawodny and Dale Dougherty at O’Reilly Radar are expressing some pretty serious frustration with the current state of SMTP. I have to say, I’ve been feeling it too.

A couple of months back, our little server came under massive load; this had happened before, and normally in those situations it was a joe-job attack. Switching off all filtering and just collecting the targeted domain’s mail in a buffer for later processing would work to ameliorate the problem, by allowing the load to “drain”. Not this time, though.

Instead, when I turned off the filtering, the load was still too high — the massive volume of spam (and spam blowback / backscatter) was simply too much for the Postfix MTA. The MTA could not handle all the connections and SMTP traffic in time to simply collect all the data and store it in a file!

Looking into the “attack” afterwards, once the load was back under control, it looked likely that it wasn’t really an attack — it was just a volume spike. Massive SMTP load, caused by spammers increasing the volume of their output for no apparent reason. (Since then, spam volumes have been increasing still further on a nearly weekly basis.)

This is the effect of botnets — the amount of compromised hosts is now big enough to amplify spam attacks to server-swamping levels. Our server is not a big one, but it serves less than 50 users’ email I’d say; the user-to-CPU-power ratio is pretty good compared to most ISPs’ servers.

So here’s the thing. New SMTP-based methods of delivering nonspam email — whether based on DKIM, SPF, webs of trusted servers, or whatever — will not be able to operate if they have to compete for TCP connection slots with spammers, since spammers can now swamp the SMTP listener for port 25 with connections. In effect, spam will DDoS legitimate email, no matter what authentication system that legit mail uses to authenticate itself.

This, in my opinion, is a big problem.

What’s the fix? A “new SMTP” on a whole different port, where only authed email is permitted? How do you make that DoS-resistant? Ideas?

(Obviously, counting on spammers to notice or care is not a good approach.)

Tags: anti-spam, botnets, dos, email, filtering, security, smtp, spam

Last week, I received an odd-looking mail from “Claims Administration Center” ClaimsAdministrationCenter /at/ enotice.info, sent to my private email address — the one listed in an image on http://jmason.org/ (it never gets spam).

The mail reads:

Mittlholtz v . International Medical Research, Inc., Sophie Chen, John Chen, and Allan Wang (”IMR Defendants”), aka Meco, et al. v. IMR, et al., case No. GIC846200.

We are requesting by order of the Court filed with the Superior Court for the County of San Diego, CA, that you post the attached Summary notice as a Public Service Announcement on your web-site.

Below is a link to the PDF Summary Notice (Note: The document is in the .PDF format. To view the documents you will need the Adobe Acrobat Reader)

http://echo.bluehornet.com/ct/ct.php?t=….

This message was intended for: webaddress@jmason.org You were added to the system January 17, 2007. For more information please follow the URL below: http://echo.bluehornet.com/subscribe/source.htm?c=…

Follow the URL below to update your preferences or opt-out: http://echo.bluehornet.com/phase2/survey1/survey.htm?CID=…

Googling for GIC846200, I find it on a cached “civil new filed cases index” page at sandiego.courts.ca.gov:

CASE NUMBER FILE DATE CATEGORY LOCATION

GIC846200 04/21/2005 A72120 - Personal Injury (Other) San Diego MECO vs INTERNATIONAL MEDICAL RESEARCH INCORPORATED

So the case exists. I have no idea who either of the parties are, however.

The URLs in the message were all web-bugged; but bluehornet seem legit in general.

The URL http://www.enotice.info/ times out. Seems to have no spam-related Google Groups hits, although there are a lot of discussions about some iffy-looking class-action suit about Google Adsense.

After quite a bit of discomfort and asking around about the reputation of both bluehornet.com and enotice.info, I eventually succumbed and clicked through. The Summary URL above, after logging my click, redirects to this PDF file, which reads:

This case, called Mittleholtz v . International Medical Research, Inc., Sophie Chen, John Chen, and Allan Wang (’IMR Defendants’), et al., case No. GIC846200, is a class action lawsuit that alleges that the IMR Defendants unlawfully distributed a product containing synthetic chemicals, the presence of which was also concealed from the public as a result of the IMR Defendants’ alleged failure to conduct any testing for adulteration by synthetic chemicals, including but not limited to diethylstilbestrol (DES) and warfarin (or coumadin), which is the active chemical in bloodthinners. Defendants deny the allegations. The Court has not formed any opinions concerning the merits of the lawsuit nor has it ruled for or against the Plaintiffs as to any of their claims. The sole purpose of this notice is to inform you of the lawsuit so that you may make an informed decision as to whether you wish to remain in or opt out of this class action.

You have legal rights and choices in this case. You can:

• Join the case. You do not have to do or pay anything to be part of this case. And, you have to accept the final result in the case.

• Exclude yourself and file your own lawsuit. If you want your own lawyer, you will have to exclude yourself as set forth below and pay your lawyer’s fees and costs.

• Exclude yourself and not sue. If you do not wish to be part of this case and do not want to bring your own lawsuit, please mail a first class letter stating that you want to be excluded from the Mittleholtz v IMR class action (Case No. GIC846200), or you may fill out the letter available at www.gilardi.com/mittleholtzsettlement. Make sure the letter has your full name, address and signature. Mail it to: PC-SPES Litigation, Class Administrator, c/o Gilardi & Co. LLC, P O Box 8060 San Rafael, CA 94912-8060 by March 23, 2007.

  *This is only a summary. For complete notice and further information go to: www.gilardi.com/mittleholtzsettlement or call the toll-free number 1-877-800-7853.

So in other words, it’s hand-targeted unsolicited, but probably not bulk, email, flogging a class-action suit about ’synthetic chemicals’ (presumably as opposed to the ‘organic’ variety). I suspect, given the phrasing in the initial mail, they probably googled for a keyword or company name, and found a hit somewhere in taint.org’s 5 years of archives — hence the PSA request.

In fact, I bet this forwarded story is what they found through Googling. Pity they didn’t include a URL for that!

Does sending legal notices like this through email not seem particularly risky, given the lack of reliability of the medium?

An odd situation, all told…

Tags: bluehornet.com, class-action, enotice.info, law, mail, smtp

There’s a common misconception about spam, email, and email authentication; Matt Cutts has been the most recent promulgator, asking ‘Where’s my authenticated email?’, in which various members of the comment thread consider this as an anti-spam question.

Here’s the thing — email these days is authenticated. If you send a mail from GMail, it’ll be authenticated using both SPF and DomainKeys. However, this alone will not help in the fight against spam.

Put simply — knowing that a mail was sent by ‘jm3485 at massiveisp.net’, is not much better than knowing that it was sent by IP address 192.122.3.45, unless you know that you can trust ‘jm3485 at massiveisp.net’, too. Spammers can (and do) authenticate themselves.

Authentication is just a step along the road to reputation and accreditation, as Eric Allman notes:

Reputation is a critical part of an overall anti-spam, anti-phishing system but is intentionally outside the purview of the DKIM base specification because how you do reputation is fundamentally orthogonal to how you do authentication.

Conceptually, once you have established an identity of an accountable entity associated with a message you can start to apply a new class of identity-based algorithms, notably reputation. … In the longer term reputation is likely to be based on community collaboration or third party accreditation.

As he says, in the long term, several vendors (such as Return Path and Habeas) are planning to act as accreditation bureaus and reputation databases, undoubtedly using these standards as a basis. Doubtless Spamhaus have similar plans, although they’ve not mentioned it.

But there’s no need to wait — in the short term, users of SpamAssassin and similar anti-spam systems can run their own personal accreditation list, by whitelisting frequent correspondents based on their DomainKeys/DKIM/SPF records, using whitelist_from_spf, whitelist_from_dkim, and whitelist_from_dk.

Hopefully more ISPs and companies will deploy outbound SPF, DK and DKIM as time goes on, making this easier. All three technologies are useful for this purpose (although I prefer DKIM, if pushed to it ;).

It’s worth noting that the upcoming SpamAssassin 3.2.0 can be set up to run these checks upfront, “short-circuiting” mail from known-good sources with valid SPF/DK/DKIM records, so that it isn’t put through the lengthy scanning process.

That’s not to say Matt doesn’t have a point, though. There are questions about deployment — why can’t I already run “apt-get install postfix-dkim-outbound-signer” to get all my outbound mail transparently signed using DKIM signatures? Why isn’t DKIM signing commonplace by now?

Tags: accreditation, anti-spam, authentication, dkim, domainkeys, email, habeas, reputation, return-path, smtp, spam, spamhaus, spf

As I’ve noted before, we still have a major problem with sites generating bounce/backscatter storms in response to forged mail — whether deliberately targeted, as a “Joe-Job”, or as a side-effect of attempts to evade over-simplistic sender address verification as seen in spam, viruses, and so on.

Sites sending these bounces have a broken mail configuration, but there are thousands remaining out there — it’s very hard to fix an old mail setup to avoid this issue. As a result, even if your mail server is set up correctly and can handle the incoming spam load just fine, a single spam run sent to other people can amplify the volume of response bounces in a Smurf-attack-style volume multiplication, acting as a denial of service. I’ve regularly had serious load problems and backlogs on my MX, due solely to these bounces.

However, I think I’ve now solved it, with only a little loss of functionality. Here’s how I did it, using Postfix and SpamAssassin.

(UPDATE: if you use the algorithm described below, you’ll block mail from people using Sender Address Verification! Use this updated version instead.)

Firstly, note that if you adopt this, you will lose functionality. Third party sites will not be able to generate bounces which are sent back to senders via your MX — except during the SMTP transaction.

However, if a message delivery attempt is run from your MX, and it is bounced by the host during that SMTP transaction, this bounce message will still be preserved. This is good, since this is basically the only bounce scenario that can be recommended, or expected to work, in modern SMTP.

Also, a small subset of third-party bounce messages will still get past, and be delivered — the ones that are not in the RFC-3464 bounce format generated by modern MTAs, but that include your outbound relays in the quoted header. The idea here is that “good bounces”, such as messages from mailing lists warning that your mails were moderated, will still be safe.

OK, the details:

In Postfix

Ideally, we could do this entirely outside Postfix — but in my experience, the volume (amplified by the Smurf attack effects) is such that these need to be rejected as soon as possible, during the SMTP transaction.

Update: I’ve now changed this technique: see this blog post for the current details, and skip this section entirely!

(If you’re curious, though, here’s what I used to recommend:)

In my Postfix configuration, on the machine that acts as MX for my domains – edit ‘/etc/postfix/header_checks’, and add these lines:

/^Return-Path: <>/                              REJECT no third-party DSNs
/^From:.*MAILER-DAEMON/                         REJECT no third-party DSNs

Edit ‘/etc/postfix/null_sender’, and add:

<>              550 no third-party DSNs

Edit ‘/etc/postfix/main.cf’, and ensure it contains these lines:

header_checks = regexp:/etc/postfix/header_checks
smtpd_sender_restrictions = check_sender_access hash:/etc/postfix/null_sender

(If you already have an ’smtpd_sender_restrictions’ line, just add ‘check_sender_access hash:/etc/postfix/null_sender’ to the end.) Finally, run:

sudo postmap /etc/postfix/null_sender
sudo /etc/init.d/postfix restart

This catches most of the bounces — RFC-3464-format Delivery-Status-Notification messages from other mail servers.

In SpamAssassin

Install the Virus-bounce ruleset. This will catch challenge-response mails, “out of office” noise, “virus scanner detected blah” crap, and bounce mails generated by really broken groupware MTAs — the stuff that gets past the Postfix front-line.

Once you’ve done these two things, that deals with almost all the forged-bounce load, at what I think is a reasonable cost. Comments welcome…

Tags: anti-spam, backscatter, bounces, email, howto, smtp, spamassassin, vbounce, virus-bounces, viruses

Here’s a great presentation from Joe St Sauver presented at the London Action Plan meeting recently: Infected PCs Acting As Spam Zombies: We Need to Cure the Disease, Not Just Suppress the Symptoms

Some key points in brief:

Despite all our ongoing efforts: the spam problem continues to worsen, with nine out of every ten emails now spam; spam volume has increased by 80% over just the past few months and users face a constantly morphing flood of malware trying to take over their computers. Bottom line: we’re losing the war on spam.

The root cause of today’s spam problems is spam zombies, with 85% of all spam being delivered via spam zombies.

The spam zombie problem grows worse every day (with over ninety one million new spam zombies per year)

Users don’t, won’t, or can’t clean up their infected PCs; and ISPs can’t be expected to clean up their infected customers’ PCs.

Filtering port 25 and doing rate limiting is like giving cough syrup to someone with lung cancer — it may suppress some overt symptoms but it doesn’t cure the underlying disease.

Filtered and rate-limited spam zombies CAN still be used for many, many OTHER bad things, and they represent a huge problem if left to languish in a live infected state.

Joe’s take — “we’re in the middle of a worldwide cyber crisis”. I agree. He suggests a new strategy:

It is common for universities to produce and distribute a one-click clean-up-and-secure CD for use by their students and faculty. It’s now time for our governments to produce and distribute an equivalent disk for everyone to use.

I agree the existing schemes are clearly not working; this is an interesting suggestion. Read/listen to the presentation in full for more details; pick up PDF, PPT and video here.

Tags: anti-spam, botnets, ddos, internet, joe-st-sauver, london-action-plan, security, smtp, windows, zombies

As all right-thinking people know by now, Challenge-response spam filtering is broken and abusive, since it simply shifts the work of filtering spam out of your email, onto innocent third-parties — either your legitimate correspondents, people on mailing lists you read, or even random people you have never heard of (due to spam blowback).

I’ve ranted about this in the past, but I’m not alone in this opinion — and frequently find myself explaining it. To avoid repeating myself, here’s a canonical collection of postings from around the web on this topic.

• Spamcop FAQ: Why are auto responders bad?:

Description: This “selfish” method of spam filtering replies to all email with a “challenge” - a message only a living person can (theoretically) respond to. There are several problems with this method which have been well known for many years.

1. Does not scale: If everyone used this method, nobody would ever get any mail.
2. Annoying: Many users refuse to reply to the challenge emails, don’t know what they are or don’t trust them.
3. Ineffective: Because of confusion about these emails, many of them are confirmed by people who did not trigger them. This results in the original malicious email being delivered.
4. Selfish: This is the problem we are mainly concerned with. By using challenge/response filtering, you are asking innumerable third parties to receive your challenge emails just so that a relatively few legitimate ones get through to the intended recipient.

• Karsten M. Self: Challenge-Response Anti-Spam Systems Considered Harmful:

C-R systems in practice achieve an unacceptably high false-positive rate (non-spam treated as spam), and may in fact be highly susceptible to false-negatives (spam treated as non-spam) via spoofing.

Effective spam management tools should place the burden either on the spammer, or, at the very least, on the person receiving the benefits of the filtering (the mail recipient). Instead, challenge-response puts the burden on, at best, a person not directly benefitting, and quite likely (read on) a completely innocent party. The one party who should be inconvenienced by spam consequences ¿ the spammer ¿ isn’t affected at all.

Worse: C-R may place the burden on third parties either inadvertantly (via spoofed sender spam or virus mail), or deliberately (see Joe Job, below). Such intrusions may even result in subversion of the C-R system out of annoyance. Many recent e-mail viruses spoof the e-mail sender, including Klez, Sobig variants, and others.

• John Levine: Challenge-response systems are as harmful as spam:

The collateral damage from widely used C/R systems, even with implementations that avoid the stupid bugs, will destroy usable e-mail. [jm: in fairness, this was written in 2003.]

Challenge systems have effects a lot like spam. In both cases, if only a few people use them they’re annoying because they unfairly offload the perpetrator’s costs on other people, but in small quantities it’s not a big hassle to deal with. As the amount of each goes up, the hassle factor rapidly escalates and it becomes harder and harder for everyone else to use e-mail at all.

• Ed Felten: A Challenging Response to Challenge-Response:

I’m skeptical of CR as a response to email. If you’re the first on your block to adopt CR, and if nobody else uses anti-spam technology, then CR might provide you some modest benefit. But it¿s hard to see how CR can be widely successful in a world where most people use some kind of spam defense.

• Jeremy Zawodny: TMDA Users Can Blow Me (heh):

If these systems are so brain-dead as to not bother adding my address to the whitelist when the user sends me e-mail, I have serious trouble understanding why anyone is using them.

Is it just me? Is this too hard to figure out?

Anyway, there’s another 5 minutes I’ll never get back. It’s too bad there’s no mail header to warn me that “this message is from a TDMA user”, because then I’d be able to procmail ‘em right to /dev/null where they belong.

Ugh.

This bullshit is not going to “solve” the spam problem, people. If that’s your solution, please let me opt out. Forever.

• Michele Neylon: Why C/R is not a good idea:

C/R slows down and impedes communication by placing unwanted barriers between you and your clients/suppliers.

If you must insist on using some form of C/R please make sure that you whitelist my address before you contact me as I will not reply to challenges.

• TidBITS policy on Challenge-Response:

We will not answer any challenges generated in response to our mailing list postings. Thus, if you’re using a challenge-response system and not receiving TidBITS, you’ll need to figure that out on your own. Also, if you send us a personal note and we receive a challenge to our reply, we may or may not respond to it, depending on our workload at the time.

• Fedora Project policy on UOL — a Brazilian ISP that uses C-R extensively:

uol.com.br uses a very broken method of anti-spam. Everytime someone sends an email message to one of their members, they send back a verification message, asking the original sender to click a link before they will allow the message through. These messages are themselves a form of spam, and the resulting back-scatter of these messages is altogether bad for the Internet, the UOL member, and all of the UOL member’s contacts. UOL is aware of the complaints against them, and they refuse to correct the issue, claiming that their members love the service.

• Matt Sergeant: C-R spam solutions:

I hate C/R systems. With a passion. I absolutely will not respond to them. They go in the trash. I don’t get them very often but I get them more and more. I think they have the potential to seriously damage email communication as we know it. And I’m not alone in this opinion.

• Richi Jennings: ‘Challenge/Response makes you a spammer.’

• BusinessWeek: Stephen Wildstrom: A Spam-Fighter More Noxious Than Spam: ‘Challenge-response filtering systems are likely to wipe out e-mails you want, too.’

• and lots more at Spamlinks.net

Phew.

Tags: abuse, anti-spam, backscatter, blowback, bounces, c-r, challenge-response, email, filtering, rants, smtp

Like a few other anti-spammers, I found myself under a hitherto-unprecedented level of spam blowback this weekend. Disappointingly, there are still thousands of SMTP servers configured to send bounce messages in response to spam.

Even with the anti-bounce ruleset for SpamAssassin, the volume was so great that our creaky old server had a lot of difficulty keeping up — once the messages got to SpamAssassin, the load issues had already been created. Also, Postfix’s anti-spam features really weren’t designed to deal with blowback.

While attempting to take some shortcuts in the setup on our server to deal with this, a great idea occurred to me — why not come up with an app that uses Amazon EC2 to flexibly provision enough server power and bandwidth to pre-filter the SMTP traffic for an MX under attack?

I’m basically thinking of qpsmtpd, with SpamAssassin and/or other antispam blobs active, running in an Amazon EC2 server image. Multiple images can be brought up, and added to the attacked domain’s MX record at an equal priority, to take load off the main (overloaded) MX.

Now to cogitate a little — details to follow…

Tags: amazon, anti-spam, aws, ec2, qpsmtpd, s3, smtp

Update: as of Oct 2007, this advice is obsolete. The Spamcop algorithms have been greatly improved, as far as I and others can tell.

I’ve been hearing increasing reports of false positives using bl.spamcop.net.

One today spurred me to check out exactly how many times it I’m seeing it misfiring on nonspam in my own mail collection. The results have been pretty astonishing.

In my nonspam collection, it fired on 1043 messages out of 8415 in July; 12.4% of the mail. It gets worse for August, though — 884 messages out of 3729 since the start of August. That’s a staggering 23% of my nonspam mail this month. ;)

Most of that is due to the listings of GMail and Yahoo! Groups, both of which seem to have been listed for large swathes of the past month and a half.

Now, an important point — it can work pretty well as a single input to a scoring system, like Spamcop itself or SpamAssassin. In fact, I didn’t lose any mail as a result of those listings; SpamAssassin assigns only 1.5 points to the RCVD_IN_BL_SPAMCOP_NET rule, so it’s easily corrected by other rules.

However, people using it to block or reject spam outright, or who’ve changed the score of the RCVD_IN_BL_SPAMCOP_NET rule, need to turn that off ASAP — as they are losing mail.

Tags: anti-spam, blocklists, dnsbls, smtp, spamcop

Apparently, spammers are now exploiting a hole, or holes, in multiple PHP scripts which use the mail() API.

The holes are described at the SecurePHP wiki; basically, the script author inserts CGI fields directly into a message template without stripping newlines, and this allows attackers to create new headers, take over the message body, and generally take over the mail message and destinations entirely.

Funnily enough, these are the same holes Ronald F. Guilmette and I found in FormMail 1.9, and described in our Jan 2002 advisory Anonymous Mail Forwarding Vulnerabilities in FormMail 1.9 (PDF) on page 10, Exploitation of email and realname CGI Parameters. Ah, plus ca change…

Worth noting that perl’s venerable taint checking would have spotted these, if it were used.

Tags: email, exploits, formmail, php, security, smtp, vulnerabilities

Spam: eWeek recently published an article entitled ‘Spammers’ New Tactic Upends DNS’ , which notes that:

One .. technique finding favor with spammers involves sending mass mailings in the middle of the night from a domain that has not yet been registered. After the mailings go out, the spammer registers the domain early the next morning.

By doing this, spammers hope to avoid stiff CAN-SPAM fines through minimal exposure and visibility with a given domain. The ruse, they hope, makes them more difficult to find and prosecute.

The scheme, however, has unintended consequences of its own. During the interval between mailing and registration, the SMTP servers on the recipients’ networks attempt Domain Name System look-ups on the nonexistent domain, causing delays and timeouts on the DNS servers and backups in SMTP message queues.

This had me stumped when I read it, since an email from a nonexistent domain is a pretty reliable spamsign (it’s used in the NO_DNS_FOR_FROM rule in SpamAssassin, for example, which hits about 2% of spam), has been a rule in the default ruleset for several years, and there’s no sign of that behaviour in our spam traps.

After some discussion, Suresh Ramasubramanian came up with this explanation of what’s really happening:

Verisign now allows immediate (well, within about 10 minutes) updates of .com/.net zones (also same for .biz) while whois data is still updated once or twice a day. That means if spammer registers (a) new domain he’ll be able to use it immediatly (sic) and it’ll not yet show up in whois (and so not be immediatly identifiable to spam reporting tools) - and spammers are in fact using this “feature” more and more!

That does sound a much more likely explanation, and matches what’s been seen in the traps.

So: WHOIS, not DNS.

Tags: article, dns, domain, eweek, explanation, rule, smtp, spam, spammer, spammers

Spam: Bram shares a spam-filtering tip — ‘most of the viruses I get have a Message-Id tacked on by the local mailserver. A little bit of messing with procmail and suddenly my junk mail level is under control.’

This is what the SpamAssassin rule MSGID_FROM_MTA_SHORT does. It gets:

  4.432   6.7680   0.0560    0.992   0.94    3.67  MSGID_FROM_MTA_SHORT

6.7680% of spam is hit, but so is 0.0560% of ham mail — which makes it 99.2% accurate. By default in 2.6x, it gets a score of 3.67 points.

There’s a lot of divergence between people’s corpora — for instance, I currently have no ham mails that hit this, so it’s 100% accurate for my current mail collection; but some other people have an 80% hit-rate.

This is because some large-scale legitimate mass-mailers — for no apparent reason — also omit the Message-ID when they send the message across the internet. This isn’t quite a contravention of RFC 2822, but that RFC strongly recommends using the header:

Though optional, every message SHOULD have a ‘Message-ID:’ field.

(see RFC 2119 for what ‘SHOULD’ means — it’s a strong recommendation.)

The moral for legit senders: make sure you read the RFCs before you start sending SMTP; otherwise you’ll look like a spammer.

The moral for spamfilter developers: watch out for the legit bulk mail senders; some of them do really bizarre things with SMTP. ;)

Tags: from, ham, legit, mail, msgid, mta, rfc, short, smtp, spam, tip

Spam: Been thinking about a distributed ‘web of trust’ approach to fighting spam.

• Kevin Burton conversing with Raph about a web-based trust metric.
• Bram writes about using (non-distributed) trust metrics against spam. With code ;)
• Trust Management on the WWW, by none other than Rohit Khare and Adam Rifkin!
• Trust Networks on the Semantic Web, a paper.

Combine those with another key point — that we do not need PKI, crypto, or any other changes to identify senders in current SMTP — and it could be done today, I think.

Why we don’t need crypto to identify an SMTP sender

Every email message delivered via SMTP across the internet will contain these headers:

• the From line
• one or more Received headers

Traditionally, whitelisting uses just the From line, which is vulnerable to spoofing. SpamAssassin used this up to version 2.3x. Spammers started spoofing mails where ‘From’ was the same as ‘To’, and since most people had themselves in the whitelist, that worked. boo.

In 2.3x or 2.4x, we added code to extract the IP addresses from the Received headers, and use a combined token — ( from_address, ip_address ) — as the sender’s address.

(In fact, we use just the top 24 bits of each IP to deal with situations like DHCP or dialup pools, where a relay may get a different IP every now and again. That’s close enough, at least.)

This is much harder to forge without doing a full-scale TCP spoofing attack; which is why the SpamAssassin auto-whitelist generally works well.

So basically, to identify someone strongly enough to provide a spam fix in plain old vanilla current SMTP, gen up a string containing their ‘From’ address, along with all the /24 masks of the IP addresses found in the ‘Received’ headers.

Remove your relays’ IP addresses, and you have an unspoofable ID for that person’s SMTP traffic. Any spammer who wants to spoof that, will have to compromise their mail server (or a server in the same /24). That’s not cost-effective for spamming.

Note that whitelisting based on that is effectively what the SpamAssassin auto-whitelist does. But for that to be more useful than the AWL, it has to extend over the internet to those people your friends haven’t corresponded with yet; ie. it’s got to be distributed.

(If you would like to comment on this scheme, I’d prefer if you could post comments at this QuickTopic forum.)

Tags: address, code, crypto, line, received, sender, smtp, spam, trust, web

Been thinking about a distributed ‘web of trust’ approach to fighting spam.

• Kevin Burton conversing with Raph about a web-based trust metric.
• Bram writes about using (non-distributed) trust metrics against spam. With code ;)
• Trust Management on the WWW, by none other than Rohit Khare and Adam Rifkin!
• Trust Networks on the Semantic Web, a paper.

Combine those with another key point — that we do not need PKI, crypto, or any other changes to identify senders in current SMTP — and it could be done today, I think.

Why we don’t need crypto to identify an SMTP sender

Every email message delivered via SMTP across the internet will contain these headers:

• the From line
• one or more Received headers

Traditionally, whitelisting uses just the From line, which is vulnerable to spoofing. SpamAssassin used this up to version 2.3x. Spammers started spoofing mails where ‘From’ was the same as ‘To’, and since most people had themselves in the whitelist, that worked. boo.

In 2.3x or 2.4x, we added code to extract the IP addresses from the Received headers, and use a combined token — ( from_address, ip_address ) — as the sender’s address.

(In fact, we use just the top 24 bits of each IP to deal with situations like DHCP or dialup pools, where a relay may get a different IP every now and again. That’s close enough, at least.)

This is much harder to forge without doing a full-scale TCP spoofing attack; which is why the SpamAssassin auto-whitelist generally works well.

So basically, to identify someone strongly enough to provide a spam fix in plain old vanilla current SMTP, gen up a string containing their ‘From’ address, along with all the /24 masks of the IP addresses found in the ‘Received’ headers.

Remove your relays’ IP addresses, and you have an unspoofable ID for that person’s SMTP traffic. Any spammer who wants to spoof that, will have to compromise their mail server (or a server in the same /24). That’s not cost-effective for spamming.

Note that whitelisting based on that is effectively what the SpamAssassin auto-whitelist does. But for that to be more useful than the AWL, it has to extend over the internet to those people your friends haven’t corresponded with yet; ie. it’s got to be distributed.

(If you would like to comment on this scheme, I’d prefer if you could post comments at this QuickTopic forum.)

Tags: address, code, crypto, line, received, sender, smtp, spam, trust, web

Spam: I’ve been meaning to collate a page about why I don’t like greylisting. My previous posting is relatively useful, but it needs an update, so here it is:

First off, every single message is delayed until a database match is found for the combination of sending IP, envelope-from and envelope-to. As Alan Leghart pointed out, ‘So…we punish everyone in the world, and hope that a delay of one or more hours is considered ‘acceptable’? Maybe some people already expect a mail to take several hours to reach a recipient. In that case, you need to fix your mail server.’

Secondly, large mailing lists that use VERP (generating keyed From addresses for each mail for good bounce-handling) will require manual whitelisting for each list, or each host.

Yahoo! Groups, for example,
uses VERP for all its lists, and also will not retry delivery if the first attempt fails.

There’s even buggy SMTP servers that do not support retrying, believe it or not.

(Once again, as for many spamfilter designs, the unusual SMTP clients are the ‘edge cases’ that cause the most trouble.)

Manual whitelisting == work == what spam filtering is trying to reduce == bad.

Thirdly, and most seriously, it assumes spammers would never introduce retries into their spam-tools if it took off. Tempfailing, what this is based on, is effective right now because spamtools don’t retry. But every proposed spam solution has to consider what would happen if every server admin in the world implements it, and spammers then want to subvert it.

For a spamtool to retry, it just needs to track 4xx responses, and if it encounters one, save these items of data:

• From, To addrs and HELO string used
• proxy IP used (btw proxies are almost never shut down successfully, so the spammer can generally assume this can be reused next time)
• random seed used to generate random hashbuster tokens etc., so the body text matches

That’s really not a lot of data — 64 bytes per address that requires a retry. Then, an hour or more later, do the retry.

So, IMO, ‘greylisting‘ will work fine in the short term, until it becomes reasonably common — then the spamtool developers will start adding retry code.

Then we’re back to square one — except some legit mail takes much longer to get delivered, and the bandwidth wasted by spam has doubled, due to all those retrying spams. That’s not really progress.

Tags: collate, mail, message, page, server, smtp, spam, update, verp, world

I’ve been meaning to collate a page about why I don’t like greylisting. My previous posting is relatively useful, but it needs an update, so here it is:

First off, every single message is delayed until a database match is found for the combination of sending IP, envelope-from and envelope-to. As Alan Leghart pointed out, ‘So…we punish everyone in the world, and hope that a delay of one or more hours is considered ‘acceptable’? Maybe some people already expect a mail to take several hours to reach a recipient. In that case, you need to fix your mail server.’

Secondly, large mailing lists that use VERP (generating keyed From addresses for each mail for good bounce-handling) will require manual whitelisting for each list, or each host.

Yahoo! Groups, for example,
uses VERP for all its lists, and also will not retry delivery if the first attempt fails.

There’s even buggy SMTP servers that do not support retrying, believe it or not.

(Once again, as for many spamfilter designs, the unusual SMTP clients are the ‘edge cases’ that cause the most trouble.)

Manual whitelisting == work == what spam filtering is trying to reduce == bad.

Thirdly, and most seriously, it assumes spammers would never introduce retries into their spam-tools if it took off. Tempfailing, what this is based on, is effective right now because spamtools don’t retry. But every proposed spam solution has to consider what would happen if every server admin in the world implements it, and spammers then want to subvert it.

For a spamtool to retry, it just needs to track 4xx responses, and if it encounters one, save these items of data:

• From, To addrs and HELO string used
• proxy IP used (btw proxies are almost never shut down successfully, so the spammer can generally assume this can be reused next time)
• random seed used to generate random hashbuster tokens etc., so the body text matches

That’s really not a lot of data — 64 bytes per address that requires a retry. Then, an hour or more later, do the retry.

So, IMO, ‘greylisting‘ will work fine in the short term, until it becomes reasonably common — then the spamtool developers will start adding retry code.

Then we’re back to square one — except some legit mail takes much longer to get delivered, and the bandwidth wasted by spam has doubled, due to all those retrying spams. That’s not really progress.

Tags: collate, mail, message, page, server, smtp, spam, update, verp, world

Given all the fuss over blocking dynamic IPs due to spam, I’ve long sent outgoing SMTP via my server (which lives on a static IP). I download my mail from that using fetchmail over an SSH tunnel, and have done for a while. It’s very reliable, and that way it really doesn’t matter where I download from — quite neat. Also means I don’t have to futz with SMTP AUTH, IMAP/SSL, Certifying Authorities, or any of the other hand-configured complex PKI machinery required to use SSL for authentication.

However, I’ve been using plain old SMTP for outgoing traffic, by just poking a hole in the access db for the IP I’m on. A bit messy and generally not-nice.

So I decided to make it sensible and deliver using SMTP-in-an-SSH-tunnel. In the same SSH tunnel, in fact ;) With Postfix, it turned out very easy — here’s how to do it:

Add this option to the SSH commandline in the SSH tunneling script (I’m presuming you have one ;):

-L 8025:127.0.0.1:25

That’ll port-forward port 25 on the remote system to port 8025 on localhost, so that if a connection is made to port 8025 on localhost, it’ll talk to port 25 on the remote host. Std SSH tunneling there.

Now for Postfix — add this to /etc/postfix/main.cf:

default_transport = smtp:localhost:8025

This means that Postfix will always use SMTP to localhost on port 8025 for any non-local deliveries.

Run service postfix reload (cough, Red Hat-ism) and that’s it! A whole lot easier than I was expecting… Postfix rocks.

Tags: download, ips, localhost, port, postfix, smtp, spam, ssh, ssl, tunnel

Spam: SMTP Sender Authentication, by David Jeske of Y! Groups (pointer from Jeremy.

Schemes similar to this — calling back to a sending server to verify that a mail was really sent via that host — have been proposed before in several venues, the most high-profile and public being the ASRG list. Here is a message I sent to that list in April 2003 discussing a few of those schemes:

• J C Lawrence’s ‘forward chained digital signatures’ on Received headers
• William at elan.net’s ‘complex callback verification requirying full message tracking server functionality with dns extensions’
• Russ Nelson’s Q249
• Our own ‘porkhash’

I still like this style of system, I think, but in terms of deployability and simplicity, I’m supporting Sender-Permitted From for now — which similarly forces senders to use registered relays for a given SPF-supporting domain, but using DNS as the protocol and IP addresses as the hard-to-forge identity component.

Another bonus of SPF is that it’s simple, easy to implement, has *running code* out there now, and is being pushed strongly by a pragmatic and sane driving person (in the form of Meng Weng Wong). It’s not always easy in the anti-spam field to find a solution like that ;)

BTW, SPF also, similarly, breaks envelope sender forging. However, I agree, this is one egg that has to be broken to help stop spam (or at least force spammers to use their own domains and IPs.)

Tags: authentication, list, mail, message, pointer, schemes, sender, server, smtp, spam, spf

SMTP Sender Authentication, by David Jeske of Y! Groups (pointer from Jeremy.

Schemes similar to this — calling back to a sending server to verify that a mail was really sent via that host — have been proposed before in several venues, the most high-profile and public being the ASRG list. Here is a message I sent to that list in April 2003 discussing a few of those schemes:

• J C Lawrence’s ‘forward chained digital signatures’ on Received headers
• William at elan.net’s ‘complex callback verification requirying full message tracking server functionality with dns extensions’
• Russ Nelson’s Q249
• Our own ‘porkhash’

I still like this style of system, I think, but in terms of deployability and simplicity, I’m supporting Sender-Permitted From for now — which similarly forces senders to use registered relays for a given SPF-supporting domain, but using DNS as the protocol and IP addresses as the hard-to-forge identity component.

Another bonus of SPF is that it’s simple, easy to implement, has *running code* out there now, and is being pushed strongly by a pragmatic and sane driving person (in the form of Meng Weng Wong). It’s not always easy in the anti-spam field to find a solution like that ;)

BTW, SPF also, similarly, breaks envelope sender forging. However, I agree, this is one egg that has to be broken to help stop spam (or at least force spammers to use their own domains and IPs.)

Tags: authentication, list, mail, message, pointer, schemes, sender, server, smtp, spf

Spam: A nasty new development — spammers are now exploiting closed relays to send spam, by brute-force attacking their SMTP AUTH interfaces. SMTP AUTH is a system used to allow legitimate mail server users to send outgoing mail securely, by authenticating them first. ( sample documentation here.)

This ROKSO file indicates one spammer’s modus operandi:

These relays were abused using SMTP AUTH. That is, the spammer supplied a valid username/password pair to the server, was authenticated, and therefore granted permission to send mail anywhere. Such attacks are therefore successful only when weak passwords are used. This spamhaus constantly scans the net to find abusable servers to use in subsequent spam runs. All brands of servers (sendmail, exchange, mdaemon, rockcliffe, etc) are equally targeted, as long as they support SMTP AUTH. The attacker tries several username/password pairs - such as with ‘admin/admin’ - following a certain pattern and hoping to find a combination that lets him in.

An analysis done in july 2003 has shown that a total of 276 combinations are attempted (of course new ones can have been added in the meanwhile): Usernames: webmaster, admin, root, test, master, web, www, administrator, backup, server, data, abc each with the following passwords: username, username12, username123, 1, 111, 123, 1234, 12345, 123456, 1234567, 12345678, 654321, 54321, 00000000, 88888888, admin, root, pass, passwd, password, super, !@#$%^&* as well as with a blank password.

MDaemon users beware! The account creation tool of recent versions of MDaemon defaults the password to the account name. If the default is accepted, the account will be open to be exploited by this spamhaus.

Incredible. There’s no way at the SMTP/IP level to tell that this relay was compromised; blacklisting will definitely cause collateral damage in response; so content analysis is pretty much necessary, as far as I can see.

And in another worrying development: it turns out that the latest Outlook worm, W32.Swen, doesn’t bother trying to randomly generate usernames etc. or send via SMTP directly. Instead, it asks the user for their username, password and SMTP server!

Tags: account, admin, auth, mail, mdaemon, password, server, smtp, spam, username

A nasty new development — spammers are now exploiting closed relays to send spam, by brute-force attacking their SMTP AUTH interfaces. SMTP AUTH is a system used to allow legitimate mail server users to send outgoing mail securely, by authenticating them first. ( sample documentation here.)

This ROKSO file indicates one spammer’s modus operandi:

These relays were abused using SMTP AUTH. That is, the spammer supplied a valid username/password pair to the server, was authenticated, and therefore granted permission to send mail anywhere. Such attacks are therefore successful only when weak passwords are used. This spamhaus constantly scans the net to find abusable servers to use in subsequent spam runs. All brands of servers (sendmail, exchange, mdaemon, rockcliffe, etc) are equally targeted, as long as they support SMTP AUTH. The attacker tries several username/password pairs - such as with ‘admin/admin’ - following a certain pattern and hoping to find a combination that lets him in.

An analysis done in july 2003 has shown that a total of 276 combinations are attempted (of course new ones can have been added in the meanwhile): Usernames: webmaster, admin, root, test, master, web, www, administrator, backup, server, data, abc each with the following passwords: username, username12, username123, 1, 111, 123, 1234, 12345, 123456, 1234567, 12345678, 654321, 54321, 00000000, 88888888, admin, root, pass, passwd, password, super, !@#$%^&* as well as with a blank password.

MDaemon users beware! The account creation tool of recent versions of MDaemon defaults the password to the account name. If the default is accepted, the account will be open to be exploited by this spamhaus.

Incredible. There’s no way at the SMTP/IP level to tell that this relay was compromised; blacklisting will definitely cause collateral damage in response; so content analysis is pretty much necessary, as far as I can see.

And in another worrying development: it turns out that the latest Outlook worm, W32.Swen, doesn’t bother trying to randomly generate usernames etc. or send via SMTP directly. Instead, it asks the user for their username, password and SMTP server!

Tags: account, admin, auth, mail, mdaemon, password, server, smtp, spam, username

Every now and again, someone says ‘to fix spam, we must ditch SMTP and start all over again’. Eric Rescorla describes why this is not the case.

Great blog — I think I’ll add that to my list. (found via Ed Felten.)

Tags: blog, case, ditch, fix, list, smtp, someone, spam

just got back from a super-quick booze-soaked weekend visit to Ben in SF. It was so good to visit a city once again, and get the opportunity to paint the town red, hit the bars, eat in plentiful cheap restaurants, and generally enjoy city life (which I’ve been missing massively since the move from Dublin). But now back in post-suburban Irvine to cope with the hangover.

Also got to meet up with Komal, one of my co-workers up there — which was cool. Unfortunately it was a super-speedy weekend whistle-stop tour though, so having a good social meet-up with all the guys will have to wait until the next visit. ;)

Net: ‘The Canadian scientists who broke the genetic code for SARS … say they couldn’t have done it without the Internet. … The key to that collaboration was ordinary e-mail‘.

It also turns out the ProMED mailing list was the central point at which SARS reports were collated in the early stages, even despite evasion and cover-up by the Chinese state.

So there you go — as usual, SMTP is the killer app — or in this case, a life-saving app! All the more reason to figure out ways to deal with spam and return SMTP to its top spot in the protocol pantheon.

Good thing the FTC Spam Forum went so well, then. Sounds like there was unprecedented agreement between the non-spam folks, clear understanding of the issues by quite a few of the Washington denizens, and maybe even some good footage of the other side digging holes for themselves.

Health: US, Asian Airlines Disagree on SARS. Me, I just wish the airlines would stop being so bloody cheap, and bring in more fresh air rather than recirculating. ;)

Date: Sun, 04 May 2003 12:20:16 -0400
From: STEPHEN JONES (spam-protected)
To: (spam-protected) (spam-protected)
Subject: Internet is a good thing says Steve Jones clone

Internet played a key role in decoding SARS genome, scientists say

DENNIS BUECKERT

OTTAWA (CP) - The Canadian scientists who broke the genetic code for SARS just weeks after the disease appeared say they couldn’t have done it without the Internet.

Scientists from the Michael Smith Genome Sciences Centre of the B.C. Cancer Agency say their achievement relied on rapid communication with scientists around the world. The key to that collaboration was ordinary e-mail, said Steven Jones of the Vancouver-based research agency in a teleconference Thursday sponsored by Science magazine.

“Within a day of us having a press release announcing our participation in the sequencing we had an amazing amount of e-mail from scientists all around the world,” Jones said.

As soon as the sequence was decoded, the B.C. researchers posted it on the Internet.

“People were, within minutes of that, able to download the sequence and analyse it in their own laboratories and their own computers,” Jones said.

“The Internet has had a profound impact on how this data has been shared and how scientists have collaborated.”

A short time later, researchers at the Atlanta Centers for Disease Control published the sequence of a coronavirus taken from another SARS patient.

The genetic coding for the two viruses were virtually identical, boosting confidence that the coronavirus was in fact the causal agent.

Now both sequences are posted on the World Wide Web for the benefit of researchers in many countries racing to find a reliable test for SARS, and a vaccine to prevent it.

Scientists say the speed of the decoding was amazing.

The first reports of the new disease came from China in November, and on March 13 cases were reported in Toronto and Vancouver. The sequences were posted on the net on April 15.

By contrast, it took years to identify the agents behind diseases like AIDS and hepatitis C.

Mel Crajdon of the B.C. Centre for Disease Control said all evidence points to the coronavirus as being the cause of SARS, despite some seemingly contradictory findings.

Earlier this week Frank Plummer, who heads the National Microbiology Laboratory in Winnipeg, said he was puzzled by the number of people who show evidence of the SARS coronavirus but not symptoms of the disease.

Crajdon suggested the apparent anomaly is due to imperfect understanding of how the disease presents itself, as well as lack of reliable tests for the presence of the virus.

“I’m not surprised by the results that have been obtained to date and I think that they will rapidly improve,” he said.

More than 5,400 cases of SARS have been diagnosed worldwide, with at least 394 deaths. In Canada, there have been 23 deaths, all in the Toronto area.

• - -

On the Net:

SARS sequences: http://sciencemag.org/features/data/sars

SARS data: http://aaas.org

SARS Comments: http://eurekalert.org

Tags: app, city, code, internet, sars, smtp, spam, thing, visit, weekend

Some folks reckon that mailservers should have reverse DNS — in other words, that the SMTP server should have a fully-valid forward-to-reverse mapping for its address, to cut down on spam and forgeries. All well and good.

Some other folks reckon that filtering on it is therefore a good way to cut down on spam.

It’s a nice idea, apart from 2 things:

• filtering based on this suffers the same problem some DNSBLs have: a false positive hurts the user, rather than the person who is at fault; also the user is virtually powerless to fix it.

• the correlation between spam and missing reverse DNS is no longer as strong as it used to be, as far as I can tell; spammers know they should pick a relay or proxy with a reverse DNS entry to get through filters, and as it becomes a requirement for relaying in general, more hosts have this anyway (regardless of exploitability or not).

Tags: address, dns, idea, mapping, reverse, server, smtp, spam, user, way

Aaron shares his rss-by-mail script. My reaction (cut from mail): “Together with my Mailman-archives-to-RSS script, and my blog (which is updated by mail), soon the semantic web will run entirely on SMTP…” (cackles evilly).

Well, maybe not yet — but it’s getting there. a bit.

Tags: bit, blog, mail, mailman-archives-to, reaction, rss, script, smtp, web