Skip to content

Month: January 2007

Wikipedia and rel=”nofollow”

Published January 24, 2007

Apparently, Wikipedia has (possibly temporarily) decided to re-add the rel=”nofollow” attribute to outbound links from their encyclopedia pages.

There’s been a lot of heat and light generated about this, most missing one thing: there’s no reason why Google needs to pay attention.

Google, or any other search engine, can treat links in the Wikipedia pages any way they like — including ignoring ‘nofollow’, applying extra anti-spam heuristics of their own, or even trusting the links more highly.

‘Nofollow’ has had pretty much no effect on web-spam, and now is generally festooned all over weblog posts across the internet, both spammed and non-spammed posts, at that. It’d be interesting to see if it’s yet flipped to mean a higher correlation with nonspam than spam content…

Update: It appears Wikipedia used ‘nofollow’ before, so this is not exactly new, either.

more on social whitelisting with OpenID

Published January 23, 2007

An interesting post from Simon Willison, noting that he is now publishing a list of “non-spammy” OpenID identities (namely people who posted one or more non-spammy comments to his blog).

I attempted to comment, but my comments haven’t appeared — either they got moderated as irrelevant (I hope not!) or his new anti-comment-spam heuristics are wonky ;) Anyway, I’ll publish here instead.

It’s possible to publish a whitelist in a “secure” fashion — allowing third parties to verify against it, without explicitly listing the identities contained. One way is using Google’s enchash format. Another is using something like the algorithm in LOAF.

Also, a small group of people (myself included) tried social-network-driven whitelisting a few years back, with IP addresses and email, as the Web-o-Trust.

Social-network-driven whitelisting is not as simple as it first appears. Once someone in the web — a friend of a friend — trusts a marginally-spammy identity, and a spam is relayed via that identity, everyone will get the spam, and tracking down the culprit can be hard unless you’ve designed for that in the first place (this happened in our case, and pretty much killed the experiment). I think you need to use a more complex Advogato-style trust algorithm, and multiple “levels” of outbound trust, instead of the simplistic Web-o-Trust model, to avoid this danger.

Basically, my gut feeling is that a web of trust for anti-spam is an attractive concept, possible, but a lot harder than it looks. It’s been suggested repeatedly ever since I started writing SpamAssassin, but nobody’s yet come up with a working one… that’s got to indicate something ;) (Mind you, the main barrier has probably been waiting for workable authentication, which is now in place with DK/SPF/DKIM.)

In the meantime, the concept of a trusted third party who publishes their concept of an identity’s reputation — like Dun and Bradstreet, or Spamhaus — works very nicely indeed, and is pretty simple and easy to implement.

SpamArchive.org no more

Published January 22, 2007

Remember SpamArchive.org, the site that allowed random Internet users to upload their spam? It was set up back in 2002 by CipherTrust, one of the commercial anti-spam vendors, to offer a large, ‘standard’ database of known spam to be used for testing, developing, and benchmarking anti-spam tools, and for anti-spam researchers. It got a bit of coverage at Slashdot and Wired News at the time.

It never really was too useful for its supposed purposes, though, at least for us in SpamAssassin, since:

  1. it collected submissions from random internet users, without vetting, and therefore couldn’t be guaranteed to be 100% valid;

  2. it ‘anonymized’ the headers too much for the spam to be useful in testing a filter like SpamAssassin, which requires correct header data for valid results;

  3. collecting spam has never been a problem; avoiding it is ;)

Anyway, looks like Ciphertrust/Secure Computing have since lost interest, since they’ve allowed the domain to lapse. It has instead been picked up by a domain speculator:

Domain ID:D134033677-LROR
Domain Name:SPAMARCHIVE.ORG
Created On:30-Nov-2006 18:52:13 UTC
Last Updated On:01-Dec-2006 12:42:26 UTC
Expiration Date:30-Nov-2007 18:52:13 UTC
Sponsoring Registrar:PSI-USA, Inc. dba Domain Robot (R68-LROR)
Status:TRANSFER PROHIBITED
Registrant ID:ABM-9376887
Registrant Name:Robert Farris
Registrant Organization:Virtual Clicks
Registrant Street1:P.O. Box 232471
Registrant Street2:
Registrant Street3:
Registrant City:San Diego
Registrant State/Province:US
Registrant Postal Code:92023
Registrant Country:US
Registrant Phone:+1.7205968887
Registrant Phone Ext.:
Registrant FAX:
Registrant FAX Ext.:
Registrant Email:[email protected]
Name Server:NS1.DIGITAL-DNS-SERVER.COM
Name Server:NS2.DIGITAL-DNS-SERVER.COM

A visit to http://www.spamarchive.org/ now reveals a parking page, which grabs the browser window, forces it to front, maximises it, attempts to bookmark it, add it to the Firefox sidebar — and who knows what else ;)

apres-Barcamp!

Published January 21, 2007

Well, that was great fun — well worth the trip down. Got to put a load of faces to names, meeting up with a fair few people I’ve been conversing with online — and a few I hadn’t met before, online or off. Plenty of thought-provoking and interesting chats, too!

My talk went down well, I think. Unfortunately, we didn’t quite know how to operate the projector, so the attendees, while they got to hear me talk, didn’t get to read the leftmost quarter or so of each slide ;)

To make up for it, here they are:

OpenOffice 2 source (234k), PDF (320k), HTML

(PS: Regarding GUI interfaces to managing EC2 — a question that came up in the Q&A — here’s one that looks pretty interesting…)

Barcamp!

Published January 17, 2007

I was wavering for a minute there, but I’ve decided to head down to Waterford for Barcamp Ireland – SouthEast — a bit last-minute, but there you go! Tickets and hotel booked.

I’m hoping to give a quick, 20-minute intro to Amazon’s EC2 and S3 web services — what they are, how they’re used, some interesting features and a few gotchas to watch out for.

Also, I’m up for dinner on the Saturday night, given there’s a promise of free booze ;)

Any taint.org readers heading down?

Debunking the “cocaine on 100% of Irish banknotes” story

Published January 11, 2007

BBC: Cocaine on ‘100% of Irish euros’:

One hundred percent of banknotes in the Republic of Ireland carry traces of cocaine, a new study has found.

Researchers used the latest forensic techniques that would detect even the tiniest fragments to study a batch of 45 used banknotes.

The scientists at Dublin’s City University said they were “surprised by their findings”.

Also at RTE, Irish Examiner, PhysOrg.com, Bloomberg.com, even at Kazakhstan’s KazInform.

This story is (of course) being played widely in the media as “OMG Ireland must use more coke than anywhere else” — in particular, in comparison with a previous study in the US:

The most recent survey carried out in the US showed 65% of dollar notes were contaminated with cocaine.

The DCU press-release has a few more details:

Using a technique involving chromatography/mass spectrometry, a sample of 45 bank notes were analysed to show the level of contamination by cocaine. …

62% of notes were contaminated with levels of cocaine at concentrations greater than 2 nanograms/note, with 5% of the notes showing levels greater than 100 times higher, indicating suspected direct use of the note in either drug dealing or drug inhalation. … The remainder of the notes which showed only ultra-trace quantities of cocaine was most probably the result of contact with other contaminated notes, which could have occurred within bank counting machines or from other contaminated surfaces.

However, looking at an abstract of what I think is the paper in question, Evaluation of monolithic and sub 2 µm particle packed columns for the rapid screening for illicit drugs — application to the determination of drug contamination on Irish euro banknotes, Jonathan Bones, Mirek Macka and Brett Paull, Analyst, 2007, DOI: 10.1039/b615669j, that says:

A study comparing recently available 100 × 3 mm id, 200 × 3 mm id monolithic reversed-phase columns with a 50 × 2.1 mm id, 1.8 µm particle packed reversed-phase columns was carried out to determine the most efficient approach … for the rapid screening of samples for 16 illicit drugs and associated metabolites. … Method performance data showed that the new LC-MS/MS method was significantly more sensitive than previous GC-MS/MS based methods for this application.

My emphasis. I’d guess that that means that comparing this result to banknote-analysis experiments carried out elsewhere using different methods is probably invalid — perhaps this method is more efficient at picking up ‘contact with other contaminated notes, which could have occurred within bank counting machines or from other contaminated surfaces’, as noted in the DCU release?

Email authentication is not anti-spam

Published January 10, 2007

There’s a common misconception about spam, email, and email authentication; Matt Cutts has been the most recent promulgator, asking ‘Where’s my authenticated email?’, in which various members of the comment thread consider this as an anti-spam question.

Here’s the thing — email these days is authenticated. If you send a mail from GMail, it’ll be authenticated using both SPF and DomainKeys. However, this alone will not help in the fight against spam.

Put simply — knowing that a mail was sent by ‘jm3485 at massiveisp.net’, is not much better than knowing that it was sent by IP address 192.122.3.45, unless you know that you can trust ‘jm3485 at massiveisp.net’, too. Spammers can (and do) authenticate themselves.

Authentication is just a step along the road to reputation and accreditation, as Eric Allman notes:

Reputation is a critical part of an overall anti-spam, anti-phishing system but is intentionally outside the purview of the DKIM base specification because how you do reputation is fundamentally orthogonal to how you do authentication.

Conceptually, once you have established an identity of an accountable entity associated with a message you can start to apply a new class of identity-based algorithms, notably reputation. … In the longer term reputation is likely to be based on community collaboration or third party accreditation.

As he says, in the long term, several vendors (such as Return Path and Habeas) are planning to act as accreditation bureaus and reputation databases, undoubtedly using these standards as a basis. Doubtless Spamhaus have similar plans, although they’ve not mentioned it.

But there’s no need to wait — in the short term, users of SpamAssassin and similar anti-spam systems can run their own personal accreditation list, by whitelisting frequent correspondents based on their DomainKeys/DKIM/SPF records, using whitelist_from_spf, whitelist_from_dkim, and whitelist_from_dk.

Hopefully more ISPs and companies will deploy outbound SPF, DK and DKIM as time goes on, making this easier. All three technologies are useful for this purpose (although I prefer DKIM, if pushed to it ;).

It’s worth noting that the upcoming SpamAssassin 3.2.0 can be set up to run these checks upfront, “short-circuiting” mail from known-good sources with valid SPF/DK/DKIM records, so that it isn’t put through the lengthy scanning process.

That’s not to say Matt doesn’t have a point, though. There are questions about deployment — why can’t I already run “apt-get install postfix-dkim-outbound-signer” to get all my outbound mail transparently signed using DKIM signatures? Why isn’t DKIM signing commonplace by now?

How to deal with joe-jobs and massive bounce storms

Published January 10, 2007

As I’ve noted before, we still have a major problem with sites generating bounce/backscatter storms in response to forged mail — whether deliberately targeted, as a “Joe-Job”, or as a side-effect of attempts to evade over-simplistic sender address verification as seen in spam, viruses, and so on.

Sites sending these bounces have a broken mail configuration, but there are thousands remaining out there — it’s very hard to fix an old mail setup to avoid this issue. As a result, even if your mail server is set up correctly and can handle the incoming spam load just fine, a single spam run sent to other people can amplify the volume of response bounces in a Smurf-attack-style volume multiplication, acting as a denial of service. I’ve regularly had serious load problems and backlogs on my MX, due solely to these bounces.

However, I think I’ve now solved it, with only a little loss of functionality. Here’s how I did it, using Postfix and SpamAssassin.

(UPDATE: if you use the algorithm described below, you’ll block mail from people using Sender Address Verification! Use this updated version instead.)

Firstly, note that if you adopt this, you will lose functionality. Third party sites will not be able to generate bounces which are sent back to senders via your MX — except during the SMTP transaction.

However, if a message delivery attempt is run from your MX, and it is bounced by the host during that SMTP transaction, this bounce message will still be preserved. This is good, since this is basically the only bounce scenario that can be recommended, or expected to work, in modern SMTP.

Also, a small subset of third-party bounce messages will still get past, and be delivered — the ones that are not in the RFC-3464 bounce format generated by modern MTAs, but that include your outbound relays in the quoted header. The idea here is that “good bounces”, such as messages from mailing lists warning that your mails were moderated, will still be safe.

OK, the details:

In Postfix

Ideally, we could do this entirely outside Postfix — but in my experience, the volume (amplified by the Smurf attack effects) is such that these need to be rejected as soon as possible, during the SMTP transaction.

Update: I’ve now changed this technique: see this blog post for the current details, and skip this section entirely!

(If you’re curious, though, here’s what I used to recommend:)

In my Postfix configuration, on the machine that acts as MX for my domains — edit ‘/etc/postfix/header_checks’, and add these lines: 
/^Return-Path: <>/                              REJECT no third-party DSNs
/^From:.*MAILER-DAEMON/                         REJECT no third-party DSNs
Edit ‘/etc/postfix/null_sender’, and add: 
<>              550 no third-party DSNs
Edit ‘/etc/postfix/main.cf’, and ensure it contains these lines: 
header_checks = regexp:/etc/postfix/header_checks
smtpd_sender_restrictions = check_sender_access hash:/etc/postfix/null_sender
(If you already have an ‘smtpd_sender_restrictions’ line, just add ‘check_sender_access hash:/etc/postfix/null_sender’ to the end.) Finally, run: 
sudo postmap /etc/postfix/null_sender
sudo /etc/init.d/postfix restart
This catches most of the bounces — RFC-3464-format Delivery-Status-Notification messages from other mail servers.

In SpamAssassin

Install the Virus-bounce ruleset. This will catch challenge-response mails, “out of office” noise, “virus scanner detected blah” crap, and bounce mails generated by really broken groupware MTAs — the stuff that gets past the Postfix front-line.

Once you’ve done these two things, that deals with almost all the forged-bounce load, at what I think is a reasonable cost. Comments welcome…

Kernighan and Pike on debugging

Published January 8, 2007

While reading the log4j manual, I came across this excellent quote from Brian W. Kernighan and Rob Pike’s “The Practice of Programming”:

As personal choice, we tend not to use debuggers beyond getting a stack trace or the value of a variable or two. One reason is that it is easy to get lost in details of complicated data structures and control flow; we find stepping through a program less productive than thinking harder and adding output statements and self-checking code at critical places. Clicking over statements takes longer than scanning the output of judiciously-placed displays. It takes less time to decide where to put print statements than to single-step to the critical section of code, even assuming we know where that is. More important, debugging statements stay with the program; debugging sessions are transient.

+1 to that.

5 things revisited

Published January 5, 2007

Hey Danny! I’ve already filled out my “5 Things” list. Surprisingly (or thankfully) nobody has commented on #5 ;)

Great Things, btw. I might adopt #4, and see if it works.

It’s great fun following the web of “5 Things” links as they percolate through the interwebs. now if only the people I nominated would get on with their lists…

Script: knewtab

Published January 5, 2007

Here’s a handy script for konsole users like myself:

knewtab — create a new tab in a konsole window, from the commandline

usage: knewtab {tabname} {command line …}

Creates a new tab in a “konsole” window (the current window, or a new one if the command is not run from a konsole).

Requires that the konsole app be run with the “–script” switch.

Download ‘knewtab.txt’