taint.org: Justin Mason’s Weblog

Web: Almost every project and organisation has, at some stage, bemoaned having stale data on their website, and wished there was a better way to keep it up to date; or wished their FAQ was more complete; or wished they had the time to HTML-ize all their know-how and get it up there.

Well, here’s what we did in SpamAssassin to deal with this problem. (Seeing as I’ve talked about this three times in the past month, I’ll write it up here so I can just point at the URL next time!)

First off, we experimented with having the site checked into CVS, FAQ-o-matic, and the Python FAQ software (which was pretty good). All were OK, but very specific in format, using the traditional question-answer FAQ layout — that’s good for FAQs, but not so good for a lot of other stuff — and keeping it updated was still limited to a small group, therefore the info got stale again.

So we moved to a Wiki. Here’s my tips for Wiki-izing your website so that the end results are better than what went in.

Use good wiki software: unusable software will be a pain to use, and the info will still go stale. We used Moin Moin - http://moin.sourceforge.net/ - partly because I like Python (it’s nearly perl! ;), it can produce RSS, and it was pretty easy to install.

Don’t worry: people won’t vandalise it (much). It turns out that vandalism and people throwing up crappy info isn’t a serious problem at all. You should increase the barrier, in the following ways:

Require user accounts: set the security policy so that a user account must be set up before editing is possible. This means you won’t get wiki-spammed, and also has the side effect of imposing a pretty big barrier to casual vandals.

Send changes to a list: set all changes to be mailed to a mailing list as diffs. This is the most important tip. If you already have a mailing list with the knowledgeable part of the community on it, use that list — because they’re the ones who’ll be able to recognise if erroneous info is put up, and will be annoyed about this enough to bother fixing it. There’s a bonus side-effect of this; even if some people didn’t like the wiki to start with, they’ll eventually be needled into using it by wanting to fix stuff they perceive as wrong. And then they get sucked in ;)

Use diff for the mailed changes: Moin by default will only send out change messages saying ’something changed on this page!’. That’s not good enough, unfortunately — you want to mail out what the new text looks like, and highlight exactly where the change happened. Moin can do this nicely, with this patch, which adds a mail_commits_address, where all diffs on every page are sent, using the normal diff mechanism.

Ensure the wiki software can revert quickly: If someone does make a bad change, Moin supports one-click reversion of the page to what it was beforehand. That’s great for dealing with spam, or clueless vandalism.

Keep one or two static pages: If you’re worried about some script kiddie thinking that defacing a wiki makes them look cool, then keep one or two of the primary user-facing pages as static data. For example, take a look at the link-bar at the top of http://spamassassin.apache.org/ ; five of the ten links are to static pages, the other five are now wiki-ized. In particular, our front page and our downloads page are both static, but our docs are predominantly Wiki’d.

Publicize Mozex: most techie groups will have techie users, and we hate using browser text-boxes to edit text. Mozex — http://mozex.mozdev.org/ — saves the day here — it’s a godsend.

Shepherd new changes: in the early stages, you want one or two people who tidy up changes from Wiki newbies, as they go in. They need to keep it looking pretty, and perform Refactoring of stuff that could be laid out better or should become multiple pages. Eventually, others will get the hang of that (and do a much better job than you do ;).

That’s the lot. Most of these are to, essentially, migrate aspects of your already-existing and already-working community into this new outlet. In our experience, it’s worked really well — our Wiki is now the most reliable source of info about SpamAssassin, and is extensive and up-to-date.

Tags: faq, info, list, problem, python, software, time, use, web, website, wiki

Code: Berkeley DB, the de-facto std for open-source high-performance database files on UNIX, is displaying some odd behaviour — it appears to be sleeping for 1 second inside the database library code, under load, for some versions of libdb. If you’re curious, there’s More info here.

Tags: behaviour, berkeley, code, database, info, libdb, library, load, std, unix

Spam: Let me take this moment to welcome our UK friends to the ’spam now illegal’ club; unlike the US, the European and Australian anti-spam laws seem to be shaping up nicely, requiring opt-in before ‘email marketing’ can be sent.

This actually happened in Ireland a couple of weeks ago, but I think I forgot to mention it here, so here’s the details:

Announcement, full text, full text as HTML. (It’s section 13 you want to read. Note that OpenOffice seems to have miscounted the bullet points in the HTML version ;)

The good stuff:

• it’s opt-in, not opt-out like the cruddy CAN-SPAM act in the US. so that’s a fundamentally anti-spam position. Thanks EU!
• each spam counts as a separate offence = lots of damages, I’d guess.

• forging/disguising of originating header info is prohibited.

The bad:

• if you run a mailing list, and you’re not sure that you got everyone’s permission to receive your mails (and if not, why not?), you’d better do a reconfirmation run quick ;)
• no private right of action; but that’s pretty much std for Europe. we’re reliant on the Regulator to take action against spammers.
• spamming to mailing lists is not prohibited — but then, I haven’t seen that blocked by any other law.
• it’s unclear if spamming to role addresses (e.g. ‘foo-admin’, ‘info@company‘, etc.) is prohibited. I would guess that if they wind up in the mailbox of a ‘natural person’ it would be. But this may have to be worked out in court.
• talks about ‘direct marketing’. Does this mean that faked-up ‘newsletters’ will be a loophole? Also, means that religious and political spam is permitted. But I haven’t seen much of that in Ireland… yet…

• won’t be any good against US-based spammers. No surprise there. HOWEVER it may be useful against large multinational companies taking advantage of CAN-SPAM’s relaxed regime to indulge in a little spamming, if they have an Irish office.

  And, of course, it’ll mean that Ireland won’t develop Florida’s reputation any time soon, which is a good thing.

• Will it be useful against spammers in other parts of the EU? That’s another question. Anyone know? I know of a bunch in France I’d really like to deal with.

  Brian Nisbet reckons it may.

I was reminded by this letter from the Department of Communications received by UCC , which notes:

But the Minister has announced that he intends using Ireland’s Presidency of the EU to initiate global partnership in clamping down on ’spam.’

Global? Just don’t ask for any help from the Florida state government. ;)

Spam: Other (big) spam news: ‘Gaven Stubberfield’ arrested for ‘falsifying his identity so that his e-mails could not be traced’. SBL say that Jeremy Jaynes, aka ‘Gaven Stubberfield’ is the eight-most prolific spammer in the world, and is ‘notorious for ‘horsey porn’ spam’.

Irish: Irish WWW pioneer Peter Flynn now has a weblog, it seems. As far as I can tell, Peter was responsible for much of the good stuff at celt.ucc.ie, which reminds me to post this link to Pangur Bán I’ve been meaning to post.

Messe ocus Pangur Bán,
cechtar nathar fria saindan:
bíth a menmasam fri seilgg,
mu memna céin im saincheirdd.

In my case, it’s mise agus Bubba Liath, otherwise pretty close despite the intervening 11 centuries…

Tags: action, can-spam, club, html, info, mailing, marketing, moment, spam, text

So a little more investigation shows that the massive numbers of IPs spamming my referrer logs (like 1000 different IPs every day), are not open proxies as I at first thought; I tested 130, and none had any of the well-known proxy ports open.

My current guess is that they’re malware, such as those ‘ad banner spyware’ programs, and the makers of that software must be doing deals with spam companies to set up the spyware to periodically load URLs in order to referrer-spam for the spam bureau’s customers.

In this case, all the spammed URLs are owned and registered by one porn operation, which is either operating from Switzerland (according to the tech contact info) or Los Angeles (according to the DNS info in whois). (More likely the latter.)

All the IPs doing the spam page loads, are running on Windows XP and Windows 2000 systems as far as I can see, with ports 1025 and 5000 open, so alternatively, maybe they’re trojaned… but there doesn’t seem to be any good evidence indicating that. (those ports are reasonably innocuous.)

Anyone got any ideas? Here’s some sample access_log lines for 100 IPs, gzipped, if anyone wants to check them out.

Tags: anyone, day, info, investigation, ips, referrer, spam, spyware, urls, windows

While trekking in Nepal, I had a copy of the Lonely Planet Guide to Trekking in the Himalayas, borrowed from our mates Caolan and Barbara. It was especially notable for its incredible medical section, which contained lots of info on what drugs to use to treat various diseases, described symptomatically (of course, in most of the world, most of the common illnesses boast symptoms similar to “I have greenish foul-smelling gravy squirting from both ends of my body”. But it’s good to be able to tell them apart).

It was also notable, because anyone who had a copy knew all about altitude sickness, and were indescribably paranoid. The ones who were charging up the trails as fast as they could generally did not have a copy, and no doubt half of them came back down again in slightly nasty circumstances.

Anyway, it was the best medical info I’ve ever read. Reading the paper today, I came across a reference to e-med.co.uk, which claims to be medical info, including treatment details, for people who might be far away from a doctor. The perfect resource for a know-it-all who doesn’t want to spend money and time on a doctor, just to be told to go home and take an aspirin! Unfortunately it seems to be a “consultation by email” service, rather than “look it all up” one. Ah well.

Caolan and Barbara should be somewhere around Oz by now. I must see if I can dig up the URL of their travelogue site, it’s great fun.

Tags: caolan, copy, course, doctor, guide, info, lonely, planet, section, trekking