The Life of a SpamAssassin Rule

Spam: during a recent discussion on the SpamAssassin dev list, the question came up as to how long a rule could expect to maintain its effectiveness once it was public — the rule secrecy issue.

In order to make a point — that certain types of very successful rules can indeed last a long time — I picked out one rule, MIME_BOUND_DD_DIGITS. Here’s a smartened-up copy of what I found out.

This rule matches a certain format of MIME boundary, one observed in 17.4637% of our spam collection and with 0 nonspam hits. Since we have a massive collection of mails, received between Jan 2004 to May 2005, and a rule with a known history, we can then graph its effectiveness over time.

The rule’s history was:

  • bug 3396: the initial contribution from Bob Menschel, May 15 2004
  • r10692: arrived in SVN: May 16 2004
  • r20178: promoted to ‘MIME_BOUND_DD_DIGITS’: May 20 2004 (funnily enough, with a note speculating about its lifetime from felicity!)
  • released in the SpamAssassin 3.0.0 release: mid-Sep 2004

So, we would expect to see a drop in its effectiveness against spam in late May 2004 and onwards, if the spammers were reacting to SVN changes; or post September 2004, if they react to what’s released.

By graphing the number of hits on mails within each 2-hour window, we can get a good idea of its effectiveness over time:

The red bars are total spam mails in each time period; green bars, the number of spam mails that hit the rule in each period. May 15 2004 and Sep 20 2004 are marked; Jan 2004 is at the left, and May 2005 is at the right-most extreme of the graph. (There’s a massive spike in spam volume at the right — I think this is Sober.Q output, which disappears after a week or so.)

It appears that the rule remains about even in effectiveness in the 4 months it’s in SVN, but unreleased; it declines a little more after it makes it into a SpamAssassin release. However, it trails off very slowly — even in May 2005, it’s still hitting a good portion of spam.

Given this, I suspect that most spammers are not changing structural aspects of their spam in response to SpamAssassin with any particular alacrity, or at least are not capable of doing so.

To speculate on the latter, I think many spammers are using pirated copies of the spamware apps, so cannot get their hands on updated versions through ‘legitimate’ channels.

Speculating on the former — in my opinion there’s a very good chance that SpamAssassin just isn’t a particular big target for them to evade, compared to the juicy pool of gullible targets behind AOL’s filters, for example. ;)

Tags: , , , , , , , , ,

Comments (1)

Time Traveller Spammer caught

Wired: Turn Back the Spam of Time. An article about the time-travel spammer, now fingered as Robert ‘Robby’ Todino:

The anonymous e-mail offered $5,000 to any vendor capable of promptly delivering a collection of far-fetched gadgets for conducting time travel. Among the mysterious devices sought by the message’s author were an ‘Acme 5X24 series time transducing capacitor with built-in temporal displacement’ and an ‘AMD Dimensional Warp Generator module containing the GRC79 induction motor.’

He’s genuinely interested, it seems — but has a few psychological difficulties. (Thanks to Gary Stock for spotting it.)

Tags: , , , , , , , , ,

Comments

Audioscrobbler

Audioscrobbler is cool. Check it out — this is its log of my xmms listening habits, neatly cross-linked and referenced. (The cheesy ‘Liberty X’ listens were Catherine, I swear.)

Anyway, AS is a bit like Napster’s ‘explore other person’s music collection’ feature, which was cool for picking up recommendations — but this one is based on actual plays, and without the link to a service that the RIAA would want to see shut down ASAP. ;)

It can come out with some pretty bizarre results — for example, ‘people who listen to Thievery Corporation also listen to Radiohead’, according to this. Mind you, that’s probably correct…

Prediction: I’ll wind up being top of the list for listening to Acen’s tunes by the end of 2 weeks. That’s the plan at least ;)

Tags: , , , , , , , , ,

Comments

[IP] do read last Para. Time to correct the record re. the pillaged Museum in B (fwd)

Lost from the Baghdad museum: truth (Guardian). hmmm! It seems we’ve been had:

(In April, it was widely reported that) 100,000-plus priceless items were looted (from the Baghdad museum) either under the very noses of the Yanks, or by the Yanks themselves. And the only problem with it is that it’s nonsense. It isn’t true. It’s made up. It’s bollocks.

Incredible — it seems (a) the museum was looted — to a degree; the vast majority of ‘missing’ items had actually been moved into safe storage, and ‘most of the serious looting was an inside job’.

And (b) the academics and journalists who reported ‘170,000 items … stolen or destroyed’ had been led by the nose by Dr Donny George, the museum’s director of research. It just wasn’t true:

Over the past six weeks it has gradually become clear that most of the objects which had been on display in the museum galleries were removed before the war. Some of the most valuable went into bank vaults, where they were discovered last week. Eight thousand more have been found in 179 boxes hidden ‘in a secret vault’. And several of the larger and most remarked items seem to have been spirited away long before the Americans arrived in Baghdad.

George is now quoted as saying that that items lost could represent ‘a small percentage’ of the collection and blamed shoddy reporting for the exaggeration. ‘There was a mistake,’ he said. ‘Someone asked us what is the number of pieces in the whole collection. We said over 170,000, and they took that as the number lost. Reporters came in and saw empty shelves and reached the conclusion that all was gone. But before the war we evacuated all of the small pieces and emptied the showcases except for fragile or heavy material that was difficult to move.’

This indictment of world journalism has caused some surprise to those who listened to George and others speak at the British Museum meeting. One art historian, Dr Tom Flynn, now speaks of his ‘great bewilderment’. ‘Donny George himself had ample opportunity to clarify to the best of (his) knowledge the extent of the looting and the likely number of missing objects,’ says Flynn. ‘Is it not a little strange that quite so many journalists went away with the wrong impression, while Mr George made little or not attempt to clarify the context of the figure of 170,000 which he repeated with such regularity and gusto before, during, and after that meeting.’ To Flynn it is also odd that George didn’t seem to know that pieces had been taken into hiding or evacuated. ‘There is a queasy subtext here if you bother to seek it out,’ he suggests.

Tags: , , , , , , , , ,

Comments

(Untitled)

http://www.uncontrol.com/ — a flash applet which provides a good collection of nature-imitating mathematical eye candy. Number 16 is beautiful.

I used to write graphics demos on the C-64, which used a lot of this kind of stuff (although a hell of a lot simpler for obvious reasons). It occurs to me that Flash makes writing demos a lot easier; it provides a decent language (scripting as opposed to 6502 assembly), it gives you a good set of drawing tools (anti-aliasing, alpha blending, and 24-bit colour), the hardware no longer limits what you can do in 2-D graphics, and you can even buy software which takes care of the text effects like zooms, scrolling, bouncing etc. In other words, all the cool tricks are done for you ;)

I wonder what demo writers are doing nowadays, as a result? One side seems to be what these guys have done — actually go for really interesting, good-looking effects, rather than just the “how did they do that” factor. I would imagine the other side of the demo “bleeding edge” is doing a hell of a lot of 3-D stuff. (By hand. In assembler. ;)

Tags: , , , , , , , , ,

Comments

(Untitled)

One to buy; a collection of J.G. Ballard’s short stories. I’m a big Ballard fan, so I’ll be keeping an eye out. Great review too:

The drowned worlds, scorched cities and overgrown jungles of his early fiction; his concentration on the new media landscape of celebrity and stylised catastrophe; his exploration of the connections between sex, eroticism and death; his fetishism of motorways, cars, technology and high-rise buildings - Ballard wrote about the twentieth century in its own idiom, at a time when most other literary writers were no more than grappling with the same old tired clichés of the English class system.

Date: Sun, 04 Nov 2001 11:29:27 -0000
From: “Tim Chapman” (spam-protected)
To: forteana (spam-protected)
Subject: Complete Stories - Observer review

http://www.observer.co.uk/review/story/0,6903,587030,00.html

The Ballard of Shanghai jail

The poetry of disaster gleams among the anti-utopian’s collected short stories in JG Ballard’s The Complete Stories

Jason Cowley Sunday November 4, 2001 The Observer

The Complete Stories

JG Ballard

Flamingo £25, pp1,189

When I worked at the Times, a couple of years ago, a shout used to echo through the newsroom at moments of great national trauma, the death of Princess Diana, say, or a terrorist outrage - ‘Call JG Ballard’. Strangely, at such moments, JG Ballard seldom seemed to be at home or was, at least, sensibly not answering the phone.

Yet the news editor, for all his harassed panic, was right to think that Ballard might have something to contribute at a time of crisis, because no other contemporary British writer possesses his prescience and perspicacity, his instinct for catastrophe. No other writer foresaw, in quite the same way, how televised images of fame and death were to become all-powerful in our culture.

Reading this book of collected stories, spanning more than 1,000 pages and 40 years, is a peculiarly enriching experience. Every sentence Ballard writes is absolutely characteristic. Ever since he began publishing stories in the mid-1950s, in low-circulation science-fiction magazines such as New Worlds and Science Fantasy, he sought to find new ways of writing about our emerging consumer society, not as other sci-fi writers did through speculating about space travel or the far future, but through constructing his own cool, detached psychopathology of post-industrial society.

The drowned worlds, scorched cities and overgrown jungles of his early fiction; his concentration on the new media landscape of celebrity and stylised catastrophe; his exploration of the connections between sex, eroticism and death; his fetishism of motorways, cars, technology and high-rise buildings - Ballard wrote about the twentieth century in its own idiom, at a time when most other literary writers were no more than grappling with the same old tired clichés of the English class system.

Those who complain that he repeatedly writes the same book, that he cannot do character or convincingly animate women, misunderstand a writer who is less a formal storyteller than a prose surrealist. The motifs in his work are abandoned airfields, drained swimming pools, crashed cars, flooded lagoons, overlit motorways. His male heroes - doctors, pilots, architects, engineers - are emblematic last men, moving uneasily though flimsy, disintegrating worlds (in their impassive striving they recall the sad urban dreamscapes of Edward Hopper).

Through his interest in medicine, science and psychoanalysis, Ballard understands how powerfully we are driven by irrational and unconscious forces, that we are often no more than mysteries to ourselves.

In ‘Motel Architecture’ a man called Pangborn retreats from the world, spending his days alone in a solarium, amusing himself by endlessly replaying the shower sequence from Psycho on a bank of television screens (this story was written in 1978, before the age of video and digitised surveillance cameras). One day, he discovers there is an intruder in the solarium, eating his food and sharing his private space. Sometimes he catches glimpses of the intruder, his spectral presence and shifting shadows. Then a cleaning woman is found murdered in the solarium, lying in the ‘familiar postures he had analysed in a thousand blow-ups’. Pangborn is terrified until, in a moment of blazing self-revelation, he realises he has always been alone in the solarium, that he is his own intruder, a stranger to himself and perhaps now a murderer, too.

‘I’ve always thought that life was a kind of disaster area,’ says Ransome, the narrator of his third novel, The Drought. In Ballard’s fiction society is always close to or actually breaking down, and civility is threatened with extinction.

In many stories, he constructs closed, artificial communities - a tropical island paradise, an internment camp, a luxury high-rise apartment block, a hi-tech business park, a seaside leisure resort - then watches as they collapse under the strain of their own internal contradictions. ‘Is this the promised end?’ asks Kent in King Lear. ‘Or,’ replies Edgar, ‘image of that horror?’

Through reading Ballard, we have lived vicariously with a sense of an ending, simultaneously embracing what we most fear and perhaps most desire - the ruin of cities, the collapse of communities, the wilful embracing of deviance and obscenity.

Many of the stories here can be read as sketches for the later novels they became. ‘Dead Time’, in particular, is a template for Empire of the Sun (1984), the marvellous autobiographical novel which liberated Ballard from the cult of avant garde celebration and carried him to an international audience.

As a detainee, between the ages of 12 and 15, in the Lunghua prison camp in Shanghai, Ballard watched as Chinese soldiers were decapitated, as the streets of Shanghai were bombed by low-flying aircraft and as his fellow internees were harassed and brutalised. In Empire he writes of returning to the International Settlement where his parents lived in colonial seclusion to find the houses inexplicably deserted, and of watching the distant glow of the atom bomb explosion in Hiroshima, ‘that spectral mushroom cloud’.

In ‘Dead Time’, the young narrator, liberated from an internment camp, hides for hours under a pile of corpses to avoid detection from the Japanese, and later journeys across a ravaged landscape in search of his missing parents, a search that Ballard enacts again and again in his fiction, as if seeking to return to that Edenic first moment, the world of tranquillity that was destroyed the day the Japanese arrived in Shanghai and took him away from home.

If Ballard is an anti-utopian writer, a pessimist of human nature, it is because by the time he returned to England, as a young adult after the war, he had seen and experienced the worst of the world and of man’s potential for depravity. He was without hope or illusion, his imagination forever after to be shadowed by the ruined towns, abandoned aircraft, crashed cars and arbitrary disappearances and injustices of his childhood. And so, as the political philosopher John Gray has written, Ballard’s fictional achievement is to have communicated a vision of what fulfilment might mean in a time of nihilism. And who would argue that ours is not a time of nihilism and that Ballard is not the ideal chronicler of our disturbed modernity?

Tags: , , , , , , , , ,

Comments