Spam volumes at accidental-DoS levels

Both Jeremy Zawodny and Dale Dougherty at O’Reilly Radar are expressing some pretty serious frustration with the current state of SMTP. I have to say, I’ve been feeling it too.

A couple of months back, our little server came under massive load; this had happened before, and normally in those situations it was a joe-job attack. Switching off all filtering and just collecting the targeted domain’s mail in a buffer for later processing would work to ameliorate the problem, by allowing the load to “drain”. Not this time, though.

Instead, when I turned off the filtering, the load was still too high — the massive volume of spam (and spam blowback / backscatter) was simply too much for the Postfix MTA. The MTA could not handle all the connections and SMTP traffic in time to simply collect all the data and store it in a file!

Looking into the “attack” afterwards, once the load was back under control, it looked likely that it wasn’t really an attack — it was just a volume spike. Massive SMTP load, caused by spammers increasing the volume of their output for no apparent reason. (Since then, spam volumes have been increasing still further on a nearly weekly basis.)

This is the effect of botnets — the amount of compromised hosts is now big enough to amplify spam attacks to server-swamping levels. Our server is not a big one, but it serves less than 50 users’ email I’d say; the user-to-CPU-power ratio is pretty good compared to most ISPs’ servers.

So here’s the thing. New SMTP-based methods of delivering nonspam email — whether based on DKIM, SPF, webs of trusted servers, or whatever — will not be able to operate if they have to compete for TCP connection slots with spammers, since spammers can now swamp the SMTP listener for port 25 with connections. In effect, spam will DDoS legitimate email, no matter what authentication system that legit mail uses to authenticate itself.

This, in my opinion, is a big problem.

What’s the fix? A “new SMTP” on a whole different port, where only authed email is permitted? How do you make that DoS-resistant? Ideas?

(Obviously, counting on spammers to notice or care is not a good approach.)

This entry was posted in Uncategorized and tagged , , , , , , , . Bookmark the permalink. Both comments and trackbacks are currently closed.


  1. Posted March 6, 2007 at 15:24 | Permalink

    Make this new authed-email protocol transfer, say, 5 cents from the sender to the recipient with each email.

    There are some fun details to work out. (How’s the payment handled? Between the parties’ ISPs, who have registered their financial information. How’s it funded? ISPs take a cut of each nickel. Is a nickel too much/too little? Recipients should be able to name their own price, but my guess is a nickel will be about what most people decide on. But won’t there still be spam? Yes, but it will be roughly 5 orders of magnitude more expensive and at least that much rarer, and everyone can set their recipient price to get as much or as little spam as they want. Aren’t micropayments a proven failure? ISPs handle transactions amongst themselves, will bill users on a monthly basis, and will presumably shuffle around hundreds of dollars a month between themselves, not pennies an hour. What about mailing lists? Recipients must be able to waive payment from a particular registered sender, and senders must be able to abort sending if payment is demanded. What about a user’s machine being taken over by a virus whose payload is mailing the author empty cash? That’s a feature, not a bug, and that possibility would, I hope, inspire everyone to finally, finally, finally take security seriously.)

    But the details are not hard. The key idea is simply this. The root of all SMTP’s problems is that it pretends mail is free, both sending and receiving. That’s simply not true. Email is not “too cheap to meter.” SMTP’s founded on a lie.

    I don’t know if pay-to-send (“$MTP”?), exactly as I’ve described it, is the best solution. But I do know that any actual solution — as opposed to some of the excellent, brilliant, thus-far-highly-successful band-aids currently in use — must assign costs to the perpetrators, not victims. This proposal seems, to me, the most logical and workable in this class of solutions.

  2. Posted March 6, 2007 at 17:18 | Permalink

    It isn’t just an SMTP problem – blog comment spammers also DDOS machines. The problem of unwanted traffic is pervasive in the Internet. It isn’t just at the TCP/IP layer (like SMTP and HTTP spam) either – spam on Usenet travels over the NNTP overlay network, and BGP has problems with bogus announcements. If you are going to succeed then you need something much more comprehensive than just fiddling with email abuse.

  3. Posted March 6, 2007 at 17:40 | Permalink

    Jamie: how do you solve the issue that a micropayment infrastructure to support pay-to-send email would dwarf the credit card system, the biggest payment infrastructure existing in the world today? This post covers the reasons why it’d be very hard to get working, IMO. Given the problem we’re talking about here, where the current volume of spam swamps a single local CPU’s network stack, I’d have a hard time seeing how extending that to perform a micropayment debit, over the net, would be less DoS’able…

    (having said that, hashcash is good for this, since verifying a HC stamp is trivial in CPU expense and requires no network access. hmm.)

    Tony: very good point. In fact, this blog has been knocked off the air several times by botnet-wielding blog comment spammers, too, I should have remembered that.

  4. Posted March 6, 2007 at 17:52 | Permalink

    Have you really had enough spam to swamp the network stack, or has it just swamped the MTA? Were you accepting the spam (and incurring disk load) or blocking it (which should be cheap using DNSBLs). Sadly current MTAs are not good at high concurrency since they require a process per connection.

  5. Posted March 6, 2007 at 18:13 | Permalink

    Actually, not really getting swamped at the TCP stack level, true. It was accepting the spam, and getting backlogged at the MTA. (That still means that network lookups cause problems, since they increase concurrency requirements a lot….)

  6. David Malone
    Posted March 6, 2007 at 21:11 | Permalink

    I think some people reckon that this is a problem with the internet in general – people don’t have to ask before they send traffic to you. This is an attempt to deal with this at the IP layer – I’m not sure what the answer at the SMTP layer would be – I guess it might look something like OpenBSD’s spamd?

  7. Posted March 6, 2007 at 21:37 | Permalink

    Pay per play ain’t the answer, no matter how it’s implemented. SMTP-AUTH would be a good sticking plaster, if only for a while. IMHO ISPs just need to be convinced that the botted machines are more trouble that they’re worth, and they’re going to have to start cutting them off. Should we be punishing the ISPs at a higher level than RBLs? Can we sue them, or legislate against them? Suing the spammers certainly ain’t working!

  8. Posted March 6, 2007 at 21:56 | Permalink

    The problem of spam overwhelming systems (not withstanding existing filtering mechanisms) is a very real one, and one of the reasons why I presented “Capacity Planning: A Core Anti-Spam Competency” at the last meeting of the Messaging Anti-Abuse Working Group (MAAWG), see (or .pdf).

    Believe it or not, in my opinion, so far we’ve been VERY fortunate when it comes to observed spam volumes — bad as they’ve been, there’s still a lot more spammer horsepower left under the hood, and I’m increasingly concerned that at some point that unutilized capacity is going to be unleashed on us all.

    Joe St Sauver

  9. Posted March 7, 2007 at 11:06 | Permalink

    The paper that David Malone cites says that TCP flows are too symmetric for their technique to apply, because of the acking. So you would have to apply the idea at the level of the application protocol, e.g. looking at the error rate. This is exactly what Richard Clayton’s “extrusion detection” does.

  10. Posted March 7, 2007 at 15:09 | Permalink

    Justin, my idea is different: the sender pays the recipient, not some central authority.

    Paying a central authority to send email is just dumb, there’s no need for that, it’s a 18th-century idea. There may be need for a single authority (though I’m not convinced of it) to handle some registration bureaucracy, like making sure that each ISP is traceable to a human being and provides a valid credit card or line of credit, i.e., so spammers can’t register a hundred fly-by-night pseudo-ISPs. But that can be paid for with a tiny fraction of the actual stamp fees.

    (IMHO the bar should be pretty damn high for proving you’re a valid ISP. This is why I don’t think there needs to be a single authority. There should be multiple registries, and ISPs should be free to pick and choose which ones they trust. As this system gets off the ground, having a good reputation will make or break a registry. Let the free market decide what constitutes “proving you’re not a spammer.”)

    So the sender pays the recipient. What actually happens of course is that the sender’s ISP trades some bits with the recipient’s ISP, and at the end of the month or whatever, if one ISP owes the other some significant amount, a bill is sent.

    Likely, for a typical pair of ISPs X and Y, X will owe Y $15,123 and Y will owe X $15,345, so relatively little money will actually trade hands. ISPs that send much more unwanted email than they receive will owe significant amounts of money, say, $50 to each of a hundred other ISPs — obviously, this is by design.

    So the scale of this system is nothing like the credit-card infrastructure, which involves tens of millions of merchants and billions of customers. The long tails of ISPs will number of the tens of thousands, but measured by the number of transactions, 99% of the traffic will be between a small number of key players. Each month, AOL and MSN will check in with each other: “you owe me $1,920,000, and I owe you $1,930,000, so the $10,000 check is in the mail.” “Thanks!”

    And each ISP’s customer will see his or her bill adjusted. They’ll pay more each month if they sent more unwanted mail than they received, and less if they got more than they sent. Again, this is by design. (A corollary is that people who want to receive spam can put themselves on spam-me lists, set their receive-fee to a penny or a dime or whatever they feel appropriate, and maybe at the end of the month they will get paid by their ISP to have an email address. Note that I cannot possibly conceive that sending such people email would be a profitable business model, so I doubt this will really happen, which is, of course, a very good thing.)

    Imagine how cheap it would be to run a credit-card company if you had no customers to deal with, only merchants — if you only had 10,000 merchants instead of 10,000,000 — and they all were trading money amongst themselves!

    The bits that are traded are the equivalent of “IOU 5 cents for message ID xyz,” encrypted with the sender’s ISP’s key. The recipient hits the network to look up the sender’s ISP’s public key and then decrypts the message; if it says the above, the recipient logs the encrypted text and plaintext and delivers the mail normally. At the end of the month, the logs provide an impossible-to-forge accounting of who owes whom how much.

    The recipient can cache public keys, and I’m just guessing at what access patterns would be but it’s my guess that 99.9% of public key lookups will be cache hits, so the main overhead is the CPU to encrypt/decrypt a small message.

    Yes, the current volume of email can swamp a machine, but that’s because the email is 90-99% spam! When you cut the volume of email by 1-2 orders of magnitude, little things like crypto become trivial.

  11. Posted March 7, 2007 at 15:25 | Permalink

    I think that the future will be closed email transfer networks either restricting by access lists or IPsec tunnels or similar. Once you make a resource reachable to the general internet it can be DOS’d.

    Maybe we have something to learn from the Telcos in terms of establishing bilateral peerings between trusted parties.


  12. Posted March 7, 2007 at 15:26 | Permalink

    Here are the other objections in the “Sender Pays” article you linked to:

    • “What do you do when a ‘bad guy’ steals the e-postage stamps off Aunt Millie’s hard disk, without her knowledge?”

    With the system I propose, Millie doesn’t have any stamps on her hard disk. At the customer level, the only way to steal is to send mail from the customer’s IP using (one of) the customer’s From address(es). If Aunt Millie is running an open wireless network, and her nephew gets mad at her, parks in her driveway, and runs a program to spam 10,000 people “from” her address, then it’s between her and her ISP to handle that. I imagine most ISPs would let users protect themselves by automatically cutting them off after $x until they call in to customer service and straighten out whether fraud is going on or not. Her nephew has always been able to annoy her by calling Australia and leaving the phone line open for hours; this is just something service oriented companies have to deal with.

    The point is there’s nothing for Millie to lose in a hard drive crash, worry about backing up, accidentally delete, etc.

    At the ISP level, each ISP needs to keep its private key under strict control, as it does with any other valuable company secrets. Registries already need to provide a way to invalidate a public key, of course, so it’s not asking much to let ISPs report those keys stolen. Again, this is something those kinds of companies already deal with.

    • “Users hate micropayments.” Not if their email clients have been written to incorporate them automatically.

    My mail client would be configured to automatically send any mail where the recipient demands up to $0.10. Maybe yours would be set to send any mail up to a cost of $0.05. The defaults would work themselves out quickly I’m sure.

    And when I receive mail, my mail client would be configured to automatically refund the sending cost for anyone in my address book. Yours might not; up to you. If the cost weren’t refunded automatically, a good mail client would provide a “Refund” button, right next to “Reply,” that closes the window and refunds the sender’s nickel. It would be good netiquette to refund legitimate mail (but I imagine friends wouldn’t get too worked-up over a forgotten click here and there). Of course if I Reply to your mail I’m refunding your money anyway by sending it back to you!

  13. Stuart
    Posted March 7, 2007 at 21:00 | Permalink

    Most people already pay to send email. They pay for the servers and the bandwidth. The problem is that spammers typically steal the resources they use to send spam.

    Stop the botnets – Stop the spam.

  14. Devdas Bhagat
    Posted March 8, 2007 at 00:46 | Permalink

    Hmmmm, I suggest going back to the original RBL. Take out the ISPs hosting the zombies right at the network.

    What use are a million zombies, if they cannot communicate?

    Rejecting mail at the edge, before DATA works much better than trying to accept and filter it based on content. Blocking sending client on dynamic IPs works well.

    Instead of trying to have a sender pays model, why not punsih the people who run unprotected systems? Let Aunt Millie’s ISP bill her for being incompetent and running an open wireless network., or for being part of a botnet.

  15. Posted March 8, 2007 at 00:50 | Permalink

    Instead of blocking zombies outright, perhaps the Google home page could turn into an anti-virus download site if the connecting host is on the XBL…

  16. Rob Mueller
    Posted March 8, 2007 at 01:03 | Permalink

    I posted this same thing in the O’Reilly comments, but I’m repeating it here because I believe it really is important.

    Running an email service, we see the exact same issues with excessive SMTP connections and email floods due to backscatter when our domains are used on forged email. When you have botnets of >1 million machines sending spam, if they really tried to hit us hard, there would be virtually nothing we could do about it even with dozens and dozens of machines.

    There’s one simple solution that would basically solve the current spam problem.

    Block all port 25 connections from dialup/adsl hosts

    A lot of ISPs are already doing this, and it’s a good thing because it forces users to either use the ISPs email server via an authenticated SMTP session, or a 3rd party trusted server via an alternate SMTP port (either SSL one, or the “mail submission” port, or some arbitrary other one).

    This simple act completely changes the skew of the whole problem.

    • Botnets are immediately blocked from sending spam directly to other email systems
    • This means that spammers either have to setup their own systems, which have very limited IP sets and are blocked easily, or they have to change the way their botnets work

    Assuming they change the way their botnets work, they have a couple of options

    • Hijack user accounts to try and send via the ISPs SMTP. In this cases, the ISPs would quickly be able to detect users sending the spam, and block their machines completely and inform them that they have a virus on their machine. We kill two birds with one stone.
    • Start signing up and using free webmail type accounts to send spam (we’re already seeing this). Then it’s up to the webmail providers to detect and stop this. The poorly run ones will end up on block lists with a poor reputation, the smart ones solve the problem

    That would vastly, vastly reduce the spam problem almost immediately. The real problem is that ISPs don’t want to block port 25, because suddenly then something that wasn’t their problem, does become their problem. On the other hand, the amount of bandwidth ISPs must be loosing due to botnets must be huge, so you would think it would be in their interest to do it.

  17. David Malone
    Posted March 8, 2007 at 09:52 | Permalink

    I’m not sure the bilateral peerings between trusted parties will even be spammer-proof. To some extent BGP is secured that way (and money often changes hands). However, that didn’t stop spammers getting involved in prefix hijacking. I guess it isn’t an entirely fair comparison, because BGP has a pretty strong transitive trust thing going on.

  18. Posted March 8, 2007 at 10:20 | Permalink


    May not be 100% but sure will be better than nothing. The end to end pricible while nice is a pain sometimes.

    Prefix hijacking only happens because it is commerically tolerated. At some point the liability for allowing this will be large enough to make it worth stopping.

    I would be really shocked if the gmails/hotmails/yahoos/outblazes of the world do not have private mail exchange peerings going on already.

  19. Posted March 8, 2007 at 12:41 | Permalink

    Another counter-example to the bilateral peering solution to spam is usenet. (Remember where spam first appeared?)

    Although blocking port 25 will help with email spam to some extent, it will not deal with blog comment spam or DDOS attacks or other instances of the general problem of unwanted traffic.

  20. Derek Harding
    Posted March 9, 2007 at 07:44 | Permalink

    I don’t think the problem in this case is with SMTP but with the botnets. They can be utilized for all kinds of nefarious activities of which spam is only one.

    The only solution that I can see is for ISPs to take responsibility for the systems on their networks and what those systems do. This means monitoring, controlling & restricting connectivity for home systems just as (most) companies do for their desktops.

    The result for email is that home machines need to be prevented from sending email except through ISP relays. Will this screw with home server maintainers like me? Yep, but that’s just how it’s gotta be.

  21. Posted March 12, 2007 at 18:16 | Permalink

    I’ve solved this before with a commercial product called TrafficControl from a company named MailChannels. It essentially proxies SMTP connections and allowed us to go from 300 simultaneous connections to 3,000 on the same hardware!

    While it is commercial, it’s build upon a good Open Source foundation of Apache and mod_perl. It also has some neat spam filtering abilities itself, by throttling down the bandwidth to SMTP servers on RBLs, by OS, etc. that you configure. This works because most bots give up quickly if they aren’t getting good flow. I encourage you to check it out.

  22. Posted March 13, 2007 at 00:29 | Permalink

    I was having similar issues with zombies tying up all my Sendmail processes, during the fall, by opening up 8 or 10 connections each. I’ve since limited concurrent connections per IP to 2 or 3 (2 on most MXes) and haven’t had any problems with all the processes being tied up anymore. I also haven’t noticed any delays in receiving legit mail.

    It sounds like I’m supporting a similar number of users, a couple hundred across a dozen or so domains, so this may be an option for you.

  23. Posted March 14, 2007 at 23:53 | Permalink

    The “accidental DoS” caused by spam is probably the number one issue facing email admins these days and is one of the main reasons people are switching their edge protection solution. We found that in the last quarter of 2006, 5% of companies in the world switched their edge MTA. That’s really an astonishing number if you think about it — that one in five of companies would change their edge MTA on an annual basis.

    See the MTA survey for more details.

    Ken Simpson, CEO MailChannels

  24. Devdas Bhagat
    Posted March 15, 2007 at 13:33 | Permalink

    Ken, typo correction: 1 in 5 is 20%. 5% is 1 in 20.

    Derek, ISPs taking responsibility is absolutely the right answer. On the other hand, asking them to do this off the bat has a few other problems.

    The biggest problem here is that ISPs don’t control the end points, and there are very strong arguments for why they shouldn’t. So ISPs can only do so much to limit damage, and they can’t really prevent that damage.

    ISPs trying to control end points would definitely reduce innovation, and robably the freedom to run alternate operating systems.

    Microsoft could do a LOT to help in that respect by setting good defaults on their systems, and possibly allowing a very limited set of programs to run as administrator (This doesn’t need to be a DRM solution, only allowing programs to run as admin in “safe mode” will do the trick just as well. This includes installers as well..).