Amazon EC2′s spam and malware problems

Over the past few weeks, I’ve increasingly heard of spam and abuse problems originating in Amazon EC2.

This has culminated in a blog post yesterday by Brian Krebs at the Washington Post:

It took me by surprise this weekend to discover that that mounds of porn spam and junk e-mail laced with computer viruses are actively being blasted from digital real estate leased to [Amazon].

He goes on to discuss how EC2 space is now actively blocked by Outblaze, and has been listed by Spamhaus in their PBL list. A spokesperson for Amazon said:

“We have a clear acceptable use policy and whenever we have received a complaint of spam or malware coming through Amazon EC2, we have moved swiftly to strictly enforce the use policy by network isolating (or even terminating) any offending instances,” Kinton said. She added that Amazon has since taken action against the EC2 systems hosting the [malware].

However as Seth Breidbart noted in the comments, ‘note that Amazon will terminate the instance. That means that the spammer just creates another instance, which gets a new IP address, and continues spamming.’ True enough – as described, instance termination simply isn’t good enough.

My recommendations:

  • as John Levine noted, it’s likely that Amazon need to treat EC2-originated traffic similarly to how an ISP treats their DSL pools – filtering outbound traffic for nastiness, in particular rate-limiting port 25/tcp connections on a per-customer basis, so that an instance run by (or infiltrated by) a spammer cannot produce massive quantities of spam before it is detected and cut off.

    However, I’m not talking about blocking port 25/tcp outbound entirely. That’s not appropriate — an EC2 instance is analogous to a leased colo box in a server farm, and not being able to send mail from our instances would really suck for EC2 users (like myself and my employers).

  • It would help if there were a way to look up customer IDs from the IP address of the EC2 nodes they’re using — either via WHOIS or through rDNS. Even an opaque customer ID string would allow anti-abuse teams to correlate a single customer’s activity as they cycle through EC2 instances. This would allow those teams to deal with the reputation of Amazon’s customers, instead of Amazon’s own rep, analogous to how “traditional” hosters use SWIP to publicize their reassignments of IPs between their customers.

There’s some more discussion buried in a load of knee-jerking on the NANOG thread. Here’s a few good snippets:

Jon Lewis: ‘I got the impression the only thing Amazon considers abuse is use of their servers and not paying the bill. If you’re a paying customer, you can do whatever you like.’ (ouch.)

Ken Simpson: ‘IMHO, Amazon will eventually be forced to bifurcate their EC2 IP space into a section that is for “newbies” and a section for established customers. The newbie space will be widely black-listed, but will also have a lower rate of abuse complaint enforcement. The only scalable way to deal with a system like EC2 is to provide clear demarcations of where the crap is likely to originate from.’

Bill Herrin: ‘From an address-reputation perspective EC2 is no different than, say, China. Connections from China start life much closer to my filtering threshold that connections from Europe because a far lower percentage of the connections from China are legitimate. EC2 will get the same treatment.’

There’s also an earlier thread here.

Anyway, this issue is on fire — Amazon need to get the finger out and deal with it quickly and effectively, before EC2 does start to run into widespread blocks. I’m already planning migration of our mail-sending components off of EC2; we’re already seeing blocks of mail sent from it, and it’s looking likely that these will increase. :(

(It’s worth noting that a block of EC2′s netblocks today will produce a load of false positives, mainly on transactional mail, if you’re contemplating it. So I wouldn’t recommend it. But a lot of sites are willing to accept a few FPs, it seems.)

This entry was posted in Uncategorized and tagged , , , , , , , . Bookmark the permalink. Both comments and trackbacks are currently closed.

20 Comments

  1. fred arnold
    Posted July 2, 2008 at 20:32 | Permalink

    WTF is EC2? Yes I can look it up, but since it’s crucial to understanding the story, a brief description seems appropriate.

  2. Bob Howdy
    Posted July 2, 2008 at 20:40 | Permalink

    No doubt, some kind of description of EC2 would be good…

  3. Posted July 2, 2008 at 20:51 | Permalink

    “Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale computing easier for developers.” — From The Elastic Compute Cloud homepage

  4. Joar
    Posted July 2, 2008 at 20:52 | Permalink

    It’s pretty easy to extrapolate from the text what EC2 is.. Just read and think.

  5. xet7
    Posted July 2, 2008 at 20:55 | Permalink

    Definition: Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale computing easier for developers. http://aws.amazon.com/ec2

  6. Dan
    Posted July 2, 2008 at 21:10 | Permalink

    EC2 is Elastic Cloud Computing. It’s a system which allows you to create a virtual server instance and set up a website. You pay based on the amount of space and traffic, with low traffic sites being extremely cheap. Because you’re running on a virtual machine and not a real server, you have full root control over your site and can install any software you like, from the operating system on up to web servers, frameworks, etc. An average person who’s interested in web design or services can set up a hobby site or play around with ideas and pay almost nothing. If you get lucky and have a hit on your hand, you can add more instances to handle the traffic and pay Amazon based on the amount of traffic you get. The problem, of course, is that such a system is ripe for abuse if not policed. The positive side is that you must create an account to pay Amazon, so there IS a trail back to you. Amazon just needs to set up a system to allow malware trafficers and spammers to be held accountable and not to just kill the virtual machine that’s causing the problem, since the spammer can just fire up a new virtual machine, which will be automatically assigned a new IP, and go right back to what he was doing.

  7. Posted July 2, 2008 at 21:22 | Permalink

    sorry about that guys — I’ve added a link to http://aws.amazon.com/ec2 to clarify that.

  8. Anon C
    Posted July 2, 2008 at 21:40 | Permalink

    Is Ec2 incoherent to you?

    duh :)

    T”he weblog of Justin Mason; incoherent ramblings about Apache SpamAssassin, anti-spam, perl, software development, and the web, from an Irish software developer.”

  9. Posted July 2, 2008 at 21:44 | Permalink

    EC2 is a service by Amazon where people can pay (fairly low) fees to execute code on Amazon’s server farm.

    This is certainly Amazon’s problem, and they need step up and take responsibility. Their terms of use already prohibit spamming, and they have actual identity (in the form of real payments received) for their users, so they are in a position to solve it if they choose to do so.

  10. Posted July 2, 2008 at 23:06 | Permalink

    I’m a Linux system administrator. My resume includes some of the big names on the internet (Google, for instance) and I work daily with virtual systems, so I am qualified enough to say the following….

    Who’s the moron who set up a system that allows any one to keep creating instant spam boxes? Holy f****! When you set the damn thing up an instance should have automatically been tied to the account and the account LOCKED when an instance is killed due to abuse.

  11. DavidW
    Posted July 2, 2008 at 23:55 | Permalink

    I’ve been seeing web server scans coming from the EC2 IP space for a while now. It was only a matter of time before the service was used as bot for scanning, spamming, and malware distribution.

  12. aj r
    Posted July 3, 2008 at 00:06 | Permalink

    I had a server getting hit many, many times by an Amazon-owned IP probing for default user accounts via SSH. The server appears offline now, but it was at 67.202.12.24 . ‘whois’ showed that it was owned by Amazon and in use by “Amazon Development Centre South Africa” but ‘traceroute’ led back up to Washington, not SA.

  13. hostyle
    Posted July 3, 2008 at 00:29 | Permalink

    Does EC2 allow / provide windows installs? Or is this spammers actively buying bandwidth? I’m too lazy to check, but curious if this is just more botnetted unsecured windows installs – albeit inside Amazons infrastucture – or if its spammers becoming a tiny bit more public…

  14. Jack
    Posted July 3, 2008 at 03:30 | Permalink

    Regulation of the Internet the way ISPs and system administrators do it is dumb. We waste more resources fighting spam than it is worth. Amazon would be smart to just let it happen. It would teach people they can’t force others to do regulation. Regulation doesn’t work anyway. Fighting spam by black listing, legal processes, and other similar means is just plain stupid. If administrators want to resolve the issue then they need to get together and propose NEW standards. Not standards that required regulation or identification. At least not the kind of identification where you are relying on a human being. Identification needs to be done through a trust system. Those who build up trust can send as much mail as they want. Those who are new need not be trusted until they have sent lots of mail to trustworthy parties. If everybody is in agreement on who is trustworthy then we won’t have spam. If we aren’t in agreement they can’t spam because we’ll individually decide at which threshold to not receive mail from less trustworthy parties. For the stupid people all they’d have to do is reject all mail from new users. Once somebody who is trusted has said “i trust this person” by either accepting all mail or individually adding them to an exceptions list others will be able to receive mail if the threshold is low. If enough users trust this person who are trusted by enough other users you have a system where spam won’t exist.

  15. Robin
    Posted July 3, 2008 at 04:38 | Permalink

    Maybe Amazon is spamming on purpose eh

    Time to sue amazon for junk mail haha

  16. lane
    Posted July 3, 2008 at 06:14 | Permalink

    I get most of your points, but early on you say it is “inappropriate” for an ISP to block outbound port 25 traffic.

    Why?

    just because it “really sucks” not to be able to send outbound port 25 traffic, does not, in my mind, rise to the level of corporate policy concern.

    As I understand it, Earthlink does such blocking, requiring their users to authenticate to an earthlink.net server before sending outbound port 25 traffic.

    Not that earthlink is the perfect model, by no means.

    But it seems to me perfectly legitimate for an ISP (or any public IP provider) to force all port 25 traffic to a specific set of hosts in order to prevent (not just reduce, but absolutely prevent) unauthorized port 25 traffic. If more providers of email accounts were to implement such policy then SPAM would become more like a bad memory than the constant plague that it has become.

    Bozos with blogs should maybe champion the idea, rather than make long-winded posts about how “inappropriate” it is for a business to mind its own.

  17. Posted July 3, 2008 at 09:48 | Permalink

    @lane: I was talking specifically about Amazon EC2, not about ISPs in general. Since you failed to parse that, looks like you’re the bozo… ;)

  18. Posted July 3, 2008 at 17:52 | Permalink

    I found a while ago that a lot of sites block mail from EC2, I recommend relaying it through an SMTP provider. I’ve written a blog article about this: http://pauldowman.com/2008/02/17/smtp-mail-from-ec2-web-server-setup/

  19. Posted July 14, 2008 at 12:19 | Permalink

    by the way, there’s a long thread of comments about this article over on the closed Cloud Computing mailing list:

    http://groups.google.com/group/cloud-computing/browse_thread/thread/ff9790036428d63e

    Most of the posters there miss the point that EC2 differs from conventional colo hosting in that it’s trivial for an EC2 user to perform “whackamole” filter evasion by shutting down and starting up EC2 instances up to many times per minute, whereas in the conventional colo picture, that required multi-day manual hardware installs and changed contracts — hence was almost impossible without explicit ISP cooperation with the spammer. In other words, EC2 has a whole new type of problem, needing new fixes.

    (The thread also diverges into “is a botnet a form of cloud computing?” The answer is, of course, yes. ;)

  20. Andrew
    Posted February 20, 2011 at 00:19 | Permalink

    For anyone who wants to know if their ISP is blocking outbound port 25 try the online test at http://port25.icannotconnect.com