taint.org: Justin Mason’s Weblog

Amazon Cloudwatch:

This is nifty. Monitor EC2 instances and load balancers; CPU, data transfer rates, disk usage, disk activity, HTTP/TCP request counts/latency, “healthy/unhealthy” instances (see below). This data is both exposed via web service APIs, but also usable as input for their new “Auto Scaling” elastic scaling feature. Ideal for someone to write a Nagios plugin for. Also, I’m looking forward to some kick-ass sysadmin dataviz for this.

Auto-Scaling:

Elastically scale out (or in) your grid of EC2 instances, based on Amazon CloudWatch metrics. An officially-supported form of a myriad of third-party apps. I expect to hear of people accidentally spending a fortune due to accidental misuse of this ;)

Elastic Load Balancing:

Load balance across multiple EC2 instances, report metrics to Cloudwatch such as requests/second and request latency, and — most usefully of all in my opinion — shift traffic away from EC2 instances that fail to respond to a “health-check” HTTP GET with a 200, or fail to accept a TCP connection.

In other words, this provides a way to do decent HA on EC2, which is something that’s been much needed for a long time, and is quite tricky to set up using Linux-HA. I’ve done the latter, and found it full of potential reliability pitfalls; I found that Elastic IP addresses were not useful for quickly failing over to backup servers; in some cases, I found it taking about 5 minutes to fail over :( The only (relatively) snappy way to implement it was to set up a dynamic DNS record with a short TTL, point to it using a CNAME, and use “ddclient” to switch it when failing over. And even that could leave sites down for as long as it takes the DNS client to time out the existing cached CNAME.

Elastic Load Balancing supports HTTP or generic TCP connections. Unfortunately, it doesn’t support “real” termination of HTTPS connections, which is unfortunate. (You can terminate them as generic TCP connections, though.)

More details on the RightScale blog, at the AWS dev blog, and Werner Vogel’s blog.

Tags: amazon, auto-scaling, aws, cloudwatch, ec2, elastic, high-availability, http, load-balancing, scalability

Introducing Karmic Koala, Ubuntu 9.10:

What if you want to build an EC2-style cloud of your own? Of all the trees in the wood, a Koala’s favourite leaf is Eucalyptus. The Eucalyptus project, from UCSB, enables you to create an EC2-style cloud in your own data center, on your own hardware. It’s no coincidence that Eucalyptus has just been uploaded to universe and will be part of Jaunty – during the Karmic cycle we expect to make those clouds dance, with dynamically growing and shrinking resource allocations depending on your needs.

A savvy Koala knows that the best way to conserve energy is to go to sleep, and these days even servers can suspend and resume, so imagine if we could make it possible to build a cloud computing facility that drops its energy use virtually to zero by napping in the midday heat, and waking up when there’s work to be done. No need to drink at the energy fountain when there’s nothing going on. If we get all of this right, our Koala will help take the edge off the bear market.

AWESOME — exactly where the Linux server needs to go. Eucalyptus is the future of server farms. Really looking forward to this…

Tags: ec2, eucalyptus, karmic-koala, linux, ubuntu, virtualization

Recently, there’s been a bit of discussion online about whether or not it makes sense for companies to host server infrastructure at Amazon EC2, or on traditional colo infrastructure. Generally, these discussions have focussed on one main selling point of EC2: its elasticity, the ability to horizontally scale the number of server instances at a moment’s notice.

If you’re in a position to gain from elasticity, that’s great. But it is still worth noting that even if you aren’t in that position, there’s another good reason to host at an EC2-like cloud; if you want to deploy another copy of the app, either from a different version-control branch (dev vs staging vs production deployments), or to run separate apps with customizations for different customers. These aren’t scaling an existing app up, they’re creating new copies of the app, and EC2 works nicely to do this.

If you can deploy a set of servers with one click from a source code branch, this is entirely viable and quite useful.

Another reason: EC2-to-S3 traffic is extremely fast and cheap compared to external-to-S3. So if you’re hosting your data on S3, EC2 is a great way to crunch on it efficiently. Update: Walter observed this too on the backend for his Twitter Mosaic service.

Tags: amazon, aws, cloud-computing, ec2, hosting, s3, servers

A few days ago, Amazon announced that they would be supporting Windows on EC2. IMO, you’d have to be mad to dream of running a server on that platform, so I was totally like “meh”.

However, James Murty pointed out the perfect use case that I’d missed:

Although I much prefer “Unixy” platforms for my own development, I can imagine situations where it would be very handy to have a Windows machine easily available — such as for running those vital but irritating programs that are only made available for Windows. Australian Tax Office, I’m looking at you…

He’s spot on! This is a great use case. If you need to do a little ‘doze work, a quick recompile, or a connect to another stupid platform-limited service — indeed, like the Irish tax office’s Revenue Online Service, for that matter — simply fire up a ‘doze instance, do your hour’s work, SDelete any private files, and shut it down again. All of that will cost 12.5 cents.

This will save me a lot of pain with VMWare, I suspect…

More techie details at RightScale; a trial run.

Tags: amazon, aws, ec2, on-demand, utility-computing, vista, windows, xp

• Addictions, Think Amazon, not Google : Brian “Krow” Aker: ‘Google’s AppEngine is much closer to [...] “Digitial Sharecropping” [...] S3 and EC2 have little tie in to them. You can end up with a physical addiction to the services but the mental addiction to a framework does not exist. S3 is just storage and EC2 for most is just a hosted Linux image.’ +1!
  (tags: google gae aws ec2 amazon hosting s3)

• The Porterhouse to get a shiny new bottling line : hooray, great news for Irish beer drinkers sick of Guinness
  (tags: porterhouse beer ireland brewing)

Tags: amazon, aws, beer, brewing, ec2, gae, google, hosting, ireland, porterhouse, s3

• Ubuntu and Debian AMIs for Amazon EC2 : they look well-maintained
  (tags: amis debian ubuntu ec2 aws)

• Making an AMI use EBS as the root filesystem : using some nasty pivot_root hackery
  (tags: root filesystems aws ec2 amazon ebs disks linux)

• Richard Hulse: Clearing the Cache on Matrix : IPC::DirQueue used in the field as part of a cache-invalidation system for a NZ radio station
  (tags: ipc-dirqueue cache-revalidation users user-stories)

Tags: amazon, amis, aws, cache-revalidation, debian, disks, ebs, ec2, filesystems, ipc-dirqueue, linux, root, ubuntu, user-stories, users

• EC2 hack: make metadata visible in ‘ec2-describe-instances’ output : by creating one-off security groups to hold the metadata. hack, will be deprecated by AWS in a future release, but hey it works right now
  (tags: ec2 via:elasticgrid hacks patterns aws)

Tags: aws, ec2, hacks, patterns, via:elasticgrid

GoDaddy is rejecting mail with URLs that appear in the Spamhaus PBL. As this thread on the Amazon EC2 forum notes, this is creating false positives, causing nonspam mail to be rejected. Here’s what GoDaddy reportedly said about this policy:

Unfortunately, our system is set to reject mails sent from or including links listed in the SBL, PBL or XBL. Because the IP address associated to [REMOVED] is listed in the PBL, any emails containing a link to this site will be rejected. This includes plain-text emails including this information.

If this is true, it’s utterly broken.

Spamhaus explicitly warn that this is not to be done, on the PBL page:

Do not use PBL in filters that do any ‘deep parsing’ of Received headers, or for other than checking IP addresses that hand off to your mailservers.

And more explicitly in the Spamhaus PBL FAQ:

PBL should not be used for URI-based blocking! Consider the false positive potential: legitimate webservers hosted with services such as dyndns.com or ath.cx! Or consider that ISPs and other networks are encouraged to list any IP ranges which should not send mail, and that could include web servers! Use SBL or XBL (or sbl-xbl.spamhaus.org) for URI blocking as described in our Effective Spam Filtering section. Use PBL only for SMTP (mail).

Critically, the PBL now lists all Amazon EC2 space, since Spamhaus interpret Amazon’s policy as forbidding email to be delivered via direct SMTP from there. (Note — email, not HTTP.)

With this filter in place at GoDaddy, that now means that if you mail a URL of any page on any site hosted at EC2 to a user of GoDaddy, your mail won’t get through.

Note: this is much worse than blocks of SMTP traffic from EC2. In that case, an EC2 user can relay their legit SMTP traffic via an off-EC2 host. In this case, there is no similar option in HTTP that isn’t insufferably kludgy. :(

Tags: anti-spam, broken, ec2, filtering, godaddy, pbl, spamhaus, sucks

• Dublin’s new M50 electronic toll tags : what a mess! there are no less than 8 tag operators and a bewildering array of prices and penalties. typical
  (tags: ireland dublin m50 tolls roads pricing)

• Sustained IO on EBS == No Bueno : worrying stats for Amazon EBS data throughput, dropping from 160 MB/s to a rather paltry 42.4 MB/s
  (tags: amazon ebs ec2 speed networking hosting benchmarks)

• MXLogic on the economics of spam : Sam Masiello of MXLogic works out that phishers may be netting a 7300% profit margin; this is why spam’s not going away. mind you he does this by believing Gartner figures, which is never a good idea
  (tags: sam-masiello mxlogic via:tzink phishing spam money)

Tags: amazon, benchmarks, dublin, ebs, ec2, hosting, ireland, m50, money, mxlogic, networking, phishing, pricing, roads, sam-masiello, spam, speed, tolls, via:tzink

• 30% Of Internet Users Admit To Buying From Spam : ugh. we need a reminder of the Boulder Pledge. Mind you, Marshal have put out what appear to be inaccurate figures in the past regarding the Rustock botnet, so apply a pinch of salt
  (tags: marshal spam boulder-pledge commerce shopping statistics via:techdirt)

• more details on EC2 Elastic Block Store : achieves 70MB/s on an m1.small instance; ‘performance exceeds what we’ve seen for filesystems striped across the four local drives of x-large instances’. pretty good for a network filesystem, although not great compared to fast local SATA disks. also: snapshots are incremental and perform nicely compared to local S3 copy-and-upload
  (tags: ebs amazon performance benchmarks s3 ec2 disks snapshotting filesystems)

• Amazon Elastic Block Store (EBS) : ‘Prior to Amazon EBS, block storage within an Amazon EC2 instance was tied to the instance itself so that when the instance was terminated, the data within the instance was lost. Now with Amazon EBS, users can chose to allocate storage volumes that persist reliably and independently from Amazon EC2 instances.’ — can even snapshot to S3
  (tags: amazon ebs ec2 aws s3 cloud-computing hosting)

Tags: amazon, aws, benchmarks, boulder-pledge, cloud-computing, commerce, disks, ebs, ec2, filesystems, hosting, marshal, performance, s3, shopping, snapshotting, spam, statistics, via:techdirt

ZSFA — I Want The Mutt Of Feed Readers Zed recommends Newsbeuter. must take a look

We Want A Dead Simple Web Tablet For $200. Help Us Build It. having worked on a project to do just this, believe me, this is doomed. DOOMED

Science Clouds ‘compute cycles in the cloud for scientific communities .. allows you to provision customized compute nodes .. that you have full control over using a leasing model based on the Amazon’s EC2 service.’ Wonder if they’d like to give SA some time ;)

Tags: appliances, apps, atom, cloud-computing, ec2, feeds, grids, hardware, linux, news, newsbeuter, oh-dear, open-source, rss, scaling, science, software, spamassassin, tablets, techcrunch, virtualization, web, zed-shaw

Over the past few weeks, I’ve increasingly heard of spam and abuse problems originating in Amazon EC2.

This has culminated in a blog post yesterday by Brian Krebs at the Washington Post:

It took me by surprise this weekend to discover that that mounds of porn spam and junk e-mail laced with computer viruses are actively being blasted from digital real estate leased to [Amazon].

He goes on to discuss how EC2 space is now actively blocked by Outblaze, and has been listed by Spamhaus in their PBL list. A spokesperson for Amazon said:

“We have a clear acceptable use policy and whenever we have received a complaint of spam or malware coming through Amazon EC2, we have moved swiftly to strictly enforce the use policy by network isolating (or even terminating) any offending instances,” Kinton said. She added that Amazon has since taken action against the EC2 systems hosting the [malware].

However as Seth Breidbart noted in the comments, ‘note that Amazon will terminate the instance. That means that the spammer just creates another instance, which gets a new IP address, and continues spamming.’ True enough – as described, instance termination simply isn’t good enough.

My recommendations:

• as John Levine noted, it’s likely that Amazon need to treat EC2-originated traffic similarly to how an ISP treats their DSL pools – filtering outbound traffic for nastiness, in particular rate-limiting port 25/tcp connections on a per-customer basis, so that an instance run by (or infiltrated by) a spammer cannot produce massive quantities of spam before it is detected and cut off.

  However, I’m not talking about blocking port 25/tcp outbound entirely. That’s not appropriate — an EC2 instance is analogous to a leased colo box in a server farm, and not being able to send mail from our instances would really suck for EC2 users (like myself and my employers).

• It would help if there were a way to look up customer IDs from the IP address of the EC2 nodes they’re using — either via WHOIS or through rDNS. Even an opaque customer ID string would allow anti-abuse teams to correlate a single customer’s activity as they cycle through EC2 instances. This would allow those teams to deal with the reputation of Amazon’s customers, instead of Amazon’s own rep, analogous to how “traditional” hosters use SWIP to publicize their reassignments of IPs between their customers.

There’s some more discussion buried in a load of knee-jerking on the NANOG thread. Here’s a few good snippets:

Jon Lewis: ‘I got the impression the only thing Amazon considers abuse is use of their servers and not paying the bill. If you’re a paying customer, you can do whatever you like.’ (ouch.)

Ken Simpson: ‘IMHO, Amazon will eventually be forced to bifurcate their EC2 IP space into a section that is for “newbies” and a section for established customers. The newbie space will be widely black-listed, but will also have a lower rate of abuse complaint enforcement. The only scalable way to deal with a system like EC2 is to provide clear demarcations of where the crap is likely to originate from.’

Bill Herrin: ‘From an address-reputation perspective EC2 is no different than, say, China. Connections from China start life much closer to my filtering threshold that connections from Europe because a far lower percentage of the connections from China are legitimate. EC2 will get the same treatment.’

There’s also an earlier thread here.

Anyway, this issue is on fire — Amazon need to get the finger out and deal with it quickly and effectively, before EC2 does start to run into widespread blocks. I’m already planning migration of our mail-sending components off of EC2; we’re already seeing blocks of mail sent from it, and it’s looking likely that these will increase. :(

(It’s worth noting that a block of EC2’s netblocks today will produce a load of false positives, mainly on transactional mail, if you’re contemplating it. So I wouldn’t recommend it. But a lot of sites are willing to accept a few FPs, it seems.)

Tags: abuse, amazon, anti-spam, aws, blocklists, ec2, malware, policy

Mike Culver, Amazon’s “Web Services Evangelist”, will be in Dublin next week to evangelize about the goodness that is Amazon S3, EC2, SQS and so on. It seems he’ll be talking at the following locations:

• in the Auditorium of the Digital Exchange, Crane Street, Dublin 8 on Tuesday October 30th, 3-5pm; here’s a flyer the Amazonites have been passing around. (upcoming.org page)

• according to Damien, later that evening, he’s in the Westin Hotel on Westmoreland St., D2, starting at 7pm; note, it seems you need to book places at this, see Damien’s post.

• and again at the Irish Linux User’s Group on Thursday November 1st at 19:30 in the Irish Computer Society in Dublin (map).

I guess these are all going to be same talk, bar the Q&A ;)

There was some kind of an ICTE get-together mooted for Friday 2nd.

Also, the ILUG annual general meeting is scheduled on the following Saturday, 3rd November, also at the ICS. Gareth Eason notes ‘we’re hoping to start at 3pm sharp, with talks from Dave Wilson (HEAnet), Frank Duignan, John Looney (Google), and others, followed by a relaxing wind-down in the Schoolhouse pub later on.’ (upcoming.org page)

Hopefully I’ll get to at least one of the AWS talks (probably the Digital Exchange one) and the ILUG AGM… busy week!

Tags: amazon, aws, dublin, ec2, icte, ilug, ireland, linux, mike-culver, s3, social, talks

I had a bit of an epiphany while chatting to Antoin about the qpsmtpd/EC2 idea. Craig had the same thoughts.

Here’s the thing — there’s actually no need to offload the SMTP part at all. That stuff is tricky, since you’ve got to build in a lot of fault tolerance, quality-of-service, uptime, etc. to ensure that the MX really is reachable. Since an EC2 instance will lose its “disks” once rebooted/shut down, you need to store your queues in Amazon S3 — which has differing filesystem semantics from good old POSIX — so things get quite a bit hairier. On top of that, it requires a little RFC-breakage; there are issues with using CNAMEs in MX records, reportedly.

However, if we offload just the spamd part, it becomes a whole lot simpler. The SPAMD protocol will work fine across long distances, securely, with SSL encryption active, and SpamAssassin will work fine as a filtering system in an entirely stateless mode, with no persistent-across-reboots storage. (What about the persistent-storage aspects of spamd operation? There’s just the auto-whitelist, which can be easily ignored, and I haven’t trained a Bayes database in 2 years, so I doubt I’ll need that either ;)

If the spamd server is down or uncontactable, spamc will handle this and retry with another server, or eventually give up and pass the message through, safely intact (though unscanned).

Given that there’s a cool third-party ClamAV plugin now available for SpamAssassin, this system can offload the virus-scanning work, too.

So here’s the new plan: run the MTA, MX, and the super-lean “spamc” client on the normal MX machine — and offload the “spamd” work to one or more EC2 machines.

Basically, there would be a CNAME record in DNS, listing the dynamic DNS names of the EC2 spamd instances. Then, spamc is set to point at that CNAME as the spamd host to use. As EC2 instances are started/removed, they are added/removed from that CNAME list and spamc will automatically keep up.

Pricing is reasonably affordable — don’t send over-large messages to the EC2 spamd; rate-limit total incoming SMTP traffic in the MTA; and use the SPAMD protocol’s REPORT verb to reduce the bandwidth consumption of mails in transit by ensuring that the mail messages are only transmitted one-way, MX-to-EC2, instead of both MX-to-EC2 and EC2-to-MX. That will keep the bandwidth pricing down.

Recent figures indicate that I got about 90MB of mail per day, at peak, over the past weekend (which nearly DOS’d my server and caused some firefighting) – 68MB of spam, and 13MB of blowback. At 20 cents per GB, that’s 1.8 cents per day for traffic. Plus the $0.10 per instance hour, that’s $2.42 per day to run a single EC2 instance to handle DDOS spikes. Of course, that can be shut down when load is low.

Yep, this is looking very promising. Now when are Amazon going to let me onto the beta program for EC2?…

Tags: amazon, aws, ec2, spamassassin, spamc, spamd

Like a few other anti-spammers, I found myself under a hitherto-unprecedented level of spam blowback this weekend. Disappointingly, there are still thousands of SMTP servers configured to send bounce messages in response to spam.

Even with the anti-bounce ruleset for SpamAssassin, the volume was so great that our creaky old server had a lot of difficulty keeping up — once the messages got to SpamAssassin, the load issues had already been created. Also, Postfix’s anti-spam features really weren’t designed to deal with blowback.

While attempting to take some shortcuts in the setup on our server to deal with this, a great idea occurred to me — why not come up with an app that uses Amazon EC2 to flexibly provision enough server power and bandwidth to pre-filter the SMTP traffic for an MX under attack?

I’m basically thinking of qpsmtpd, with SpamAssassin and/or other antispam blobs active, running in an Amazon EC2 server image. Multiple images can be brought up, and added to the attacked domain’s MX record at an equal priority, to take load off the main (overloaded) MX.

Now to cogitate a little — details to follow…

Tags: amazon, anti-spam, aws, ec2, qpsmtpd, s3, smtp