New EC2 Features

Amazon Cloudwatch:

This is nifty. Monitor EC2 instances and load balancers; CPU, data transfer rates, disk usage, disk activity, HTTP/TCP request counts/latency, “healthy/unhealthy” instances (see below). This data is both exposed via web service APIs, but also usable as input for their new “Auto Scaling” elastic scaling feature. Ideal for someone to write a Nagios plugin for. Also, I’m looking forward to some kick-ass sysadmin dataviz for this.

Auto-Scaling:

Elastically scale out (or in) your grid of EC2 instances, based on Amazon CloudWatch metrics. An officially-supported form of a myriad of third-party apps. I expect to hear of people accidentally spending a fortune due to accidental misuse of this ;)

Elastic Load Balancing:

Load balance across multiple EC2 instances, report metrics to Cloudwatch such as requests/second and request latency, and — most usefully of all in my opinion — shift traffic away from EC2 instances that fail to respond to a “health-check” HTTP GET with a 200, or fail to accept a TCP connection.

In other words, this provides a way to do decent HA on EC2, which is something that’s been much needed for a long time, and is quite tricky to set up using Linux-HA. I’ve done the latter, and found it full of potential reliability pitfalls; I found that Elastic IP addresses were not useful for quickly failing over to backup servers; in some cases, I found it taking about 5 minutes to fail over :( The only (relatively) snappy way to implement it was to set up a dynamic DNS record with a short TTL, point to it using a CNAME, and use “ddclient” to switch it when failing over. And even that could leave sites down for as long as it takes the DNS client to time out the existing cached CNAME.

Elastic Load Balancing supports HTTP or generic TCP connections. Unfortunately, it doesn’t support “real” termination of HTTPS connections, which is unfortunate. (You can terminate them as generic TCP connections, though.)

More details on the RightScale blog, at the AWS dev blog, and Werner Vogel’s blog.

Tags: , , , , , , , , ,

Comments

Good reasons to host inelastically on EC2

Recently, there’s been a bit of discussion online about whether or not it makes sense for companies to host server infrastructure at Amazon EC2, or on traditional colo infrastructure. Generally, these discussions have focussed on one main selling point of EC2: its elasticity, the ability to horizontally scale the number of server instances at a moment’s notice.

If you’re in a position to gain from elasticity, that’s great. But it is still worth noting that even if you aren’t in that position, there’s another good reason to host at an EC2-like cloud; if you want to deploy another copy of the app, either from a different version-control branch (dev vs staging vs production deployments), or to run separate apps with customizations for different customers. These aren’t scaling an existing app up, they’re creating new copies of the app, and EC2 works nicely to do this.

If you can deploy a set of servers with one click from a source code branch, this is entirely viable and quite useful.

Another reason: EC2-to-S3 traffic is extremely fast and cheap compared to external-to-S3. So if you’re hosting your data on S3, EC2 is a great way to crunch on it efficiently. Update: Walter observed this too on the backend for his Twitter Mosaic service.

Tags: , , , , , ,

Comments

the on-demand Windows desktop

A few days ago, Amazon announced that they would be supporting Windows on EC2. IMO, you’d have to be mad to dream of running a server on that platform, so I was totally like “meh”.

However, James Murty pointed out the perfect use case that I’d missed:

Although I much prefer “Unixy” platforms for my own development, I can imagine situations where it would be very handy to have a Windows machine easily available — such as for running those vital but irritating programs that are only made available for Windows. Australian Tax Office, I’m looking at you…

He’s spot on! This is a great use case. If you need to do a little ‘doze work, a quick recompile, or a connect to another stupid platform-limited service — indeed, like the Irish tax office’s Revenue Online Service, for that matter — simply fire up a ‘doze instance, do your hour’s work, SDelete any private files, and shut it down again. All of that will cost 12.5 cents.

This will save me a lot of pain with VMWare, I suspect…

More techie details at RightScale; a trial run.

Tags: , , , , , , ,

Comments (5)

Links for 2008-10-08

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Comments

Links for 2008-09-24

Tags: , , , , , , , , , ,

Comments

Links for 2008-09-21

Tags: , , , , ,

Comments

Links for 2008-09-10

Tags: , , , , , , , , , , , , , ,

Comments

AWS event in Dublin’s Digital Hub

Brian Scanlan mailed me with this blurb, worth blogging for any AWS users in the Dublin area:

  • Are you a software developer or IT professional working in the Dublin area?

  • Would you like to learn more about Amazon Web Services?

Amazon spent over ten years developing a world-class technology and content platform that powers Amazon web sites for millions of customers daily. Most people think “Amazon.com” when they hear the work; however developers are excited to learn that there is a separate arm of the company, known as Amazon Web Services or AWS.

Using AWS, developers can build software applications leveraging the same robust, scalable and reliable technology that powers Amazon’s retail business.

Amazon Data Services Ireland are delighted to welcome Simone Brunozzi (simoneb at amazon.com), AWS Evangelist for Europe, to Dublin, where he will give an overview of Amazon Web Services, including S3, EC2 and EBS, SimpleDB and more.

Tuesday 16th September 2008 at 7pm, The Digital Exchange Auditorium, Crane Street, Dublin 8

Maps and directions to the venue are here. Refreshments will be served.

All welcome – but places are limited, so please sign-up by mailing aws-dublin-event at amazon.com before Thursday 11th September.

I have no connection to this; not even sure if I’ll be going, as I went to the last one anyway and it was a bit short on technical tips ;) . But worth blogging anyway.

Tags: , , , ,

Comments (2)

Links for 2008-08-28

Tags: , , , , , , , , , , , , , , , , , ,

Comments

Links for 2008-08-21

Tags: , , , , , , , , , , , , , , , , , ,

Comments

Links for 2008-08-07

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Comments (1)

Amazon EC2’s spam and malware problems

Over the past few weeks, I’ve increasingly heard of spam and abuse problems originating in Amazon EC2.

This has culminated in a blog post yesterday by Brian Krebs at the Washington Post:

It took me by surprise this weekend to discover that that mounds of porn spam and junk e-mail laced with computer viruses are actively being blasted from digital real estate leased to [Amazon].

He goes on to discuss how EC2 space is now actively blocked by Outblaze, and has been listed by Spamhaus in their PBL list. A spokesperson for Amazon said:

“We have a clear acceptable use policy and whenever we have received a complaint of spam or malware coming through Amazon EC2, we have moved swiftly to strictly enforce the use policy by network isolating (or even terminating) any offending instances,” Kinton said. She added that Amazon has since taken action against the EC2 systems hosting the [malware].

However as Seth Breidbart noted in the comments, ‘note that Amazon will terminate the instance. That means that the spammer just creates another instance, which gets a new IP address, and continues spamming.’ True enough – as described, instance termination simply isn’t good enough.

My recommendations:

  • as John Levine noted, it’s likely that Amazon need to treat EC2-originated traffic similarly to how an ISP treats their DSL pools – filtering outbound traffic for nastiness, in particular rate-limiting port 25/tcp connections on a per-customer basis, so that an instance run by (or infiltrated by) a spammer cannot produce massive quantities of spam before it is detected and cut off.

    However, I’m not talking about blocking port 25/tcp outbound entirely. That’s not appropriate — an EC2 instance is analogous to a leased colo box in a server farm, and not being able to send mail from our instances would really suck for EC2 users (like myself and my employers).

  • It would help if there were a way to look up customer IDs from the IP address of the EC2 nodes they’re using — either via WHOIS or through rDNS. Even an opaque customer ID string would allow anti-abuse teams to correlate a single customer’s activity as they cycle through EC2 instances. This would allow those teams to deal with the reputation of Amazon’s customers, instead of Amazon’s own rep, analogous to how “traditional” hosters use SWIP to publicize their reassignments of IPs between their customers.

There’s some more discussion buried in a load of knee-jerking on the NANOG thread. Here’s a few good snippets:

Jon Lewis: ‘I got the impression the only thing Amazon considers abuse is use of their servers and not paying the bill. If you’re a paying customer, you can do whatever you like.’ (ouch.)

Ken Simpson: ‘IMHO, Amazon will eventually be forced to bifurcate their EC2 IP space into a section that is for “newbies” and a section for established customers. The newbie space will be widely black-listed, but will also have a lower rate of abuse complaint enforcement. The only scalable way to deal with a system like EC2 is to provide clear demarcations of where the crap is likely to originate from.’

Bill Herrin: ‘From an address-reputation perspective EC2 is no different than, say, China. Connections from China start life much closer to my filtering threshold that connections from Europe because a far lower percentage of the connections from China are legitimate. EC2 will get the same treatment.’

There’s also an earlier thread here.

Anyway, this issue is on fire — Amazon need to get the finger out and deal with it quickly and effectively, before EC2 does start to run into widespread blocks. I’m already planning migration of our mail-sending components off of EC2; we’re already seeing blocks of mail sent from it, and it’s looking likely that these will increase. :(

(It’s worth noting that a block of EC2’s netblocks today will produce a load of false positives, mainly on transactional mail, if you’re contemplating it. So I wouldn’t recommend it. But a lot of sites are willing to accept a few FPs, it seems.)

Tags: , , , , , , ,

Comments (19)

Upcoming Mike Culver talk about AWS

Mike Culver, Amazon’s “Web Services Evangelist”, will be in Dublin next week to evangelize about the goodness that is Amazon S3, EC2, SQS and so on. It seems he’ll be talking at the following locations:

  • in the Auditorium of the Digital Exchange, Crane Street, Dublin 8 on Tuesday October 30th, 3-5pm; here’s a flyer the Amazonites have been passing around. (upcoming.org page)

  • according to Damien, later that evening, he’s in the Westin Hotel on Westmoreland St., D2, starting at 7pm; note, it seems you need to book places at this, see Damien’s post.

  • and again at the Irish Linux User’s Group on Thursday November 1st at 19:30 in the Irish Computer Society in Dublin (map).

I guess these are all going to be same talk, bar the Q&A ;)

There was some kind of an ICTE get-together mooted for Friday 2nd.

Also, the ILUG annual general meeting is scheduled on the following Saturday, 3rd November, also at the ICS. Gareth Eason notes ‘we’re hoping to start at 3pm sharp, with talks from Dave Wilson (HEAnet), Frank Duignan, John Looney (Google), and others, followed by a relaxing wind-down in the Schoolhouse pub later on.’ (upcoming.org page)

Hopefully I’ll get to at least one of the AWS talks (probably the Digital Exchange one) and the ILUG AGM… busy week!

Tags: , , , , , , , , , , ,

Comments (6)

SpamAssassin as an EC2 service

I had a bit of an epiphany while chatting to Antoin about the qpsmtpd/EC2 idea. Craig had the same thoughts.

Here’s the thing — there’s actually no need to offload the SMTP part at all. That stuff is tricky, since you’ve got to build in a lot of fault tolerance, quality-of-service, uptime, etc. to ensure that the MX really is reachable. Since an EC2 instance will lose its “disks” once rebooted/shut down, you need to store your queues in Amazon S3 — which has differing filesystem semantics from good old POSIX — so things get quite a bit hairier. On top of that, it requires a little RFC-breakage; there are issues with using CNAMEs in MX records, reportedly.

However, if we offload just the spamd part, it becomes a whole lot simpler. The SPAMD protocol will work fine across long distances, securely, with SSL encryption active, and SpamAssassin will work fine as a filtering system in an entirely stateless mode, with no persistent-across-reboots storage. (What about the persistent-storage aspects of spamd operation? There’s just the auto-whitelist, which can be easily ignored, and I haven’t trained a Bayes database in 2 years, so I doubt I’ll need that either ;)

If the spamd server is down or uncontactable, spamc will handle this and retry with another server, or eventually give up and pass the message through, safely intact (though unscanned).

Given that there’s a cool third-party ClamAV plugin now available for SpamAssassin, this system can offload the virus-scanning work, too.

So here’s the new plan: run the MTA, MX, and the super-lean “spamc” client on the normal MX machine — and offload the “spamd” work to one or more EC2 machines.

Basically, there would be a CNAME record in DNS, listing the dynamic DNS names of the EC2 spamd instances. Then, spamc is set to point at that CNAME as the spamd host to use. As EC2 instances are started/removed, they are added/removed from that CNAME list and spamc will automatically keep up.

Pricing is reasonably affordable — don’t send over-large messages to the EC2 spamd; rate-limit total incoming SMTP traffic in the MTA; and use the SPAMD protocol’s REPORT verb to reduce the bandwidth consumption of mails in transit by ensuring that the mail messages are only transmitted one-way, MX-to-EC2, instead of both MX-to-EC2 and EC2-to-MX. That will keep the bandwidth pricing down.

Recent figures indicate that I got about 90MB of mail per day, at peak, over the past weekend (which nearly DOS’d my server and caused some firefighting) – 68MB of spam, and 13MB of blowback. At 20 cents per GB, that’s 1.8 cents per day for traffic. Plus the $0.10 per instance hour, that’s $2.42 per day to run a single EC2 instance to handle DDOS spikes. Of course, that can be shut down when load is low.

Yep, this is looking very promising. Now when are Amazon going to let me onto the beta program for EC2?…

Tags: , , , , ,

Comments (13)

Using qpsmtpd and Amazon EC2 to provide SMTP-DDoS protection

Like a few other anti-spammers, I found myself under a hitherto-unprecedented level of spam blowback this weekend. Disappointingly, there are still thousands of SMTP servers configured to send bounce messages in response to spam.

Even with the anti-bounce ruleset for SpamAssassin, the volume was so great that our creaky old server had a lot of difficulty keeping up — once the messages got to SpamAssassin, the load issues had already been created. Also, Postfix’s anti-spam features really weren’t designed to deal with blowback.

While attempting to take some shortcuts in the setup on our server to deal with this, a great idea occurred to me — why not come up with an app that uses Amazon EC2 to flexibly provision enough server power and bandwidth to pre-filter the SMTP traffic for an MX under attack?

I’m basically thinking of qpsmtpd, with SpamAssassin and/or other antispam blobs active, running in an Amazon EC2 server image. Multiple images can be brought up, and added to the attacked domain’s MX record at an equal priority, to take load off the main (overloaded) MX.

Now to cogitate a little — details to follow…

Tags: , , , , , ,

Comments (3)

Buying Consumer Electronics Online, in Ireland?

Hey lazyweb, hear my plea! What are my options for buying consumer electronics online, now that I’m back in Ireland?

I like online shopping. I dislike Argos, and I really hate Dixons, Currys and all the rest of the consumer-electronics high-street operations. Get me on the net and out of the nasty little shops and I’m happy. ;)

All in all, I’m a bit of an Amazon fan. However, now that I’m back in Ireland, I’ve been brought back to earth with a bang on that count; the prices are OK for items at both Amazon.com and .co.uk — but shipping is turning out to be a total disaster.

Basically, I’ve put in two orders, paid through the nose for basic shipping, and neither has turned up. For example — I ordered this phone a week and a half ago, on the 9th March, ponying up UKP 27 for the item — and a painful UKP 7 for shipping by International Mail.

Delivery estimate on ordering was for between 5 and 7 days — 14th to the 16th March. That was long enough — but it still hasn’t turned up, and Amazon.co.uk is still claiming that that is the current estimate, despite the 16th of March being 4 days ago ;)

On top of that, it appears they don’t offer any way to track the packages using that shipping method, so who knows what’s happening with the damn thing right now.

If I compare that with an order I made at Amazon.com last November, in which I nabbed a handy FM transmitter for my iPod — in that case, I got it shipped by plain old US Postal Service for $4.51, which was handily discounted as Super Saver Shipping. That — as with pretty much all my Amazon.com orders — arrived in 3-4 days, and for a hell of a lot cheaper too. If I’d had to pay for shipping (which I didn’t anyway), $4.51 vs UKP 7 works out as a third of the price, no less.

I’m guessing this is mainly down to Amazon.co.uk being shoddy in terms of how it deals with shipping to Ireland, and there are probably sites that use better-quality shipping partners.

Surely there must be better deals with vendors in Ireland, or even elsewhere in the Eurozone? Anyone know? Please drop us a line in the comments!

Update: the items arrived — 14 days after ordering. This is a moot point now, though, since Amazon.co.uk are no longer selling ‘PC & Video Games, Toys & Games, Gift items, Electronics & Photo and Home & Garden items’ to Ireland; I guess it was easier to give up on the Irish market for now. Very disappointing — but I’m waiting to see what happens next.

Tags: , , , , ,

Comments (30)

Monday morning quickies – gifts patented

FFII have discovered that Amazon.com have received a patent from the EPO ‘which covers all computerised methods of automatically delivering a gift to a third party’. It seems to cover Amazon’s ‘One-Click’ ordering system, as well.

Wierd: Tiny town to reek of sex. Don’t get excited — it’s only moth pheromones. (via Peter Darben on the forteana list.)

Medical slang, including:

  • ATS: Acute Thespian Syndrome
  • Departure lounge — Geriatric ward
  • DBI: Dirtbag index (calculated by the number of tattoos on the body multiplied by number of recent missing teeth, to estimate days without a bath)
  • NFN: Normal for Norfolk
  • Pumpkin positive: When you shine a penlight into the patient’s mouth and his brain is so small his whole head lights up
  • PFO: Pissed, fell over
  • Scepticaemia: What doctors develop with experience

And — finally! — an explanation for that ER term:

  • Stat: Immediately, shortened from the Latin statim

Linux: GrokLaw on SCO and Sun’s Linux indemnification FUD. Well worth a read – especially the bit where Mr. GrokLaw finds an old SCO contract that does include indemnification terms. Indemnification, that is, with some pretty serious get-out clauses and stings in the tail.

Weather: Mont Blanc closed due to record heatwave. ‘This year, for the first time since its conquest in 1786, the heatwave has made western Europe’s highest peak too dangerous to climb. Mont Blanc is closed. The conditions have been so extreme, say glaciologists and climate experts, and the retreat of the Alps’ eternal snows and glaciers so pronounced, that the range — and its multi-billion-pound tourist industry — may never fully recover.’

Food: Cooking for the Mafia. ‘Conrad Gallagher was the highest flier in the gaudy firmament of New Ireland. A Michelin star at the age of 26, and a swank restaurant, called Peacock Alley’. Not too long afterwards, things had not gone so well — he was in the Brooklyn Detention Centre. Pretty terrifying article — a US jail is not one of the nicest places in the world…

Spam: The Howard Dean election campaign ran into a wrinkle last week — and pretty soon was apparently ‘joe-jobbed’. This one is going to get interesting, if the Dean campaign follow up, as joe-jobbing an election campaign is in violation of federal election law, and is apparently taken quite seriously.

Reminder: keep an eye on Spamvertized.Org for the latest news in political spam!

Tags: , , , , , , , , ,

Comments

Thanks Sergi!

So an Amazon parcel just arrived, containing Bruce Schneier’s Secrets and Lies, with a little ‘thank you’ note from someone called Sergi — a Sitescooper and WebMake fan. Thanks Sergi! (and thanks, Amazon wishlist ;)

Of course, now I feel guilty for neglecting those apps, in favour of SpamAssassin. Someday I’ll get them all up-to-date…

Tags: , , , , , , , , ,

Comments

Amazon Web Services

Tim O’Reilly: Killer Apps Share A Common Thread: Hacker Geeks.

The really interesting bit in this is the discussion of the Amazon Web Services:

Rob Federick, senior technology manager for Amazon.com, asked for a show of hands for those in the room who considered Amazon.com to be a retailer business and those who considered it to be a technology platform. O’Reilly was amongst the few who raised hands in support of the latter.

It didn’t start out that way. But Amazon soon discovered developers taking the Amazon interface and adding their own ideas. A 19-year-old developer from Romania, ‘Catlin,’ began designing store fronts that looked like the Amazon.com site, and then allowing other developers to download the source code for free.

‘We are allowing people to create and innovate in ways that Amazon.com cannot do on its own,’ Federick said.

This is incredibly significant, and shows how Amazon’s leadership has a totally different vision compared to other online retailers. The others take the ‘Altavista view’ — they want to lock their users ‘in the trunk’ as Dave Winer says; users stay on the retailer’s site, aggregators and price-comparison engines are locked out, having to jump through hacky screen-scraping hoops, etc.

In contrast, Amazon are more than happy to let other sites scrape their content using their web services, even if this could be used to show how other sites have lower prices, or possibly lose them sales. Wow. I’m sure that was hard to sell internally, but it’s a great move.

Spam: Reg: new spam trojan, called Proxy-Guzu. Yet another. :(

Tags: , , , , , , , , ,

Comments

Region coding is evil

goddammit. Just got a PS2 for my birthday (wahoo!), and immediately thought about getting hold of the Koyaanisqatsi/Powaqqatsi 2-pack DVD. But it’s region-coded to US/Canada only in the edition on Amazon.com, and not available at all at Amazon.co.uk. Region coding is evil.

Of course, I could buy it somewhere else — but I wasn’t planning to buy it, I was looking to set up an Amazon.com wishlist!

Tags: , , , , , , , , ,

Comments