Eircom’s “DDOS”, or not

I woke up this morning to hear speculation on RTE Radio as to how Eircom’s DDOS woes were possibly being caused by the Russian mob, of all things. This absurd speculation is not helped by lines in statements like this:

‘The company blamed the problems on “an unusual and irregular volume of internet traffic” directed at its website, which affected the systems and servers that provide access to the internet for its customers.’

I’m speculating, too, but it seems a lot more likely to me that this isn’t just a DDOS, and someone — possibly just a lone Irish teenager — is running an attempted DNS cache-poisoning attack. Here’s why.

Last week, there were two features of the attack in reports: DDOS levels of traffic and incorrect pages coming up for some popular websites. To operate a Kaminsky DNS cache-poisoning attack requires buckets of packets — easily perceivable as DDOS levels. This level of traffic would be the first noticeable symptom on Eircom’s network management consoles, so it’d be easy to jump to the conclusion that a simple DDOS attack was the root cause.

This week, there’s just the DDOS levels of traffic. No cache poisoning effects have been reported. This would be consistent with Eircom’s engineers getting the finger out over the weekend, and upgrading the NSes to a non-vulnerable version. ;)

Once the attacker(s) realise this, they’ll probably stop the attack.

It’s not even a good attack for a bad guy to make, by the way. Given the timing, right after major press about a North Korean DDOS on US servers. it’s extremely high-profile, and made the news in several national newspapers (albeit in rather inept fashion). If someone wanted to make money from an attack, a massive-scale packet flood indistinguishable from a DDOS against the nation’s largest ISP is not exactly a subtle way to do it.

In the meantime, apparently OpenDNS have really seen the effects, with mass switchover of Eircom’s customers to the OpenDNS resolvers. Probably just as well…

This entry was posted in Uncategorized and tagged , , , , , . Bookmark the permalink. Both comments and trackbacks are currently closed.

11 Comments

  1. Keith
    Posted July 15, 2009 at 11:18 | Permalink

    It’s odd in part because it implies that the servers provided for Eircom customers aren’t firewalled off from externally originating traffic. There seems to me to be no reason for this except inertia.

    From my standpoint, the switchover to opendns is actually an unfortunate consequence of this attack. As I’ve mentioned in a comment here before, using opendns is usually a bad idea. They have a habit of trapping NXDOMAIN, they sometimes http proxy traffic to sites and, most importantly, their servers aren’t geographically distributed enough to give optimal endpoints for big services. Since recursive DNS resolvers don’t pass along the originating IP, geo-IP based balancing relies on the recursive resolver’s IP. OpenDNS don’t have servers that are local to Ireland and so you get bad endpoints for akamai, google et al.

    To see this, do ‘dig @208.67.222.222 http://www.google.com‘ and compare with the result from any more independent source.

  2. Posted July 15, 2009 at 12:38 | Permalink

    But what other recourse than OpenDNS does a punter have?

    I made a screencast on resetting Eircom DNS for my XP friends: http://www.flickr.com/photos/irisheyes/3719985673/

  3. Posted July 15, 2009 at 14:41 | Permalink

    Eircom’s DNS servers were unusable long before this little incident.

    “Normal operations” for users involve a 3-5 second pause while the steam engines on port 53 wake themselves.

    It’s a little amusing and a little depressing that even blogger types here with 2Mbps DSL lines have apparently never used a connection with a proper DNS resolver before. I can’t think of any other reason they’d put up with the at-least-weekly DNS server outages (no doubt Support told them to reboot their routers) and crappy latency.

  4. Posted July 15, 2009 at 14:53 | Permalink

    @John — yeah, the DNS resolvers have long been the weakest link for consumer home broadband ISPs in my experience. Eircom aren’t alone in this; a lot of ISPs are lousy at it. I had shitty results from Comcast when I was living in the US for example.

    Here’s an old report on the topic:

    http://blog.opendns.com/2006/08/17/cnet-reports-isps-arent-very-good-at-dns/

    e.g., pretty shocking stat — ‘Verizon drops 3.14% of all DSL subscribers’ DNS requests’. oh dear.

    Good news for OpenDNS though, I suppose!

  5. Posted July 15, 2009 at 15:31 | Permalink

    The Kaminsky attack takes seconds or less, against servers that didn’t randomise QID. It is against DNS servers that randomise the QID that the attack turns into a DDoS level of packets attack.

    Further in blog entry here, as the trackback/pingback doesn’t seem to have made it.

  6. Eamon O Sullivan
    Posted July 15, 2009 at 17:59 | Permalink

    I also thought it was cache poisoning, all attempts to access major sites went to bing searches for those sites

  7. Alan Clegg
    Posted July 16, 2009 at 02:17 | Permalink

    Even against completely random QID, the Kaminsky attack never needed to reach “DoS” levels — think in the range of 300 packets or less on average.

  8. Posted July 16, 2009 at 12:46 | Permalink

    Alan, I thought it was dependent on the DNS server software in use.

  9. Posted July 17, 2009 at 01:44 | Permalink

    one thing that this should have taught the irish registrar domainregistry.ie is that they need to secure what they can so when is .ie going to be signed for dnssec ?

    this combined with DKIM means that the world is a little more secure and things like this are less likely to affect ireland…

    regards

    John Jones http://www.johnjones.me.uk

  10. Posted July 17, 2009 at 11:16 | Permalink

    The Kaminsky attack allows attackers to have essentially infinite attempts to poison an NS for some given domain. Prior to that, it seemed attempts were rate-limited by the TTL in valid answers.

    On servers where the Query-ID response-cookie is predictable across queries (e.g. older BIND), the attack takes few packets (the amount needed scales with the busyness of the server). Servers where QID are not predictable across packets take longer, but it’s been shown it can be done within 10 hours. Cite is on my blog entry.

  11. Posted July 28, 2009 at 17:22 | Permalink

    Just received the following from Eircom. Kaminsky discovered his bug when, mid-2008?

    Dear Customer

    As you are aware, eircom customers experienced disruption to their Internet service over the last two weeks. On behalf of eircom, I apologise for the interruption to service and would like to take this opportunity to update you on the current position.

    In early July we saw an increase in incidents known as ‘cache poisoning’. This is a malicious activity by a third party to redirect customers to fake websites. We took immediate steps to protect our customers from this activity. This involved strengthening of the filters that block unwanted or suspect traffic. Unfortunately many customers could not, as a result of the incidents, access the internet or experienced delays for a number of hours on two specific occasions.

    While our preliminary investigations have shown that, in general, our systems worked as intended when the attack occurred, we are taking additional steps to further protect our customers and their internet service. This involves upgrading and replacing some of our server equipment. Service has been fully operational since 14th July and we continue to monitor the situation closely. We want to reassure customers that eircom treats the issue of Internet security very seriously and will continue to take all necessary steps to ensure that an incident such as this does not reoccur.

    We sincerely thank you again for your patience over the past two weeks as the measures outlined above were put into place, and apologise again for service interruption. We know that our customers have made eircom Ireland’s largest ISP because of its reputation for reliability and its ability to resolve issues when they arise.

    Should you have any queries or indeed any comments on this matter, please do not hesitate to contact our customer care team at [email protected]

    Thank you for your continued custom and support,

    Kind regards

    Gerry Culligan Director Consumer Market