Links for 2014-11-28

  • OS X doesn’t support ‘ndots’ DNS resolution

    “ping” will not append the “search” domains configured in /etc/resolv.conf. Apparently this has been broken since OS X Lion, no sign of a fix. Nice work Apple

    (tags: apple fail bugs resolv dns domains osx)

  • TCP incast

    a catastrophic TCP throughput collapse that occurs as the number of storage servers sending data to a client increases past the ability of an Ethernet switch to buffer packets. In a clustered file system, for example, a client application requests a data block striped across several storage servers, issuing the next data block request only when all servers have responded with their portion (Figure 1). This synchronized request workload can result in packets overfilling the buffers on the client’s port on the switch, resulting in many losses. Under severe packet loss, TCP can experience a timeout that lasts a minimum of 200ms, determined by the TCP minimum retransmission timeout (RTOmin).

    (tags: incast networking performance tcp bandwidth buffering switch ethernet capacity)

  • Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale | Engineering Blog | Facebook Code

    Excellent real-world war story from Facebook — a long-running mystery bug was eventually revealed to be a combination of edge-case behaviours across all the layers of the networking stack, from L2 link aggregation at the agg-router level, up to the L7 behaviour of the MySQL client connection pool.

    Facebook collocates many of a user’s nodes and edges in the social graph. That means that when somebody logs in after a while and their data isn’t in the cache, we might suddenly perform 50 or 100 database queries to a single database to load their data. This starts a race among those queries. The queries that go over a congested link will lose the race reliably, even if only by a few milliseconds. That loss makes them the most recently used when they are put back in the pool. The effect is that during a query burst we stack the deck against ourselves, putting all of the congested connections at the top of the deck.

    (tags: architecture debugging devops facebook layer-7 mysql connection-pooling aggregation networking tcp-stack)

  • “Macaroons” for fine-grained secure database access

    Macaroons are an excellent fit for NoSQL data storage for several reasons. First, they enable an application developer to enforce security policies at very fine granularity, per object. Gone are the clunky security policies based on the IP address of the client, or the per-table access controls of RDBMSs that force you to split up your data across many tables. Second, macaroons ensure that a client compromise does not lead to loss of the entire database. Third, macaroons are very flexible and expressive, able to incorporate information from external systems and third-party databases into authorization decisions. Finally, macaroons scale well and are incredibly efficient, because they avoid public-key cryptography and instead rely solely on fast hash functions.

    (tags: security macaroons cookies databases nosql case-studies storage authorization hyperdex)

  • Richard Tynan on Twitter: “GCHQ Tapping Eircom owned cable”

    Cable listed as owned by Eircom and Cable and Wireless (now Vodafone?)

    (tags: vodafone cables tapping surveillance eircom internet uk)

  • Hermitage: Testing the “I” in ACID

    [Hermitage is] a test suite for databases which probes for a variety of concurrency issues, and thus allows a fair and accurate comparison of isolation levels. Each test case simulates a particular kind of race condition that can happen when two or more transactions concurrently access the same data. Each test can pass (if the database’s implementation of isolation prevents the race condition from occurring) or fail (if the race condition does occur).

    (tags: acid architecture concurrency databases nosql)

This entry was posted in Uncategorized. Bookmark the permalink. Both comments and trackbacks are currently closed.


  1. Dave Pooser
    Posted November 29, 2014 at 20:32 | Permalink

    On the ‘ndots DNS’ issue, I think that may be a feature, not a bug. Consider this scenario: I set up a WiFi access port in a crowded area or compromise an existing public WiFi router (not too hard). I add a search domain for (As it happens, I have a wildcard cert for I also point to a DNS server that has will return NXDOMAIN for Finally, I add a DNS wildcard record pointing * to my phishing server.

    The customer types in “” or clicks on his own bookmark, because he’s too smart to click on emailed links. He carefully checks for cert errors, because he knows they’re a big deal. Seeing no errors, what are the odds he carefully reads the value in the URL window? Slim and none. And he types in his credentials, and I’ve got them, and my phishing site does a little redirect to the actual BOA “you have mistyped your password” link and he continues none the wiser. Meanwhile, I’ve got his banking credentials and at my leisure I can go buy all the catnip I can snort. (Don’t you judge me!)

    • Dave Pooser
      Posted November 29, 2014 at 20:39 | Permalink

      To clarify, I use my DHCP server or the one built into the compromised router to seed the search domain and the “special” DNS server.

  2. Posted December 1, 2014 at 16:03 | Permalink

    Dave, actually, that is a good point, particularly given the zillions of new TLDs added recently. Perhaps Apple decided to do this as a quietly-rolled-out security feature….