How and why the leap second affected Cloudflare DNS
The root cause of the bug that affected our DNS service was the belief that time cannot go backwards. In our case, some code assumed that the difference between two times would always be, at worst, zero. RRDNS is written in Go and uses Go’s time.Now() function to get the time. Unfortunately, this function does not guarantee monotonicity. Go currently doesn’t offer a monotonic time source.
So the clock went “backwards”, s1 – s2 returned < 0, and the code couldn't handle it (because it's a little known and infrequent failure case). Part of the root cause here is cultural -- Google has solved the leap-second problem internally through leap smearing, and Go seems to be fundamentally a Google product at heart. The easiest fix in general in the "outside world" is to use "ntpd -x" to do a form of smearing. It looks like AWS are leap smearing internally (https://aws.amazon.com/blogs/aws/look-before-you-leap-the-coming-leap-second-and-aws/), but it is a shame they aren't making this a standard part of services running on top of AWS and a feature of the AWS NTP fleet.(tags: ntp time leap-seconds fail cloudflare rrdns go golang dns leap-smearing ntpd aws)
-
via twitter: “interesting conversation between author of a parenting book and the guy who introduced the concept of “flow”” — summary, family life is interrupt-driven (via nagging) and fundamentally hard to align with “flow”
-
wow
(tags: dick-cheney pacemakers iot internetofshit wireless security via:jzdziarski)