Links for 2014-10-30

  • IT Change Management

    Stephanie Dean on Amazon’s approach to CMs. This is solid gold advice for any company planning to institute a sensible technical change management process

    (tags: ops tech process changes change-management bureaucracy amazon stephanie-dean infrastructure)

  • Stephanie Dean on event management and incident response

    I asked around my ex-Amazon mates on twitter about good docs on incident response practices outside the “iron curtain”, and they pointed me at this blog (which I didn’t realise existed). Stephanie Dean was the front-line ops manager for Amazon for many years, over the time where they basically *fixed* their availability problems. She since moved on to Facebook, Demonware, and Twitter. She really knows her stuff and this blog is FULL of great details of how they ran (and still run) front-line ops teams in Amazon.

    (tags: ops incident-response outages event-management amazon stephanie-dean techops tos sev1)

  • RICON 2014: CRDTs

    Carlos Baquero presents several operation, state-based CRDTs for use in AP systems like Voldemort and Riak

    (tags: ap cap-theorem crdts ricon carlos-baquero data-structures distcomp)

  • Brownout: building more robust cloud applications

    Applications can saturate – i.e. become unable to serve users in a timely manner. Some users may experience high latencies, while others may not receive any service at all. The authors argue that it is better to downgrade the user experience and continue serving a larger number of clients with reasonable latency. “We define a cloud application as brownout compliant if it can gradually downgrade user experience to avoid saturation.” This is actually very reminiscent of circuit breakers, as described in Nygard’s ‘Release It!’ and popularized by Netflix. If you’re already designing with circuit breakers, you’ve probably got all the pieces you need to add brownout support to your application relatively easily. “Our work borrows from the concept of brownout in electrical grids. Brownouts are an intentional voltage drop often used to prevent blackouts through load reduction in case of emergency. In such a situation, incandescent light bulbs dim, hence originating the term.” “To lower the maintenance effort, brownouts should be automatically triggered. This enables cloud applications to rapidly and robustly avoid saturation due to unexpected environmental changes, lowering the burden on human operators.”
    This is really similar to the Circuit Breaker pattern — in fact it feels to me like a variation on that, driven by measured latencies of operations/requests. See also .

    (tags: circuit-breaker patterns brownout robustness reliability load latencies degradation)

  • Photographs of Sellafield nuclear plant prompt fears over radioactive risk

    “Slow-motion Chernobyl”, as Greenpeace are calling it. You thought legacy code was a problem? try legacy Magnox fuel rods.

    Previously unseen pictures of two storage ponds containing hundreds of highly radioactive fuel rods at the Sellafield nuclear plant show cracked concrete, seagulls bathing in the water and weeds growing around derelict machinery. But a spokesman for owners Sellafield Ltd said the 60-year-old ponds will not be cleaned up for decades, despite concern that they are in a dangerous state and could cause a large release of radioactive material if they are allowed to deteriorate further. “The concrete is in dreadful condition, degraded and fractured, and if the ponds drain, the Magnox fuel will ignite and that would lead to a massive release of radioactive material,” nuclear safety expert John Large told the Ecologist magazine. “I am very disturbed at the run-down condition of the structures and support services. In my opinion there is a significant risk that the system could fail.

    (tags: energy environment nuclear uk sellafield magnox seagulls time long-now)

  • The man who made a game to change the world

    An interview with Richard Bartle, the creator of MUD, back in 1978.

    Perceiving the different ways in which players approached the game led Bartle to consider whether MMO players could be classified according to type. “A group of admins was having an argument about what people wanted out of a MUD in about 1990,” he recalls. “This began a 200-long email chain over a period of six months. Eventually I went through everybody’s answers and categorised them. I discovered there were four types of MMO player. I published some short versions of them then, when the journal of MUD research came out I wrote it up as a paper.” The so-called Bartle test, which classifies MMO players as Achievers, Explorers, Socialisers or Killers (or a mixture thereof) according to their play-style remains in widespread use today. Bartle believes that you need a healthy mix of all dominant types in order to maintain a successful MMO ecosystem. “If you have a game full of Achievers (players for whom advancement through a game is the primary goal) the people who arrive at the bottom level won’t continue to play because everyone is better than them,” he explains. “This removes the bottom tier and, over time, all of the bottom tiers leave through irritation. But if you have Socialisers in the mix they don’t care about levelling up and all of that. So the lowest Achievers can look down on the Socialisers and the Socialisers don’t care. If you’re just making the game for Achievers it will corrode from the bottom. All MMOs have this insulating layer, even if the developers don’t understand why it’s there.”

    (tags: mmo mud gaming history internet richard-bartle)

  • Testing fork time on AWS/Xen infrastructure

    Redis uses forking to perform persistence flushes, which means that once every 30 minutes it performs like crap (and kills the 99th percentile latency). Given this, various Redis people have been benchmarking fork() times on various Xen platforms, since Xen has a crappy fork() implementation

    (tags: fork xen redis bugs performance latency p99)

  • A Teenager Gets Grilled By Her Dad About Why She’s Not That Into Coding

    Jay Rosen interviews his 17-year-old daughter. it’s pretty eye-opening. Got to start them early!

    (tags: culture tech coding girls women feminism teenagers school jay-rosen stem)

Posted in Uncategorized | Comments closed

Links for 2014-10-29

Posted in Uncategorized | Comments closed

Links for 2014-10-28

  • David Malone planning a commemoration of Dublin Mean Time next year

    Dublin had its own time zone, 25 minutes off what would become GMT, until 1916

    (tags: 1916 dublin rising time dublin-mean-time dmt gmt perfidious-albion dunsink)

  • Roshiak

    a Riak-based clone of Roshi, the CRDT server built on top of Redis. some day I’ll write up the CRDT we use on top of Voldemort in $work. Comments:

    (tags: riak roshi crdt redis storage time-series-data)

  • Vodafone UK, Verizon add mandatory device-tracking token on all web requests

    ‘Verizon Wireless is monitoring users’ mobile internet traffic, using a token slapped onto web requests, to facilitate targeted advertising even if a user has opted out. The unique identifier token header (UIDH) was launched two years ago, and has caused an uproar in tech circles after it was re-discovered Thursday by Electronic Frontier Foundation staffer Jacob Hoffman-Andrews. The Relevant Mobile Advertising program, under which the UIDH was used, allowed a restaurant to advertised to locals only or for retail websites to promote to previous visitors, according to Verizon Wireless.’

    (tags: uidh verizon vodafone privacy tracking http cookies advertising)

  • Cuckoo Filters

    ‘In many networking systems, Bloom filters are used for high-speed set membership tests. They permit a small fraction of false positive answers with very good space efficiency. However, they do not permit deletion of items from the set, and previous attempts to extend “standard” Bloom filters to support deletion all degrade either space or performance. We propose a new data structure called the cuckoo filter that can replace Bloom filters for approximate set member- ship tests. Cuckoo filters support adding and removing items dynamically while achieving even higher performance than Bloom filters. For applications that store many items and target moderately low false positive rates, cuckoo filters have lower space overhead than space-optimized Bloom filters. Our experimental results also show that cuckoo filters out-perform previous data structures that extend Bloom filters to support deletions substantially in both time and space.’

    (tags: algorithms cs coding cuckoo-filters bloom-filters sets data-structures)

  • Irish government in favour of ISDS court-evasion for multinationals

    This has _already_ been used to trump national law. As Simon McGarr noted at : ‘Philip Morris initiated a dispute under the Australia-Hong Kong Bilateral Investment Treaty to force #plainpacks repeal and compensation’. “Plain packs” anti-smoking is being bitterly fought at the moment here in Ireland. More from the US point of view: : ‘The Obama administration’s insistence on ISDS may please Wall Street, but it threatens to undermine some of the president’s landmark achievements in curbing pollution and fighting global warming, not to mention his commitment to a single standard of justice. It’s not worthy of the president, and he should join Europe in scrapping it.’

    (tags: isds national-law law ireland sovereignty multinationals philip-morris us-politics eu free-trade)

  • Jonathan Bergknoff: Building good docker images

    Good advice

    (tags: devops reference docker tips ops containers linux)

  • Game Day Exercises at Stripe: Learning from `kill -9`

    We’ve started running game day exercises at Stripe. During a recent game day, we tested failing over a Redis cluster by running kill -9 on its primary node, and ended up losing all data in the cluster. We were very surprised by this, but grateful to have found the problem in testing. This result and others from this exercise convinced us that game days like these are quite valuable, and we would highly recommend them for others.
    Excellent post. Game days are a great idea. Also: massive Redis clustering fail

    (tags: game-days redis testing stripe outages ops kill-9 failover)

  • The Laborers Who Keep Dick Pics and Beheadings Out of Your Facebook Feed | WIRED

    “Everybody hits the wall, generally between three and five months,” says a former YouTube content moderator I’ll call Rob. “You just think, ‘Holy shit, what am I spending my day doing? This is awful.’”

    (tags: facebook wired beheadings moderation nsfw google youtube social-media filtering porn abuse)

Posted in Uncategorized | Comments closed

Links for 2014-10-27

  • PSA: don’t run ‘strings’ on untrusted files (CVE-2014-8485)


    Perhaps simply by the virtue of being a part of that bundle, the strings utility tries to leverage the common libbfd infrastructure to detect supported executable formats and “optimize” the process by extracting text only from specific sections of the file. Unfortunately, the underlying library can be hardly described as safe: a quick pass with afl (and probably with any other competent fuzzer) quickly reveals a range of troubling and likely exploitable out-of-bounds crashes due to very limited range checking

    (tags: strings libbfd gnu security fuzzing buffer-overflows)

Posted in Uncategorized | Comments closed

Links for 2014-10-24

Posted in Uncategorized | Comments closed

Links for 2014-10-23

Posted in Uncategorized | Comments closed

Links for 2014-10-22

  • Jauter

    This Java library can route paths to targets and create paths from targets and params (reverse routing). This library is tiny, without additional dependencies, and is intended for use together with an HTTP server side library. If you want to use with Netty, see netty-router.

    (tags: java jauter scala request-routing http netty open-source)

  • “Viewstamped Replication Revisited”, Liskov and Cowling [pdf]

    classic replication paper, via aphyr: ‘This paper presents an updated version of Viewstamped Replication, a replication technique that handles failures in which nodes crash. It describes how client requests are handled, how the group reorganizes when a replica fails, and how a failed replica is able to rejoin the group. The paper also describes a number of important optimizations and presents a protocol for handling reconfigurations that can change both the group membership and the number of failures the group is able to handle.’

Posted in Uncategorized | Comments closed

Links for 2014-10-21

  • BioBrick

    Holy shit we are living in the future.

    BioBrick parts are DNA sequences which conform to a restriction-enzyme assembly standard.[1][2] These Lego-like building blocks are used to design and assemble synthetic biological circuits, which would then be incorporated into living cells such as Escherichia coli cells to construct new biological systems.[3] Examples of BioBrick parts include promoters, ribosomal binding sites (RBS), coding sequences and terminators.
    (via Soren)

    (tags: via:sorenrags biobricks fabrication organisms artificial-life biology e-coli genetic-engineering)

  • Is Docker ready for production? Feedbacks of a 2 weeks hands on

    I have to agree with this assessment — there are a lot of loose ends still for production use of Docker in a SOA stack environment:

    From my point of view, Docker is probably the best thing I’ve seen in ages to automate a build. It allows to pre build and reuse shared dependencies, ensuring they’re up to date and reducing your build time. It avoids you to either pollute your Jenkins environment or boot a costly and slow Virtualbox virtual machine using Vagrant. But I don’t feel like it’s production ready in a complex environment, because it adds too much complexity. And I’m not even sure that’s what it was designed for.

    (tags: docker complexity devops ops production deployment soa web-services provisioning networking logging)

Posted in Uncategorized | Comments closed

Links for 2014-10-20

  • Load testing Apache Kafka on AWS

    This is a very solid benchmarking post, examining Kafka in good detail. Nicely done. Bottom line:

    I basically spend 2/3 of my work time torture testing and operationalizing distributed systems in production. There’s some that I’m not so pleased with (posts pending in draft forever) and some that have attributes that I really love. Kafka is one of those systems that I pretty much enjoy every bit of, and the fact that it performs predictably well is only a symptom of the reason and not the reason itself: the authors really know what they’re doing. Nothing about this software is an accident. Performance, everything in this post, is only a fraction of what’s important to me and what matters when you run these systems for real. Kafka represents everything I think good distributed systems are about: that thorough and explicit design decisions win.

    (tags: testing aws kafka ec2 load-testing benchmarks performance)

Posted in Uncategorized | Comments closed

Links for 2014-10-17

Posted in Uncategorized | Comments closed

Links for 2014-10-16

  • Landlords not liable for tenants’ water bills

    What an utter fuckup. Business as usual for Irish Water:

    However the spokeswoman said application packs for rented dwellings would be addressed to the landlord, at the landlord’s residence, and it would be the landlord’s responsibility to ensure the tenant received the application pack. Bills are to be issued quarterly, but as Irish Water will have the tenant’s PPS number, the utility firm will be able to pursue the tenant for any arrears and even apply any arrears to new accounts, when the tenant moves to a new address. Last week landlords had expressed concern over potential arrears, the liability for them and the possibility of being used as collection agents by Irish Water.

    (tags: landlords ireland irish-water tenancy rental ppsn)

  • Irish Water responds to landlords’ questions

    ugh, what a mess….

    * Every rental unit in the State is to get a pack addressed personally to the occupant. If Irish Water does not have details of a tenant, the pack will be addressed to ‘The Occupier’ * Packs will only be issued to individual rental properties in so far as Irish Water is aware of them * Landlords can contact Irish Water to advise they have let a property * Application Packs are issued relative to the information on the Irish Water mailing list. If this is incorrect or out of date, landlords can contact Irish Water to have the information adjusted *Irish Water will contact known landlords after the initial customer application campaign, to advise of properties for which no application has been received * Irish Water said that when a household is occupied the tenant is liable and when vacant the owner is liable. Both should advise Irish Water of change of status to the property – the tenant to cease liability, the landlord to take it up. Either party may take a reading and provide it to Irish Water, alternatively Irish Water will bill on average consumption, based on the date of change.

    (tags: irish-water water ireland liability bills landlords tenancy rental)

Posted in Uncategorized | Comments closed

Links for 2014-10-15

Posted in Uncategorized | Comments closed

Links for 2014-10-14

  • Dublin’s Best-Kept Secret: Blas Cafe

    looks great, around the corner from Cineworld on King’s Inn St, D1

    (tags: dublin cafes food blas-cafe eating northside)

  • “Meta-Perceptual Helmets For The Dead Zoo”

    with Neil McKenzie, Nov 9-16 2014, in the National History Museum in Dublin: ‘These six helmets/viewing devices start off by exploring physical conditions of viewing: if we have two eyes, they why is our vision so limited? Why do we have so little perception of depth? Why don’t our two eyes offer us two different, complementary views of the world around us? Why can’t they extend from our body so we can see over or around things? Why don’t they allow us to look behind and in front at the same time, or sideways in both directions? Why can’t our two eyes simultaneously focus on two different tasks? Looking through Michael Land’s defining work Animal Eyes, we see that nature has indeed explored all of these possibilities: a Hammerhead Shark has hyper-stereo vision; a horse sees 350° around itself; a chameleon has separately rotatable eyes… The series of Meta-Perceptual Helmets do indeed explore these zoological typologies: proposing to humans the hyper-stereo vision of the hammerhead shark; or the wide peripheral vision of the horse; or the backward/forward vision of the chameleon… but they also take us into the unnatural world of mythology and literature: the Cheshire Cat Helmet is so called because of the strange lingering effect of dominating visual information such as a smile or the eyes; the Cyclops allows one large central eye to take in the world around while a second tiny hidden eye focuses on a close up task (why has the creature never evolved that can focus on denitting without constantly having to glance around?).’ (via Emma)

    (tags: perception helmets dublin ireland museums dead-zoo sharks eyes vision art)

  • Grade inflation figures from Irish universities

    The figures show that, between 2004 and 2013, an average of 71.7 per cent of students at TCD graduated with either a 1st or a 2.1. DCU and UCC had the next highest rate of such awards (64.3 per cent and 64.2 per cent respectively), followed by UCD (55.8 per cent), NUI Galway (54.7 per cent), Maynooth University (53.7 per cent) and University of Limerick (50.2 per cent).

    (tags: tcd grades grade-inflation dcu ucc ucd ireland studies academia third-level)

  • webrtcH4cKS: ~ coTURN: the open-source multi-tenant TURN/STUN server you were looking for

    Last year we interviewed Oleg Moskalenko and presented the rfc5766-turn-server project, which is a free open source and extremely popular implementation of TURN and STURN server. A few months later we even discovered Amazon is using this project to power its Mayday service. Since then, a number of features beyond the original RFC 5766 have been defined at the IETF and a new open-source project was born: the coTURN project.

    (tags: webrtc turn sturn rfc-5766 push nat stun firewalls voip servers internet)

  • Google Online Security Blog: This POODLE bites: exploiting the SSL 3.0 fallback

    Today we are publishing details of a vulnerability in the design of SSL version 3.0. This vulnerability allows the plaintext of secure connections to be calculated by a network attacker.

    (tags: ssl3 ssl tls security exploits google crypto)

Posted in Uncategorized | Comments closed


It’s been a while since I wrote a long-form blog post here, but this post on the Swrve Engineering blog is worth a read; it describes how we use SSD caching on our EC2 instances to greatly improve EBS throughput.

Posted in Uncategorized | Comments closed

Links for 2014-10-13

Posted in Uncategorized | Comments closed

Links for 2014-10-12

Posted in Uncategorized | Comments closed

Links for 2014-10-11

Posted in Uncategorized | Comments closed

Links for 2014-10-10

  • How Videogames Like Minecraft Actually Help Kids Learn to Read | WIRED

    I analyzed several chunks of The Ultimate Player’s Guide using the Flesch-Kincaid Reading Ease scale, and they scored from grade 8 to grade 11. Yet in my neighborhood they’re being devoured by kids in the early phases of elementary school. Games, it seems, can motivate kids to read—and to read way above their level. This is what Constance Steinkuehler, a games researcher at the University of Wisconsin-Madison, discovered. She asked middle and high school students who were struggling readers (one 11th-grade student read at a 6th-grade level) to choose a game topic they were interested in, and then she picked texts from game sites for them to read—some as difficult as first-year-college language. The kids devoured them with no help and nearly perfect accuracy. How could they do this? “Because they’re really, really motivated,” Steinkuehler tells me. It wasn’t just that the students knew the domain well; there were plenty of unfamiliar words. But they persisted more because they cared about the task. “It’s situated knowledge. They see a piece of language, a turn of phrase, and they figure it out.”
    When my kids are playing Minecraft, there’s a constant stream of “how do you spell X?” as they craft nametags for their pets. It’s great!

    (tags: minecraft gaming kids education spelling school reading literacy)

  • “Gold” 4-star review from the Irish Times

    Niall Heery belatedly follows up Small Engine Repair, his 2006 mumblecore critical hit, with a slightly less off-centre comedy that makes imaginative use of a smashing cast. The story skirts tragedy on its leisurely passage from mishap to misadventure, but Gold remains the sort of picture you want to hug indulgently to a welcoming bosom. It gives humanism a good name.
    Go Niall! it’s a great movie, go see it

    (tags: gold movies friends reviews irish-times indie)

  • JCDecaux Developer API

    web service API for Dublin Bikes data (and other similar bikesharing services run by JCD):

    Two kinds of data are delivered by the platform: Static data provides stable information like station position, number of bike stands, payment terminal availability, etc. Dynamic data provides station state, number of available bikes, number of free bike stands, etc. Static data can be downloaded manually in file format or accessed through the API. Dynamic data are refreshed every minute and can be accessed only through the API.
    Ruby API:

    (tags: jcdecaux bikesharing dublin dublin-bikes api web-services http json open-data)

  • Best games I played in 2013

    my coworker JK’s favourite games of 2013: Gone Home, Last Of Us, Proteus, Papers Please etc. I really want to play these, since they’re all totally my bag too.

    (tags: games gaming todo want 2013)

  • Why Amazon Has No Profits (And Why It Works)

    Amazon has perhaps 1% of the US retail market by value. Should it stop entering new categories and markets and instead take profit, and by extension leave those segments and markets for other companies? Or should it keep investing to sweep them into the platform? Jeff Bezos’s view is pretty clear: keep investing, because to take profit out of the business would be to waste the opportunity. He seems very happy to keep seizing new opportunities, creating new businesses, and using every last penny to do it.

    (tags: amazon business strategy capex spending stocks investing retail)

  • Spark Breaks Previous Large-Scale Sort Record – Databricks

    Massive improvement over plain old Hadoop. This blog post goes into really solid techie reasons why, including:

    First and foremost, in Spark 1.1 we introduced a new shuffle implementation called sort-based shuffle (SPARK-2045). The previous Spark shuffle implementation was hash-based that required maintaining P (the number of reduce partitions) concurrent buffers in memory. In sort-based shuffle, at any given point only a single buffer is required. This has led to substantial memory overhead reduction during shuffle and can support workloads with hundreds of thousands of tasks in a single stage (our PB sort used 250,000 tasks).
    Also, use of Timsort, an external shuffle service to offload from the JVM, Netty, and EC2 SR-IOV.

    (tags: spark hadoop map-reduce batch parallel sr-iov benchmarks performance netty shuffle algorithms sort-based-shuffle timsort)

  • UK psyops created N. Irish Satanic Panic during the Troubles – Boing Boing

    During the 1970s, when Northern Ireland was gripped by near-civil-war, British military intelligence staged the evidence of “black masses” in order to create a Satanism panic among the “superstitious” Irish to discredit the paramilitaries. The secret history of imaginary Irish Satanism is documented in Black Magic and Bogeymen: Fear, Rumour and Popular Belief in the North of Ireland 1972-74, a new book from Sheffield University’s Richard Jenkins, who interviewed Captain Colin Wallace, the former head of British Army “black operations” for Northern Ireland.

    (tags: northern-ireland 1970s the-troubles ireland uvf ira history black-magic satanism weird fear mi5)

  • Netflix release new code to production before completing tests

    Interesting — I hadn’t heard of this being an official practise anywhere before (although we actually did it ourselves this week)…

    If a build has made it [past the 'integration test' phase], it is ready to be deployed to one or more internal environments for user-acceptance testing. Users could be UI developers implementing a new feature using the API, UI Testers performing end-to-end testing or automated UI regression tests. As far as possible, we strive to not have user-acceptance tests be a gating factor for our deployments. We do this by wrapping functionality in Feature Flags so that it is turned off in Production while testing is happening in other environments. 

    (tags: devops deployment feature-flags release testing integration-tests uat qa production ops gating netflix)

Posted in Uncategorized | Comments closed

Links for 2014-10-09

Posted in Uncategorized | Comments closed

Links for 2014-10-08

  • Trouble at the Koolaid Point — Serious Pony

    This is a harrowing post from Kathy Sierra, full of valid observations:

    You’re probably more likely to win the lottery than to get any law enforcement agency in the United States to take action when you are harassed online, no matter how visciously and explicitly. Local agencies lack the resources, federal agencies won’t bother.
    That to the power of ten in Ireland, too, I’d suspect. Fuck this. Troll culture is way out of control….

    (tags: twitter harassment feminism weev abuse trolls 4chan kathy-sierra)

  • Tehuti

    An embryonic metrics library for Java/Scala from Felix GV at LinkedIn, extracted from Kafka’s metric implementation and in the new Voldemort release. It fixes the major known problems with the Meter/Timer implementations in Coda-Hale/Dropwizard/Yammer Metrics. ‘Regarding Tehuti: it has been extracted from Kafka’s metric implementation. The code was originally written by Jay Kreps, and then maintained improved by some Kafka and Voldemort devs, so it definitely is not the work of just one person. It is in my repo at the moment but I’d like to put it in a more generally available (git and maven) repo in the future. I just haven’t had the time yet… As for comparing with CodaHale/Yammer, there were a few concerns with it, but the main one was that we didn’t like the exponentially decaying histogram implementation. While that implementation is very appealing in terms of (low) memory usage, it has several misleading characteristics (a lack of incoming data points makes old measurements linger longer than they should, and there’s also a fairly high possiblity of losing interesting outlier data points). This makes the exp decaying implementation robust in high throughput fairly constant workloads, but unreliable in sparse or spiky workloads. The Tehuti implementation provides semantics that we find easier to reason with and with a small code footprint (which we consider a plus in terms of maintainability). Of course, it is still a fairly young project, so it could be improved further.’ More background at the kafka-dev thread:

    (tags: kafka metrics dropwizard java scala jvm timers ewma statistics measurement latency sampling tehuti voldemort linkedin jay-kreps)

  • “Quantiles on Streams” [paper, 2009]

    ‘Chiranjeeb Buragohain and Subhash Suri: “Quantiles on Streams” in Encyclopedia of Database Systems, Springer, pp 2235–2240, 2009. ISBN: 978-0-387-35544-3′, cited by Martin Kleppman in as a good, short literature survey re estimating percentiles with a small memory footprint.

    (tags: latency percentiles coding quantiles streams papers algorithms)

  • Belkin Router Apocalypse

    Many Belkin routers attempt to determine if they’re connected to the internet by pinging ‘’, in a classic amateur fail move. Good reason not to run Belkin firmware if that’s the level of code quality to expect

    (tags: belkin fail ping icmp funny internet dailywtf broken)

  • Seven deadly sins of talking about “types”

    Good essay

    (tags: programming types rants coding languages fp)

  • Shellshock

    An _extremely_ detailed resource about the bash bug

    (tags: bash hacking security shell exploits reference shellshock)

Posted in Uncategorized | Comments closed

Links for 2014-10-07

Posted in Uncategorized | Comments closed

Links for 2014-10-06

  • Reddit’s crappy ultimatum to remote workers and offices

    Reddit forces all remote workers (about half the workforce, in SLC and NYC) to move to SF, provoking a shitstorm:

    In a tweet confirming the move, Reddit’s CEO justified his treatment of non-San Francisco workers with a push for Optimal Teamwork to drive the New And $50M Improved Reddit forward. I shit you not. That was the actual term! (I added the New & Improved fan fiction here). So let’s leave aside the debate over whether working remotely is as efficient as being in the same office all the time. Let’s just focus on the size of the middle finger given to the people who work at Reddit outside the Bay Area, given the choice of forced, express relocation or a pink slip. How optimal do you think these employees will feel about leadership and the rest of the team going forward? Do you think they’ll just show up at the new, apparently-not-even-in-San-Francisco-proper office with a smile from ear to ear, ready to begin in earnest on Optimal Teamwork, left-behind former colleagues be damned?

    (tags: telecommuting reddit working remote-working ceos optimal-teamwork teamwork relocation)

  • Space Jacket

    ‘I designed this jacket as a tribute to the continuing legacy of American spaceflight. I wanted it to embody everything I loved about the space program, and to eventually serve as an actual flight jacket for present-day astronauts on missions to the ISS (International Space Station). There are other “replica” flight jackets made for space enthusiasts, but I decided to come up with something boldly different, yet also completely wearable and well-suited for space.’

    (tags: space clothing fashion geekery jackets)

  • How did Twitter become the hate speech wing of the free speech party?

    Kevin Marks has a pretty good point here:

    Your tweet could win the fame lottery, and everyone on the Internet who thinks you are wrong could tell you about it. Or one of the “verified” could call you out to be the tribute for your community and fight in their Hunger Games. Say something about feminism, or race, or sea lions and you’d find yourself inundated by the same trite responses from multitudes. Complain about it, and they turn nasty, abusing you, calling in their friends to join in. Your phone becomes useless under the weight of notifications; you can’t see your friends support amongst the flood. The limited tools available – blocking, muting, going private – do not match well with these floods. Twitter’s abuse reporting form takes far longer than a tweet, and is explicitly ignored if friends try to help.

    (tags: harassment twitter 4chan abuse feminism hate-speech gamergate sea-lions filtering social-media kevin-marks)

  • Mnesia and CAP

    A common “trick” is to claim: ‘We assume network partitions can’t happen. Therefore, our system is CA according to the CAP theorem.’ This is a nice little twist. By asserting network partitions cannot happen, you just made your system into one which is not distributed. Hence the CAP theorem doesn’t even apply to your case and anything can happen. Your system may be linearizable. Your system might have good availability. But the CAP theorem doesn’t apply. [...] In fact, any well-behaved system will be “CA” as long as there are no partitions. This makes the statement of a system being “CA” very weak, because it doesn’t put honesty first. I tries to avoid the hard question, which is how the system operates under failure. By assuming no network partitions, you assume perfect information knowledge in a distributed system. This isn’t the physical reality.

    (tags: cap erlang mnesia databases storage distcomp reliability ca postgres partitions)

  • Integrating Kafka and Spark Streaming: Code Examples and State of the Game

    Spark Streaming has been getting some attention lately as a real-time data processing tool, often mentioned alongside Apache Storm. [...] I added an example Spark Streaming application to kafka-storm-starter that demonstrates how to read from Kafka and write to Kafka, using Avro as the data format and Twitter Bijection for handling the data serialization. In this post I will explain this Spark Streaming example in further detail and also shed some light on the current state of Kafka integration in Spark Streaming. All this with the disclaimer that this happens to be my first experiment with Spark Streaming.

    (tags: spark kafka realtime architecture queues avro bijection batch-processing)

  • Mandos

    ‘a system for allowing servers with encrypted root file systems to reboot unattended and/or remotely.’ (via Tony Finch)

    (tags: via:fanf mandos encryption security server ops sysadmin linux)

  • Zonify

    ‘a set of command line tools for managing Route53 DNS for an AWS infrastructure. It intelligently uses tags and other metadata to automatically create the associated DNS records.’

    (tags: zonify aws dns ec2 route53 ops)

  • Mike Perham on Twitter: “Sweet, monit just sent a DMCA takedown notice to @github to remove Inspeqtor.”

    ‘The work, Inspeqtor which is hosted at GitHub, is far from a “clean-room” implementation. This is basically a rewrite of Monit in Go, even using the same configuration language that is used in Monit, verbatim. a. [private] himself admits that Inspeqtor is “heavily influenced“ by Monit b. This tweet by [private] demonstrate intent. “OSS nerds: redesign and build monit in Go. Sell it commercially. Make $$$$. I will be your first customer.”’ IANAL, but using the same config language does not demonstrate copyright infringement…

    (tags: copyright dmca tildeslash monit inspeqtor github ops oss agpl)

  • YOU AND YOUR DAMNED GAMES, JON STONE — Why bother with #gamergate?

    So what is #gamergate? #gamergate is a mob with torches aloft, hunting for any combustible dwelling and calling it a monster’s lair. #gamergate is a rage train, and everyone with an axe to grind wants a ride. Its fuel is a sour mash of entitlement, insecurity, arrogance and alienation. #gamergate is a vindication quest for political intolerance. #gamergate is revenge for every imagined slight. #gamergate is Viz’s Meddlesome Ratbag.

    (tags: gamergate culture gaming 4chan mobs feminism)

Posted in Uncategorized | Comments closed

Links for 2014-10-05

  • ‘In 1976 I discovered Ebola, now I fear an unimaginable tragedy’ | World news | The Observer

    An interview with the scientist who was part of the team which discovered the Ebola virus in 1976:

    Other samples from the nun, who had since died, arrived from Kinshasa. When we were just about able to begin examining the virus under an electron microscope, the World Health Organisation instructed us to send all of our samples to a high-security lab in England. But my boss at the time wanted to bring our work to conclusion no matter what. He grabbed a vial containing virus material to examine it, but his hand was shaking and he dropped it on a colleague’s foot. The vial shattered. My only thought was: “Oh, shit!” We immediately disinfected everything, and luckily our colleague was wearing thick leather shoes. Nothing happened to any of us.

    (tags: ebola epidemiology health africa labs history medicine)

Posted in Uncategorized | Comments closed

Links for 2014-10-04

  • Ebola vaccine delayed by IP spat

    This is the downside of publicly-funded labs selling patent-licensing rights to private companies:

    Given the urgency, it’s inexplicable that one of the candidate vaccines, developed at the Public Health Agency of Canada (PHAC) in Winnipeg, has yet to go in the first volunteer’s arm, says virologist Heinz Feldmann, who helped develop the vaccine while at PHAC. “It’s a farce; these doses are lying around there while people are dying in Africa,” says Feldmann, who now works at the Rocky Mountain Laboratories of the U.S. National Institute of Allergy and Infectious Diseases (NIAID) in Hamilton, Montana. At the center of the controversy is NewLink Genetics, a small company in Ames, Iowa, that bought a license to the vaccine’s commercialization from the Canadian government in 2010, and is now suddenly caught up in what WHO calls “the most severe acute public health emergency seen in modern times.” Becker and others say the company has been dragging its feet the past 2 months because it is worried about losing control over the development of the vaccine.

    (tags: ip patents drugs ebola canada phac newlink-genetics health epidemics vaccines)

  • sferik/t

    “A command-line power tool for Twitter.” It really is — much better timeline searchability than the “real” Twitter UI, for example

    (tags: twitter ruby github cli tools unix search)

  • Ebola: While Big Pharma Slept

    We’ve had almost 40 years to develop, test and stockpile an Ebola vaccine. That has not happened because big pharma has been entirely focused on shareholder value and profits over safety and survival from a deadly virus. For the better part of Ebola’s 38 years ? big pharma has been asleep. The question ahead is what virus or superbug will wake them up?

    (tags: pharma ebola ip patents health drugs africa research)

Posted in Uncategorized | Comments closed

Links for 2014-10-02

Posted in Uncategorized | Comments closed

Links for 2014-10-01

Posted in Uncategorized | Comments closed

Links for 2014-09-30

Posted in Uncategorized | Comments closed

Links for 2014-09-29

  • Prototype

    Prototype is a brand new festival of play and interaction. This is your chance to experience the world from a new perspective with removable camera eyes, to jostle and joust to a Bach soundtrack whilst trying to disarm an opponent, to throw shapes as you figure out who got an invite to the silent disco, to duel with foam pool noodles, and play chase in the dark with flashlights. A unique festival that incites new types of social interaction, involving technology and the city, Prototype is a series of performances, workshops, talks, and games that spill across the city, alongside an adult playground in the heart of Temple Bar.
    Project Arts Centre, 17-18 October. looks nifty

    (tags: prototype festivals dublin technology make vr gaming)

  • Confessions of a former internet troll – Vox

    I want to tell you about when violent campaigns against harmless bloggers weren’t any halfway decent troll’s idea of a good time — even the then-malicious would’ve found it too easy to be fun. When the punches went up, not down. Before the best players quit or went criminal or were changed by too long a time being angry. When there was cruelty, yes, and palpable strains of sexism and racism and every kind of phobia, sure, but when these things had the character of adolescents pushing the boundaries of cheap shock, disagreeable like that but not criminal. Not because that time was defensible — it wasn’t, not really — but because it was calmer and the rage wasn’t there yet. Because trolling still meant getting a rise for a laugh, not making helpless people fear for their lives because they’re threatening some Redditor’s self-proclaimed monopoly on reason. I want to tell you about it because I want to make sense of how it is now and why it changed.

    (tags: vox trolls blogging gamergate 4chan weev history teenagers)

Posted in Uncategorized | Comments closed

Links for 2014-09-27

  • La Maison des Amis

    Paul Hickey’s gite near Toulouse, available for rent! ‘a beautifully converted barn on 5 acres, wonderfully located in the French countryside. 4 Bedrooms, sleeps 2-10, Large Pool, Tennis Court, Large Trampoline, Broadband Internet, 30 Mins Toulouse/Albi, 65 Mins Carcassonne, 90 Mins Rodez’

    (tags: ex-iona gites france holidays vacation-rentals vacation hotels)

  • waxpancake on Ello

    The Ello founders are positioning it as an alternative to other social networks — they won’t sell your data or show you ads. “You are not the product.” If they were independently-funded and run as some sort of co-op, bootstrapped until profitable, maybe that’s plausible. Hard, but possible. But VCs don’t give money out of goodwill, and taking VC funding — even seed funding — creates outside pressures that shape the inevitable direction of a company.

    (tags: advertising money vc ello waxy funding series-a)

  • Inviso: Visualizing Hadoop Performance

    With the increasing size and complexity of Hadoop deployments, being able to locate and understand performance is key to running an efficient platform.  Inviso provides a convenient view of the inner workings of jobs and platform.  By simply overlaying a new view on existing infrastructure, Inviso can operate inside any Hadoop environment with a small footprint and provide easy access and insight.  
    This sounds pretty useful.

    (tags: inviso netflix hadoop emr performance ops tools)

  • The End of Linux

    ‘Linux is becoming the thing that we adopted Linux to get away from.’ Great post on the horrible complexity of systemd. It reminds me of nothing more than mid-90s AIX, which I had the displeasure of opsing for a while — the Linux distros have taken a very wrong turn here.

    (tags: linux unix complexity compatibility ops rant systemd bloat aix)

Posted in Uncategorized | Comments closed

Links for 2014-09-24

Posted in Uncategorized | Comments closed

Links for 2014-09-23

  • Avoiding Chef-Suck with Auto Scaling Groups – forty9ten

    Some common problems which arise using Chef with ASGs in EC2, and how these guys avoided it — they stopped using Chef for service provisioning, and instead baked AMIs when a new version was released. ASGs using pre-baked AMIs definitely works well so this makes good sense IMO.

    (tags: infrastructure chef ops asg auto-scaling ec2 provisioning deployment)

  • Introducing

    Mark “ONEList” Fletcher’s back, and he’s reinventing the email group! awesome.

    email groups (the modern version of mailing lists) have stagnated over the past decade. Yahoo Groups and Google Groups both exude the dank air of benign neglect. Google Groups hasn’t been updated in years, and some of Yahoo’s recent changes have actually made Yahoo Groups worse! And yet, millions of people put up with this uncertainty and neglect, because email groups are still one of the best ways to communicate with groups of people. And I have a plan to make them even better. So today I’m launching in beta, to bring email groups into the 21st Century. At launch, we have many features that those other services don’t have, including: Integration with other services, including: Github, Google Hangouts, Dropbox, Instagram, Facebook Pages, and the ability to import Feeds into your groups. Businesses and organizations can have their own private groups on their own subdomain. Better archive organization, using hashtags. Many more email delivery options. The ability to mute threads or hashtags. Fully searchable archives, including searching within attachments. One other feature that has that Yahoo and Google don’t, is a business model that’s not based on showing ads to you. Public groups are completely free on Private groups and organizations are very reasonably priced.

    (tags: email groups communication discussion mailing-lists yahoo google google-groups yahoo-groups)

Posted in Uncategorized | Comments closed

Links for 2014-09-22

  • The Open Source Software Engagement Award

    SFU announces award for students who demonstrate excellence in contributing to an Open Source project

    (tags: sfu awards students open-source oss universities funding)

  • DublinDashboard

    ‘provides citizens, public sector workers and companies with real-time information, time-series indicator data, and interactive maps about all aspects of the city. It enables users to gain detailed, up to date intelligence about the city that aids everyday decision making and fosters evidence-informed analysis.’

    (tags: dublin dashboards maps geodata time-series open-data ireland)

  • mcrouter: A memcached protocol router for scaling memcached deployments

    New from Facebook engineering:

    Last year, at the Data@Scale event and at the USENIX Networked Systems Design and Implementation conference , we spoke about turning caches into distributed systems using software we developed called mcrouter (pronounced “mick-router”). Mcrouter is a memcached protocol router that is used at Facebook to handle all traffic to, from, and between thousands of cache servers across dozens of clusters distributed in our data centers around the world. It is proven at massive scale — at peak, mcrouter handles close to 5 billion requests per second. Mcrouter was also proven to work as a standalone binary in an Amazon Web Services setup when Instagram used it last year before fully transitioning to Facebook’s infrastructure. Today, we are excited to announce that we are releasing mcrouter’s code under an open-source BSD license. We believe it will help many sites scale more easily by leveraging Facebook’s knowledge about large-scale systems in an easy-to-understand and easy-to-deploy package.
    This is pretty crazy — basically turns a memcached cluster into a much more usable clustered-storage system, with features like shadowing production traffic, cold cache warmup, online reconfiguration, automatic failover, prefix-based routing, replicated pools, etc. Lots of good features.

    (tags: facebook scaling cache proxy memcache open-source clustering distcomp storage)


    In particular:

    Where you have obtained contact details in the context of the sale of a product or service, you may only use these details for direct marketing by electronic mail if the following conditions are met: the product or service you are marketing is of a kind similar to that which you sold to the customer at the time you obtained their contact details At the time you collected the details, you gave the customer the opportunity to object, in an easy manner and without charge, to their use for marketing purposes Each time you send a marketing message, you give the customer the right to object to receipt of further messages The sale of the product or service occurred not more than twelve months prior to the sending of the electronic marketing communication or, where applicable, the contact details were used for the sending of an electronic marketing communication in that twelve month period.

    (tags: email spam regulations ireland law dpc marketing direct-marketing)

Posted in Uncategorized | Comments closed

Links for 2014-09-21

Posted in Uncategorized | Comments closed

Links for 2014-09-19

Posted in Uncategorized | Comments closed

Links for 2014-09-18

  • 75% of domestic violence victims in US shelters were spied on by their abusers using spyware

    via Mikko

  • Alex Payne — Thoughts On Five Years of Emerging Languages

    One could read the success of Go as an indictment of contemporary PLT, but I prefer to see it as a reminder of just how much language tooling matters. Perhaps even more critical, Go’s lean syntax, selective semantics, and cautiously-chosen feature set demonstrate the importance of a strong editorial voice in a language’s design and evolution. Having co-authored a book on Scala, it’s been painful to see systems programmers in my community express frustration with the ambitious hybrid language. I’ve watched them abandon ship and swim back to the familiar shores of Java, or alternately into the uncharted waters of Clojure, Go, and Rust. A pity, but not entirely surprising if we’re being honest with ourselves. Unlike Go, Scala has struggled with tooling from its inception. More than that, Scala has had a growing editorial problem. Every shop I know that’s been successful with Scala has limited itself to some subset of the language. Meanwhile, in pursuit of enterprise developers, its surface area has expanded in seemingly every direction. The folks behind Scala have, thankfully, taken notice: upcoming releases are promised to focus on simplicity, clarity, and better tooling.

    (tags: scala go coding languages)

Posted in Uncategorized | Comments closed

Links for 2014-09-17

  • Texas Judge References ‘The Big Lebowski’

    “The First Amendment of the U.S. Constitution is similarly suspicious of prior restraints,” wrote Justice Lehrmann in the decision highlighting a cornerstone that has “been reaffirmed time and again by the Supreme Court, this Court, Texas courts of appeals, legal treatises, and even popular culture.” That last reference to popular culture contained an interesting footnote citing none other than Walter Sobchak, a character in ['The Big Lebowski'].

    (tags: lebowski movies coen-brothers prior-restraint law supreme-court walter-sobchak funny)

  • on using JSON as a config file format

    Ben Hughes on twitter: “JSON is fine for config files, if you don’t want to comment your config file. Which is a way of saying, it isn’t fine for config files.”

    (tags: ben-hughes funny json file-formats config-files configuration software coding)

  • Understanding weak isolation is a serious problem

    Peter Bailis complaining about the horrors of modern transactional databases and their unserializability, which noone seems to be paying attention to: ‘As you’re probably aware, there’s an ongoing and often lively debate between transactional adherents and more recent “NoSQL” upstarts about related issues of usability, data corruption, and performance. But, in contrast, many of these transactional inherents and the research community as a whole have effectively ignored weak isolation — even in a single server setting and despite the fact that literally millions of businesses today depend on weak isolation and that many of these isolation levels have been around for almost three decades.’ ‘Despite the ubiquity of weak isolation, I haven’t found a database architect, researcher, or user who’s been able to offer an explanation of when, and, probably more importantly, why isolation models such as Read Committed are sufficient for correct execution. It’s reasonably well known that these weak isolation models represent “ACID in practice,” but I don’t think we have any real understanding of how so many applications are seemingly (!?) okay running under them. (If you haven’t seen these models before, they’re a little weird. For example, Read Committed isolation generally prevents users from reading uncommitted or non-final writes but allows a number of bad things to happen, like lost updates during concurrent read-modify-write operations. Why is this apparently okay for many applications?)’

    (tags: acid consistency databases peter-bailis transactional corruption serializability isolation reliability)

  • “Left-Right: A Concurrency Control Technique with Wait-Free Population Oblivious Reads” [pdf]

    ‘In this paper, we describe a generic concurrency control technique with Blocking write operations and Wait-Free Population Oblivious read operations, which we named the Left-Right technique. It is of particular interest for real-time applications with dedicated Reader threads, due to its wait-free property that gives strong latency guarantees and, in addition, there is no need for automatic Garbage Collection. The Left-Right pattern can be applied to any data structure, allowing concurrent access to it similarly to a Reader-Writer lock, but in a non-blocking manner for reads. We present several variations of the Left-Right technique, with different versioning mechanisms and state machines. In addition, we constructed an optimistic approach that can reduce synchronization for reads.’ See also for java implementation code.

    (tags: left-right concurrency multithreading wait-free blocking realtime gc latency reader-writer locking synchronization java)

  • Russell91/sshrc

    ‘bring your .bashrc, .vimrc, etc. with you when you ssh’. A really nice implementation of this idea (much nicer than my own version!)

    (tags: hacks productivity ssh remote shell sh bash via:johnke home-directory unix)

  • Troubleshooting Production JVMs with jcmd

    remotely trigger GCs, finalization, heap dumps etc. Handy

    (tags: jvm jcmd debugging ops java gc heap troubleshooting)

Posted in Uncategorized | Comments closed

Links for 2014-09-16

Posted in Uncategorized | Comments closed

Links for 2014-09-15

  • The State of ZFS on Linux

    Linux users familiar with other filesystems or ZFS users from other platforms will often ask whether ZFS on Linux (ZoL) is “stable”. The short answer is yes, depending on your definition of stable. The term stable itself is somewhat ambiguous.
    Oh dear. that’s not a good start. Good reference page, though

    (tags: zfs linux filesystems ops solaris)

  • Screen time: Steve Jobs was a low tech parent

    “This is rule No. 1: There are no screens in the bedroom. Period. Ever.”

    (tags: screen-time kids children tv mobile technology life rules parenting)

  • CausalImpact: A new open-source package for estimating causal effects in time series

    How can we measure the number of additional clicks or sales that an AdWords campaign generated? How can we estimate the impact of a new feature on app downloads? How do we compare the effectiveness of publicity across countries? In principle, all of these questions can be answered through causal inference. In practice, estimating a causal effect accurately is hard, especially when a randomised experiment is not available. One approach we’ve been developing at Google is based on Bayesian structural time-series models. We use these models to construct a synthetic control — what would have happened to our outcome metric in the absence of the intervention. This approach makes it possible to estimate the causal effect that can be attributed to the intervention, as well as its evolution over time. We’ve been testing and applying structural time-series models for some time at Google. For example, we’ve used them to better understand the effectiveness of advertising campaigns and work out their return on investment. We’ve also applied the models to settings where a randomised experiment was available, to check how similar our effect estimates would have been without an experimental control. Today, we’re excited to announce the release of CausalImpact, an open-source R package that makes causal analyses simple and fast. With its release, all of our advertisers and users will be able to use the same powerful methods for estimating causal effects that we’ve been using ourselves. Our main motivation behind creating the package has been to find a better way of measuring the impact of ad campaigns on outcomes. However, the CausalImpact package could be used for many other applications involving causal inference. Examples include problems found in economics, epidemiology, or the political and social sciences.

    (tags: causal-inference r google time-series models bayes adwords advertising statistics estimation metrics)

  • Top 10 Historic Sites in Ireland and Northern Ireland — National Geographic

    Shamefully, I haven’t visited most of these!

    (tags: history neolithic ireland northern-ireland national-geographic tourism places)

  • Software patents are crumbling, thanks to the Supreme Court

    Now a series of decisions from lower courts is starting to bring the ruling’s practical consequences into focus. And the results have been ugly for fans of software patents. By my count there have been 11 court rulings on the patentability of software since the Supreme Court’s decision — including six that were decided this month.  Every single one of them has led to the patent being invalidated. This doesn’t necessarily mean that all software patents are in danger — these are mostly patents that are particularly vulnerable to challenge under the new Alice precedent. But it does mean that the pendulum of patent law is now clearly swinging in an anti-patent direction. Every time a patent gets invalidated, it strengthens the bargaining position of every defendant facing a lawsuit from a patent troll.

    (tags: patents law alice swpats software supreme-court patent-trolls)

  • Riding with the Stars: Passenger Privacy in the NYC Taxicab Dataset

    A practical demo of “differential privacy” — allowing public data dumps to happen without leaking privacy, using Laplace noise addition

    (tags: differential-privacy privacy leaks public-data open-data data nyc taxis laplace noise randomness)

Posted in Uncategorized | Comments closed

Links for 2014-09-14

  • Platform Game

    I’m ambivalent about Microsoft acquiring Mojang. Will they Embrace and Extend Minecraft as they’ve done with other categories? Let’s hope not. On the other hand, some adult supervision and a Plugin API would be welcome. Mojang have the financial resources but lack the will and focus needed to publish and support a Plugin API. Perhaps Mojang themselves don’t realise just how important their little game has become.

    (tags: minecraft platforms games plugins mojang microsoft)

Posted in Uncategorized | Comments closed

Links for 2014-09-11

Posted in Uncategorized | Comments closed