-
Stephanie Dean on Amazon’s approach to CMs. This is solid gold advice for any company planning to institute a sensible technical change management process
(tags: ops tech process changes change-management bureaucracy amazon stephanie-dean infrastructure)
Stephanie Dean on event management and incident response
I asked around my ex-Amazon mates on twitter about good docs on incident response practices outside the “iron curtain”, and they pointed me at this blog (which I didn’t realise existed). Stephanie Dean was the front-line ops manager for Amazon for many years, over the time where they basically *fixed* their availability problems. She since moved on to Facebook, Demonware, and Twitter. She really knows her stuff and this blog is FULL of great details of how they ran (and still run) front-line ops teams in Amazon.
(tags: ops incident-response outages event-management amazon stephanie-dean techops tos sev1)
-
Carlos Baquero presents several operation, state-based CRDTs for use in AP systems like Voldemort and Riak
(tags: ap cap-theorem crdts ricon carlos-baquero data-structures distcomp)
Brownout: building more robust cloud applications
Applications can saturate – i.e. become unable to serve users in a timely manner. Some users may experience high latencies, while others may not receive any service at all. The authors argue that it is better to downgrade the user experience and continue serving a larger number of clients with reasonable latency. “We define a cloud application as brownout compliant if it can gradually downgrade user experience to avoid saturation.” This is actually very reminiscent of circuit breakers, as described in Nygard’s ‘Release It!’ and popularized by Netflix. If you’re already designing with circuit breakers, you’ve probably got all the pieces you need to add brownout support to your application relatively easily. “Our work borrows from the concept of brownout in electrical grids. Brownouts are an intentional voltage drop often used to prevent blackouts through load reduction in case of emergency. In such a situation, incandescent light bulbs dim, hence originating the term.” “To lower the maintenance effort, brownouts should be automatically triggered. This enables cloud applications to rapidly and robustly avoid saturation due to unexpected environmental changes, lowering the burden on human operators.”
This is really similar to the Circuit Breaker pattern — in fact it feels to me like a variation on that, driven by measured latencies of operations/requests. See also http://blog.acolyer.org/2014/10/27/improving-cloud-service-resilience-using-brownout-aware-load-balancing/ .(tags: circuit-breaker patterns brownout robustness reliability load latencies degradation)
Photographs of Sellafield nuclear plant prompt fears over radioactive risk
“Slow-motion Chernobyl”, as Greenpeace are calling it. You thought legacy code was a problem? try legacy Magnox fuel rods.
Previously unseen pictures of two storage ponds containing hundreds of highly radioactive fuel rods at the Sellafield nuclear plant show cracked concrete, seagulls bathing in the water and weeds growing around derelict machinery. But a spokesman for owners Sellafield Ltd said the 60-year-old ponds will not be cleaned up for decades, despite concern that they are in a dangerous state and could cause a large release of radioactive material if they are allowed to deteriorate further. “The concrete is in dreadful condition, degraded and fractured, and if the ponds drain, the Magnox fuel will ignite and that would lead to a massive release of radioactive material,” nuclear safety expert John Large told the Ecologist magazine. “I am very disturbed at the run-down condition of the structures and support services. In my opinion there is a significant risk that the system could fail.
(tags: energy environment nuclear uk sellafield magnox seagulls time long-now)
The man who made a game to change the world
An interview with Richard Bartle, the creator of MUD, back in 1978.
Perceiving the different ways in which players approached the game led Bartle to consider whether MMO players could be classified according to type. “A group of admins was having an argument about what people wanted out of a MUD in about 1990,” he recalls. “This began a 200-long email chain over a period of six months. Eventually I went through everybody’s answers and categorised them. I discovered there were four types of MMO player. I published some short versions of them then, when the journal of MUD research came out I wrote it up as a paper.” The so-called Bartle test, which classifies MMO players as Achievers, Explorers, Socialisers or Killers (or a mixture thereof) according to their play-style remains in widespread use today. Bartle believes that you need a healthy mix of all dominant types in order to maintain a successful MMO ecosystem. “If you have a game full of Achievers (players for whom advancement through a game is the primary goal) the people who arrive at the bottom level won’t continue to play because everyone is better than them,” he explains. “This removes the bottom tier and, over time, all of the bottom tiers leave through irritation. But if you have Socialisers in the mix they don’t care about levelling up and all of that. So the lowest Achievers can look down on the Socialisers and the Socialisers don’t care. If you’re just making the game for Achievers it will corrode from the bottom. All MMOs have this insulating layer, even if the developers don’t understand why it’s there.”
Testing fork time on AWS/Xen infrastructure
Redis uses forking to perform persistence flushes, which means that once every 30 minutes it performs like crap (and kills the 99th percentile latency). Given this, various Redis people have been benchmarking fork() times on various Xen platforms, since Xen has a crappy fork() implementation
A Teenager Gets Grilled By Her Dad About Why She’s Not That Into Coding
Jay Rosen interviews his 17-year-old daughter. it’s pretty eye-opening. Got to start them early!
(tags: culture tech coding girls women feminism teenagers school jay-rosen stem)