excellent case study of production-scale usage of Docker
UNIX system service [jmason: ie a sidecar] that collects events and reliably delivers them to kafka, relieving other services on the same system from having to do so. Journals events through bolt-db so that in the event of an kafka outage, events can still be accepted, and will be delivered when kafka becomes available.
Excellent advice from Tomasz Nurkiewicz’ blog for anyone using java.util.concurrent.ExecutorService regularly. The whole blog is full of great posts btw
Good advice for Hadoop optimization
Nice work by Andrew Spyker — this should be an official feature of the re:Invent website, really
Excellent data on current EBS performance characteristics
A desktop app for finding and inserting GIFs into any conversationOh yes.
The researchers have no doubt that Regin is a nation-state tool and are calling it the most sophisticated espionage machine uncovered to date—more complex even than the massive Flame platform, uncovered by Kaspersky and Symantec in 2012 and crafted by the same team who created Stuxnet. “In the world of malware threats, only a few rare examples can truly be considered groundbreaking and almost peerless,” writes Symantec in its report about Regin. Though no one is willing to speculate on the record about Regin’s source, news reports about the Belgacom and Quisquater hacks pointed a finger at GCHQ and the NSA. Kaspersky confirms that Quisqater was infected with Regin, and other researchers familiar with the Belgacom attack have told WIRED that the description of Regin fits the malware that targeted the telecom, though the malicious files used in that attack were given a different name, based on something investigators found inside the platform’s main file.
Charted is a tool for automatically visualizing data, created by the Product Science team at Medium. Give it the link to a data file and Charted returns a beautiful, shareable chart of the data.Nice, but it’s no graphite — pretty basic.
The Uber controversy is not just—or even mainly—a technology story, it’s fundamentally a deregulation story; the story of a uniquely American fundamentalist free-market worldview being sold to us in the name of “car-sharing” and innovation.
‘Design your own beer labels’
This is classic. I love the “Rouge”:
We also wanted the Rouge to actually look like a stealth-oriented make-up artist, but our 3D artist thought the goat looked ridiculous with a pink wig and a Gucci bag, so we remade the Rouge to actually look like a Rogue.
Finally after all traditional means of infection were covered; IT started looking into other possibilities. They finally asked the Executive, “Have there been any changes in your life recently”? The executive answer “Well yes, I quit smoking two weeks ago and switched to e-cigarettes”. And that was the answer they were looking for, the made in china e-cigarette had malware hard coded into the charger and when plugged into a computer’s USB port the malware phoned home and infected the system. Moral of the story is have you ever question the legitimacy of the $5 dollar EBay made in China USB item that you just plugged into your computer? Because you should, you damn well should.(Via Elliot)
As part of a performance update to Azure Storage, an issue was discovered that resulted in reduced capacity across services utilizing Azure Storage, including Virtual Machines, Visual Studio Online, Websites, Search and other Microsoft services. Prior to applying the performance update, it had been tested over several weeks in a subset of our customer-facing storage service for Azure Tables. We typically call this “flighting,” as we work to identify issues before we broadly deploy any updates. The flighting test demonstrated a notable performance improvement and we proceeded to deploy the update across the storage service. During the rollout we discovered an issue that resulted in storage blob front ends going into an infinite loop, which had gone undetected during flighting. The net result was an inability for the front ends to take on further traffic, which in turn caused other services built on top to experience issues.I’m really surprised MS deployment procedures allow a change to be rolled out globally across multiple regions on a single day. I suspect they soon won’t.
This is a really solid talk — not surprising, alv@ is one of the speakers!
“AWS and Amazon.com operate some of the world’s largest distributed systems infrastructure and applications. In our past 18 years of operating this infrastructure, we have come to realize that building such large distributed systems to meet the durability, reliability, scalability, and performance needs of AWS requires us to build our services using a few common distributed systems primitives. Examples of these primitives include a reliable method to build consensus in a distributed system, reliable and scalable key-value store, infrastructure for a transactional logging system, scalable database query layers using both NoSQL and SQL APIs, and a system for scalable and elastic compute infrastructure. In this session, we discuss some of the solutions that we employ in building these primitives and our lessons in operating these systems. We also cover the history of some of these primitives — DHTs, transactional logging, materialized views and various other deep distributed systems concepts; how their design evolved over time; and how we continue to scale them to AWS. “Slides: http://www.slideshare.net/AmazonWebServices/spot302-under-the-covers-of-aws-core-distributed-systems-primitives-that-power-our-platform-aws-reinvent-2014
“SCE to off?” someone said. The switch was so obscure that neither of his bosses knew what he was talking about. “What the hell’s that,” blurted out Gerald Carr, who was in charge of communicating with the capsule. The rookie flight director, Gerry Griffin, didn’t know either. Sixty seconds had passed since the initial lightning strike. No one else knew what to do. The call to abort was fast approaching. Finally, Carr reluctantly gave the order in a voice far cooler than the moment. “Apollo 12, Houston, try SCE to Auxiliary, over.”
Twitter’s new massive-scale twitter search backend. Sharding galore
‘I went over to Japan right around the time Takeshi was deciding which emoji were going to make it into the first cut of Gmail emoji. The [PILE_OF_POO emoji] was absolutely one of the necessary emoji that Takeshi said we have to have. There was actually conflict because there were people back at headquarters who had no idea what emoji were, and thought that having an animated [turd] in their Gmail was offensive.’ ‘[The poop emoji] got very popular when a comic called “Dr. Slump” was broadcast in Japan back to the ‘90s. Such poop was not an object to be disliked, but it had a funny meaning. This was a very popular comedy animation where a girl played a trick on other people using the poop. The poop was this funny object to play with. It was never serious.’ ‘In Japanese that’s called “unchi.” It’s a child word with a benign meaning. ‘
We plan to send an unmanned robotic landing module to the South Pole of the Moon – an area unexplored by previous missions. We’re going to use pioneering technology to drill down to a depth of at least 20m – 10 times deeper than has ever been drilled before – and potentially as deep as 100m. By doing this, we will access lunar rock dating back up to 4.5 billion years to discover the geological composition of the Moon, the ancient relationship it shares with our planet and the effects of asteroid bombardment. Ultimately, the project will improve scientific understanding of the early solar system, the formation of our planet and the Moon, and the conditions that initiated life on Earth.Kickstarter-funded — UKP 600k goal. Just in time for xmas!
Unlike the (excellent) Typescript, it’ll infer types:
An extremely good explanation from Marc Brooker that exactly-once delivery in a distributed system is very hard.
And so on. There’s always a place to slot in one more turtle. The bad news is that I’m not aware of a nice solution to the general problem for all side effects, and I suspect that no such solution exists. On the bright side, there are some very nice solutions that work really well in practice. The simplest is idempotence. This is a very simple idea: we make the tasks have the same effect no matter how many times they are executed.
This topographic map represents Ireland. It is designed for “hillwalking”. The contour lines are extracted from SRTM public data provided by NASA. These files contain a digitized ground represented by points. The sample rate defines a grid resolution for Ireland around 90m in northing and 60m in easting. In major cases, digitized points do not correspond with summits. Carrauntoohil (1039m, the highest summit of Ireland) does not appear in SRTM data. The altitude reaches only 1018m. Data were obtain from space with a radar. Because of the relative position between the radar and the earth, a shadow appears in some conditions (along ridges, behind summits…). This shadow matches with a gap in data (Imagine you with a flashlight in a dark room. It is hard to see what is in shadow). To close these gaps, you need other data or you can do interpolation. The second solution is applied in our case. There is one square degree per SRTM file with a sample rate of 1200×1200 points/square degree at Ireland latitude. [...] All in all you obtain contour lines pretty sufficient for walking.
Marcus Ramberg says: ‘If you have a chromecast and you’re not using castnow, I don’t know what is wrong with you.’
John Allspaw with an interesting assertion that we need to ask “how”, not “why” in five-whys postmortems:
“Why?” is the wrong question. In order to learn (which should be the goal of any retrospective or post-hoc investigation) you want multiple and diverse perspectives. You get these by asking people for their own narratives. Effectively, you’re asking “how?“ Asking “why?” too easily gets you to an answer to the question “who?” (which in almost every case is irrelevant) or “takes you to the ‘mysterious’ incentives and motivations people bring into the workplace.” Asking “how?” gets you to describe (at least some) of the conditions that allowed an event to take place, and provides rich operational data.
hooray, no more uberjars or monster commandlines!
‘From 19 Nov, 2014 00:52 to 05:50 UTC a subset of customers using Storage, Virtual Machines, SQL Geo-Restore, SQL Import/export, Websites, Azure Search, Azure Cache, Management Portal, Service Bus, Event Hubs, Visual Studio, Machine Learning, HDInsights, Automation, Virtual Network, Stream Analytics, Active Directory, StorSimple and Azure Backup Services in West US and West Europe experienced connectivity issues. This incident has now been mitigated.’ There was knock-on impact until 11:00 UTC (storage in N Europe), 11:45 UTC (websites, West Europe), and 09:15 UTC (storage, West Europe), from the looks of things. Should be an interesting postmortem.
Bitsets, also called bitmaps, are commonly used as fast data structures. Unfortunately, they can use too much memory. To compensate, we often use compressed bitmaps. Roaring bitmaps are compressed bitmaps which tend to outperform conventional compressed bitmaps such as WAH, EWAH or Concise. In some instances, they can be hundreds of times faster and they often offer significantly better compression. Roaring bitmaps are used in Apache Lucene (as of version 5.0 using an independent implementation) and Apache Spark (as of version 1.2).
‘Unsupervised anomaly detection is the process of finding outliers in data sets without prior training. In this paper, a histogram-based outlier detection (HBOS) algorithm is presented, which scores records in linear time. It assumes independence of the features making it much faster than multivariate approaches at the cost of less precision. A comparative evaluation on three UCI data sets and 10 standard algorithms show, that it can detect global outliers as reliable as state-of-the-art algorithms, but it performs poor on local outlier problems. HBOS is in our experiments up to 5 times faster than clustering based algorithms and up to 7 times faster than nearest-neighbor based methods.’
iPad On A Face by Cheryl Wu is a telepresence robot, except it’s a human with an iPad on his or her face.
The entire letter could have been taken from a page of GCHQ’s Joint Threat Research and Intelligence Group (JTRIG)—though perhaps as an email or series of tweets. The British spying agency GCHQ is one of the NSA’s closest partners. The mission of JTRIG, a unit within GCHQ, is to “destroy, deny, degrade [and] disrupt enemies by discrediting them.” And there’s little reason to believe the NSA and FBI aren’t using such tactics. The implications of these types of strategies in the digital age are chilling. Imagine Facebook chats, porn viewing history, emails, and more made public to discredit a leader who threatens the status quo, or used to blackmail a reluctant target into becoming an FBI informant. These are not far-fetched ideas. They are the reality of what happens when the surveillance state is allowed to grow out of control, and the full King letter, as well as current intelligence community practices illustrate that reality richly.
A nice new concurrency primitive from Gil Tene:
Have you ever had a need for logging or analyzing data that is actively being updated? Have you ever wanted to do that without stalling the writers (recorders) in any way? If so, then WriterReaderPhaser is for you. I’m not talking about logging messages or text lines here. I’m talking about data. Data larger than one word of memory. Data that holds actual interesting state. Data that keeps being updated, but needs to be viewed in a stable and coherent way for analysis or logging. Data like frame buffers. Data like histograms. Data like usage counts. Data that changes.see also Left-Right: http://concurrencyfreaks.blogspot.ie/2013/12/left-right-concurrency-control.html
Mastercard and Visa are removing the need for users to enter their passwords for identity confirmation as part of a revamp of the existing (oft-criticised) 3-D Secure scheme. The arrival of 3D Secure 2.0 next year will see the credit card giants moving away from the existing system of secondary static passwords to authorise online purchases, as applied by Verified by Visa and MasterCard SecureCode, towards a next-gen system based on more secure biometric and token-based prompts.(via Gordon)
excellent writeup on Aeron
Newly designed protocols should prefer encryption to cleartext operation. There may be exceptions to this default, but it is important to recognize that protocols do not operate in isolation. Information leaked by one protocol can be made part of a more substantial body of information by cross-correlation of traffic observation. There are protocols which may as a result require encryption on the Internet even when it would not be a requirement for that protocol operating in isolation. We recommend that encryption be deployed throughout the protocol stack since there is not a single place within the stack where all kinds of communication can be protected.Wow. so much for IPSec
FB goes public with its take on the Clos network-based datacenter network architecture
This is handy! ‘In Linux, /dev/full or the always full device is a special file that always returns the error code ENOSPC (meaning “No space left on device”) on writing, and provides an infinite number of null characters to any process that reads from it (similar to /dev/zero). This device is usually used when testing the behaviour of a program when it encounters a “disk full” error.’
Some cool stuff that comes along with Netty: an improved ByteBuffer, a thread-local object pool, a hashed-wheel Timer, and some nice mechanical-sympathy utils.
Excellent Vanity Fair article on the AF447 disaster, covering pilots’ team-leadership skills, Clipper Skippers, Alternate Law, and autopilot design: ‘There is an old truth in aviation that the reasons you get into trouble become the reasons you don’t get out of it.’ Also interesting: ‘The best pilots discard the [autopilot] automation naturally when it becomes unhelpful, and again there appear to be some cultural traits involved. Simulator studies have shown that Irish pilots, for instance, will gleefully throw away their crutches, while Asian pilots will hang on tightly. It’s obvious that the Irish are right, but in the real world Sarter’s advice is hard to sell. The automation is simply too compelling. The operational benefits outweigh the costs. The trend is toward more of it, not less. And after throwing away their crutches, many pilots today would lack the wherewithal to walk.’ (via Gavin Sheridan)
MetricsGraphics.js is a library built on top of D3 that is optimized for visualizing and laying out time-series data. It provides a simple way to produce common types of graphics in a principled, consistent and responsive way. The library currently supports line charts, scatterplots and histograms as well as features like rug plots and basic linear regression.
The Bedford Report for the HSE in 2011 showed that only approximately 10% of serious injuries (with hospital admission to a bed) incurred by cyclists in road traffic collisions were recorded by Gardai. If a cyclist is knocked off his/her bike from impact with a motorised vehicle that is a potential criminal offence if serious injury results. Cyclists expect all such RTCs to be properly and fully investigated and recorded with appropriate follow-up. That clearly is not happening at present. Acute hospitals need to document all admission cases arising from cyclist RTCs and inform the Gardai of them.
an [XPath-style] query language for JSON. You can extract and transform elements from a JSON document.Supported by the “aws” CLI tool, and in boto.
a new networked pub/sub library from Martin “Disruptor” Thompson, based around a replicated, persistent log of messages, with exceptionally low latency. Apache-licensed. Very similar to the realtime messaging stack we’ve built in Swrve. ;) https://github.com/real-logic/Aeron
As historian Nathan Ensmenger explained to a Stanford audience, as late as the 1960s many people perceived computer programming as a natural career choice for savvy young women. Even the trend-spotters at Cosmopolitan Magazine urged their fashionable female readership to consider careers in programming. In an article titled “The Computer Girls,” the magazine described the field as offering better job opportunities for women than many other professional careers. As computer scientist Dr. Grace Hopper told a reporter, programming was “just like planning a dinner. You have to plan ahead and schedule everything so that it’s ready when you need it…. Women are ‘naturals’ at computer programming.” James Adams, the director of education for the Association for Computing Machinery, agreed: “I don’t know of any other field, outside of teaching, where there’s as much opportunity for a woman.”
Whoa. Pity they didn’t do this earlier! Patent license, and all
Interesting — I hadn’t expected this. ‘Life expectancy at birth [in the US] in 1930 was indeed only 58 for men and 62 for women, and the retirement age was 65. But life expectancy at birth in the early decades of the 20th century was low due mainly to high infant mortality, and someone who died as a child would never have worked and paid into Social Security. A more appropriate measure is probably life expectancy after attainment of adulthood.’ …. ‘Men who attained age 65 could expect to collect Social Security benefits for almost 13 years (and the numbers are even higher for women).’ In Ireland, life expectancy at birth has increased 18.4 years since 1926 — but life expectancy for men aged 65 (the pension age) has only increased by 3.8 years. This means that increased life expectancy figures are not particularly relevant to the “pension crunch” story. Via Fred Logue: https://twitter.com/fplogue/status/532093184646873089
This is pretty awesome. All changes to a DynamoDB table can be streamed to a Kinesis stream, MySQL-replication-style. The nice bit is that it has a solid way to ensure readers won’t get overwhelmed by the stream volume (since ddb tables are IOPS-rate-limited), and Kinesis has a solid way to read missed updates (since it’s a Kafka-style windowed persistent stream). With this you have a pretty reliable way to ensure you’re not going to suffer data loss.
Recently Groupon announced a product with the same product name as GNOME. Groupon’s product is a tablet based point of sale “operating system for merchants to run their entire operation.” The GNOME community was shocked that Groupon would use our mark for a product so closely related to the GNOME desktop and technology. It was almost inconceivable to us that Groupon, with over $2.5 billion in annual revenue, a full legal team and a huge engineering staff would not have heard of the GNOME project, found our trademark registration using a casual search, or even found our website, but we nevertheless got in touch with them and asked them to pick another name. Not only did Groupon refuse, but it has now filed even more trademark applications (the full list of applications they filed can be found here, here and here). To use the GNOME name for a proprietary software product that is antithetical to the fundamental ideas of the GNOME community, the free software community and the GNU project is outrageous. Please help us fight this huge company as they try to trade on our goodwill and hard earned reputation.
This is due in part to huge growth in the data volumes and data traffic that is transported over our network, which has exceeded our forecasted growth. We are making a number of improvements to our international connectivity which will add significant capacity and this work will be completed in the next two or three weeks.Guess this is what happens when Amazon poach your IP network engineers. doh! More seriously though, if you’re marketing eFibre heavily, shouldn’t you be investing in the upstream capacity to go with it?
From the very beginning, Isis fanatics have been up to speed on [social media]. Which raises an interesting question: how come that GCHQ and the other intelligence agencies failed to notice the rise of the Isis menace until it was upon us? Were they so busy hoovering metadata and tapping submarine cables and “mastering the internet” (as the code name of one of their projects puts it) that they didn’t have time to see what every impressionable Muslim 14-year-old in the world with an internet connection could see?
One of the really important pieces on my land was this white-picket fence. The picket fence is probably 100 yards or less, within 100 yards of where they wanted to build this pipeline. I [plan to] extend it 8 feet every year for the rest of my life and I’ve been doing that for 25 years. It got me thinking, where does this piece end? Does it end at the actual structure of the fence or the things growing around it, growing through it, that are part of the photography, the documentation of it? I realized at that point that [the fence], and the other sculptures and pieces and incursions and conceptual works, were actually integral to that piece of land and to my practice. I had not intended for it to be a political piece, it was just a piece, an idea the follow-through of which at some point became poetic, you go, “Wait a minute the fence actually stopped them!” But the fence doesn’t actually enclose anything. It’s just a straight line. And it’s marking something that’s actually unmarkable, which is time. And one day it’ll be gone, as will I. The land will be changed–but it was just this crazy irony that kicked into play when I was standing there with those oil negotiators.
‘A constant throughput, correct latency-recording variant of wrk. This is a must-have when measuring network service latency — corrects for Coordinated Omission error:
wrk’s model, which is similar to the model found in many current load generators, computes the latency for a given request as the time from the sending of the first byte of the request to the time the complete response was received. While this model correctly measures the actual completion time of individual requests, it exhibits a strong Coordinated Omission effect, through which most of the high latency artifacts exhibited by the measured server will be ignored. Since each connection will only begin to send a request after receiving a response, high latency responses result in the load generator coordinating with the server to avoid measurement during high latency periods.
Good post on the pain of using CSV/JSON as a data interchange format:
eventually, the schema changes. Someone refactors the code generating the JSON and moves fields around, perhaps renaming few fields. The DBA added new columns to a MySQL table and this reflects in the CSVs dumped from the table. Now all those applications and scripts must be modified to handle both file formats. And since schema changes happen frequently, and often without warning, this results in both ugly and unmaintainable code, and in grumpy developers who are tired of having to modify their scripts again and again.
15 years ago today — Sitescooper appeared in NTK!
Official NTK policy is that if you’re not reading this in its definitive, non-proportional e-mail form, you’re a fricking girl. And all the best fricking girls these days have a Palm, so JUSTIN MASON has been kindly running the Web page through his brilliant sitescooper (aka snarfnews) program, and dumping the results for download at his site. NTK is available in DOC and iSilo formats, as are all kinds of other girlish, lavender-smelling Websites you may want to read, like The Register and the Linux Weekly News. And “Dr Koop’s Health News”.
Excellent write-up of this little-known undocumented GDocs behaviour, an artifact of its operational-transformation sync mechanism
Facebook’s take on libevent, I guess:
We are excited to announce the release of Proxygen, a collection of C++ HTTP libraries, including an easy-to-use HTTP server. In addition to HTTP/1.1, Proxygen (rhymes with “oxygen”) supports SPDY/3 and SPDY/3.1. We are also iterating and developing support for HTTP/2. Proxygen is not designed to replace Apache or nginx — those projects focus on building extremely flexible HTTP servers written in C that offer good performance but almost overwhelming amounts of configurability. Instead, we focused on building a high performance C++ HTTP framework with sensible defaults that includes both server and client code and that’s easy to integrate into existing applications. We want to help more people build and deploy high performance C++ HTTP services, and we believe that Proxygen is a great framework to do so.
A good example of a design pattern — by performing a relatively constant amount of work regardless of the input, we can predict scalability and reduce the risk of overload when something unexpected changes in that input
Bicycle theft in Ireland has doubled in Ireland since the introduction of the Bike to Work scheme in 2009. Almost 4,500 bicycle thefts were reported in Dublin in 2013, but the actual number of bike thefts is likely to be in the region of 20,000 in 2013 according to Irish household surveys and international experience[3,4]. The chances of a bike thief being caught is low, with a conviction rate of only 2% or reported thefts. Approximately 230,000 bicycles are imported into Ireland each year. “Bike theft is a low-risk, high-reward crime. If cars were being stolen at this rate there would be uproar.” Says Keith Byrne, Chairperson of the Dublin Cycling Campaign. Fear of bicycle theft may discourage bicycle use and many bicycle theft victims do not buy a replacement [7,8]. “Many people give up on cycling after their bicycle is stolen and it discourages others from taking up cycling as the word about the high risk of theft spreads. We need a co-ordinated multi-agency plan to tackle bicycle theft if we are to reach the Government target of 10% of journeys by bicycle by 2020” says Keith Byrne.Amen to that.
it’s not easy, basically (via Tony Finch)
DUB6 is expanding (or is it DUB14 now? can’t keep up)
The jobs will be across a variety of positions, including software engineers, technical engineers, technical managers, customer support and IT security.
A curated list of Docker resources.
An excellent tip from Gil “HDRHistogram” Tene:
Good example of why I always “calibrate” latency tools with ^Z tests. If ^Z results don’t make sense, don’t use [the] tool. ^Z test math examples: If you ^Z for half the time, Max is obvious. [90th percentile] should be 80% of the ^Z stall time.
Jay Kreps, Neha Narkhede, and Jun Rao are leaving LinkedIn to form a Kafka-oriented realtime event processing company
“Reading papers: essential. Slavishly implementing ideas you read: not necessarily a good idea. Trust me, I wrote an Amazon Dynamo clone.” Later in the discussion, on complex conflict resolution logic (as used in Dynamo, Voldemort, and Riak): “I reviewed 200 Voldemort stores, 190 used default lww conflict resolution. 10 had custom logic, all 10 of which had bugs.” — https://twitter.com/jaykreps/statuses/528292617784537088 (although IMO I’d prefer complex resolution to non-availability, when AP is required)
Ugh, Rails fail. It is impossible to drop a column from a Rails-managed table without downtime, even if nothing in the code accesses it (!!), without ugly hacks that don’t even seem to work on recent versions of ActiveRecord.
An obscure clause in EU copyright rules means no one can publish photos of public buildings in Belgium, like the Atomium, or France’s Eiffel tower at night without first asking permission from the rights owners.Ah, copyright.
Turns out ZK isn’t a good choice as a service discovery system, if you want to be able to use that service discovery system while partitioned from the rest of the ZK cluster:
I went into one of the instances and quickly did an iptables DROP on all packets coming from the other two instances. This would simulate an availability zone continuing to function, but that zone losing network connectivity to the other availability zones. What I saw was that the two other instances noticed the first server “going away”, but they continued to function as they still saw a majority (66%). More interestingly the first instance noticed the other two servers “going away”, dropping the ensemble availability to 33%. This caused the first server to stop serving requests to clients (not only writes, but also reads).So: within that offline AZ, service discovery *reads* (as well as writes) stopped working due to a lack of ZK quorum. This is quite a feasible outage scenario for EC2, by the way, since (at least when I was working there) the network links between AZs, and the links with the external internet, were not 100% overlapping. In other words, if you want a highly-available service discovery system in the fact of network partitions, you want an AP service discovery system, rather than a CP one — and ZK is a CP system. Another risk, noted on the Netflix Eureka mailing list at https://groups.google.com/d/msg/eureka_netflix/LXKWoD14RFY/tA9UnerrBHUJ :
ZooKeeper, while tolerant against single node failures, doesn’t react well to long partitioning events. For us, it’s vastly more important that we maintain an available registry than a necessarily consistent registry. If us-east-1d sees 23 nodes, and us-east-1c sees 22 nodes for a little bit, that’s OK with us.I guess this means that a long partition can trigger SESSION_EXPIRED state, resulting in ZK client libraries requiring a restart/reconnect to fix. I’m not entirely clear what happens to the ZK cluster itself in this scenario though. Finally, Pinterest ran into other issues relying on ZK for service discovery and registration, described at http://engineering.pinterest.com/post/77933733851/zookeeper-resilience-at-pinterest ; sounds like this was mainly around load and the “thundering herd” overload problem. Their workaround was to decouple ZK availability from their services’ availability, by building a Smartstack-style sidecar daemon on each host which tracked/cached ZK data.
A good story of when Kafka _didn’t_ fit the use case:
We came up with a complicated process of app-level replication for our messages into two separate Kafka clusters. We would then do end-to-end checking of the two clusters, detecting dropped messages in each cluster based on messages that weren’t in both. It was ugly. It was clearly going to be fragile and error-prone. It was going to be a lot of app-level replication and horrible heuristics to see when we were losing messages and at least alert us, even if we couldn’t fix every failure case. Despite us building a Kafka prototype for our ETL — having an existing investment in it — it just wasn’t going to do what we wanted. And that meant we needed to leave it behind, rewriting the ETL prototype.
If the data aggregators know everything about you — including biometric data, healthcare history, where you live, where you work, what you do at the weekend, what medicines you take, etc. — and can track you as an individual, does it really matter that they don’t know your _name_? They legally track, and sell, everything else.
As the data we generate about ourselves continues to grow exponentially, brokers and aggregators are moving on from real-time profiling — they’re cross-linking data sets to predict our future behaviour. Decisions about what we see and buy and sign up for aren’t made by us any more; they were made long before. The aggregate of what’s been collected about us previously — which is near impossible for us to see in its entirety — defines us to companies we’ve never met. What I am giving up without consent, then, is not just my anonymity, but also my right to self-determination and free choice. All I get to keep is my name.
bad news for low-latency apps. See also its impact on redis: http://antirez.com/news/84
Although in some cases x1.5 is considered good practice. YMMV I guess
I found that I was able to run the algorithm in about 10 hours on an AWS large GPU instance bringing it in at about $0.65 plus tax.Bottom line: MD5 is feasibly attackable by pretty much anyone now.
Great to see museums campaigning for copyright reform — this makes perfect sense.
Display cases in the Imperial War Museum, National Library of Scotland and University of Leeds sit empty. They should contain letters from the First World War; from a young girl to her father serving as a soldier and from soldiers to their families back home. Because of current UK copyright laws the original letters cannot be displayed. At the moment the duration of copyright in certain unpublished works is to the end of the year 2039, regardless how old the work is. The Free Our History campaign wants the term of copyright protection in unpublished texts to be reduced to the author’s lifetime plus 70 years.
I am obsessed with these.
From this Ask MetaFilter post, it seems that the Roman letters are tacked onto [Chinese and Japanese] fonts almost as an afterthought, for when you need to interject a few words of English into your Chinese website, for example. Hence, they aren’t really optimised for the eye of a non-Chinese writer, or perhaps aren’t optimised much at all, and usually look like this. It’s not one specific font as I thought.Informed! (via Elliot)
LOL, hardware people writing drivers. Good reason not to buy Asus, I guess
‘People telling people to execute arbitrary code over the network. Run code from our servers as root. But HTTPS, so it’s no biggie.’ YES.
When a player adopted democracy in Civilization, their aggression would be automatically reduced by 2. Code being code, if Gandhi went democratic his aggression wouldn’t go to -1, it looped back around to the ludicrously high figure of 255, making him as aggressive as a civilization could possibly be.
good talk from Hailo’s Matt Heath, doing nice stuff with Go and a well-supported microservices architecture
Trust US banks to fuck up their attempts at security :( US “chip-and-signature” cards are still entirely forgeable because the banks fear that consumers are too stupid to use a PIN, basically.
BK: So, I guess we should all be grateful that banks and retailers in the United States are finally taking steps to move toward chip [and signature] cards, but it seems to me that as long as these chip cards still also store cardholder data on a magnetic stripe as a backup, that the thieves can still steal and counterfeit this card data — even from chip cards. Litan: Yes, that’s the key problem for the next few years. Once mag stripe goes away, chip-and-PIN will be a very strong solution. The estimates are now that by the end of 2015, 50 percent of the cards and terminals will be chip-enabled, but it’s going to be several years before we get closer to full compliance. So, we’re probably looking at about 2018 before we can start making plans to get rid of the magnetic stripe on these cards.
Turns out there are a few bugs in EMR’s S3 support, believe it or not. 1. ‘Consider disabling Hadoop’s speculative execution feature if your cluster is experiencing Amazon S3 concurrency issues. You do this through the mapred.map.tasks.speculative.execution and mapred.reduce.tasks.speculative.execution configuration settings. This is also useful when you are troubleshooting a slow cluster.’ 2. Upgrade to AMI 3.1.0 or later, otherwise retries of S3 ops don’t work.
Stephanie Dean on Amazon’s approach to CMs. This is solid gold advice for any company planning to institute a sensible technical change management process
I asked around my ex-Amazon mates on twitter about good docs on incident response practices outside the “iron curtain”, and they pointed me at this blog (which I didn’t realise existed). Stephanie Dean was the front-line ops manager for Amazon for many years, over the time where they basically *fixed* their availability problems. She since moved on to Facebook, Demonware, and Twitter. She really knows her stuff and this blog is FULL of great details of how they ran (and still run) front-line ops teams in Amazon.
Carlos Baquero presents several operation, state-based CRDTs for use in AP systems like Voldemort and Riak
Applications can saturate – i.e. become unable to serve users in a timely manner. Some users may experience high latencies, while others may not receive any service at all. The authors argue that it is better to downgrade the user experience and continue serving a larger number of clients with reasonable latency. “We define a cloud application as brownout compliant if it can gradually downgrade user experience to avoid saturation.” This is actually very reminiscent of circuit breakers, as described in Nygard’s ‘Release It!’ and popularized by Netflix. If you’re already designing with circuit breakers, you’ve probably got all the pieces you need to add brownout support to your application relatively easily. “Our work borrows from the concept of brownout in electrical grids. Brownouts are an intentional voltage drop often used to prevent blackouts through load reduction in case of emergency. In such a situation, incandescent light bulbs dim, hence originating the term.” “To lower the maintenance effort, brownouts should be automatically triggered. This enables cloud applications to rapidly and robustly avoid saturation due to unexpected environmental changes, lowering the burden on human operators.”This is really similar to the Circuit Breaker pattern — in fact it feels to me like a variation on that, driven by measured latencies of operations/requests. See also http://blog.acolyer.org/2014/10/27/improving-cloud-service-resilience-using-brownout-aware-load-balancing/ .
“Slow-motion Chernobyl”, as Greenpeace are calling it. You thought legacy code was a problem? try legacy Magnox fuel rods.
Previously unseen pictures of two storage ponds containing hundreds of highly radioactive fuel rods at the Sellafield nuclear plant show cracked concrete, seagulls bathing in the water and weeds growing around derelict machinery. But a spokesman for owners Sellafield Ltd said the 60-year-old ponds will not be cleaned up for decades, despite concern that they are in a dangerous state and could cause a large release of radioactive material if they are allowed to deteriorate further. “The concrete is in dreadful condition, degraded and fractured, and if the ponds drain, the Magnox fuel will ignite and that would lead to a massive release of radioactive material,” nuclear safety expert John Large told the Ecologist magazine. “I am very disturbed at the run-down condition of the structures and support services. In my opinion there is a significant risk that the system could fail.
An interview with Richard Bartle, the creator of MUD, back in 1978.
Perceiving the different ways in which players approached the game led Bartle to consider whether MMO players could be classified according to type. “A group of admins was having an argument about what people wanted out of a MUD in about 1990,” he recalls. “This began a 200-long email chain over a period of six months. Eventually I went through everybody’s answers and categorised them. I discovered there were four types of MMO player. I published some short versions of them then, when the journal of MUD research came out I wrote it up as a paper.” The so-called Bartle test, which classifies MMO players as Achievers, Explorers, Socialisers or Killers (or a mixture thereof) according to their play-style remains in widespread use today. Bartle believes that you need a healthy mix of all dominant types in order to maintain a successful MMO ecosystem. “If you have a game full of Achievers (players for whom advancement through a game is the primary goal) the people who arrive at the bottom level won’t continue to play because everyone is better than them,” he explains. “This removes the bottom tier and, over time, all of the bottom tiers leave through irritation. But if you have Socialisers in the mix they don’t care about levelling up and all of that. So the lowest Achievers can look down on the Socialisers and the Socialisers don’t care. If you’re just making the game for Achievers it will corrode from the bottom. All MMOs have this insulating layer, even if the developers don’t understand why it’s there.”
Redis uses forking to perform persistence flushes, which means that once every 30 minutes it performs like crap (and kills the 99th percentile latency). Given this, various Redis people have been benchmarking fork() times on various Xen platforms, since Xen has a crappy fork() implementation
Jay Rosen interviews his 17-year-old daughter. it’s pretty eye-opening. Got to start them early!
Carbon is a great idea, but fundamentally, twisted doesn’t do what carbon-relay or carbon-aggregator were built to do when hit with sustained and heavy throughput. Much to my chagrin, concurrency isn’t one of python’s core competencies.+1, sadly. We are patching around the edges with half-released third-party C rewrites in our graphite setup, as we exceed the scale Carbon can support.
MOST of the page view attempts will experience the 99%’lie server response time in modern web applications. You didn’t read that wrong.
The next Pub Standards, on Thursday 13th November, will be the last one. When I started Pub Standards in August 2010, there wasn’t very many meetups for people who build apps, interfaces and businesses. These days, there are loads! I don’t feel that Pub Standards is needed anymore. It served it’s purpose — other meetups were formed, startups were founded, projects were created and people got hired. We had a good run :)
$50 print (plus shipping of course), 16″ x 16″
Good, thought-provoking post on good client library approaches for complex client-server systems, particularly distributed stores like Voldemort or Riak. I’m of the opinion that a smart client lib is unavoidable, and in fact essential, since the clients are part of the distributed system, personally.
Dublin had its own time zone, 25 minutes off what would become GMT, until 1916
a Riak-based clone of Roshi, the CRDT server built on top of Redis. some day I’ll write up the CRDT we use on top of Voldemort in $work. Comments: https://lobste.rs/s/tim5xc
‘Verizon Wireless is monitoring users’ mobile internet traffic, using a token slapped onto web requests, to facilitate targeted advertising even if a user has opted out. The unique identifier token header (UIDH) was launched two years ago, and has caused an uproar in tech circles after it was re-discovered Thursday by Electronic Frontier Foundation staffer Jacob Hoffman-Andrews. The Relevant Mobile Advertising program, under which the UIDH was used, allowed a restaurant to advertised to locals only or for retail websites to promote to previous visitors, according to Verizon Wireless.’
‘In many networking systems, Bloom filters are used for high-speed set membership tests. They permit a small fraction of false positive answers with very good space efficiency. However, they do not permit deletion of items from the set, and previous attempts to extend “standard” Bloom filters to support deletion all degrade either space or performance. We propose a new data structure called the cuckoo filter that can replace Bloom filters for approximate set member- ship tests. Cuckoo filters support adding and removing items dynamically while achieving even higher performance than Bloom filters. For applications that store many items and target moderately low false positive rates, cuckoo filters have lower space overhead than space-optimized Bloom filters. Our experimental results also show that cuckoo filters out-perform previous data structures that extend Bloom filters to support deletions substantially in both time and space.’
This has _already_ been used to trump national law. As Simon McGarr noted at https://twitter.com/Tupp_Ed/statuses/526103760041680898 : ‘Philip Morris initiated a dispute under the Australia-Hong Kong Bilateral Investment Treaty to force #plainpacks repeal and compensation’. “Plain packs” anti-smoking is being bitterly fought at the moment here in Ireland. More from the US point of view: http://www.washingtonpost.com/opinions/harold-meyerson-allowing-foreign-firms-to-sue-nations-hurts-trade-deals/2014/10/01/4b3725b0-4964-11e4-891d-713f052086a0_story.html : ‘The Obama administration’s insistence on ISDS may please Wall Street, but it threatens to undermine some of the president’s landmark achievements in curbing pollution and fighting global warming, not to mention his commitment to a single standard of justice. It’s not worthy of the president, and he should join Europe in scrapping it.’
We’ve started running game day exercises at Stripe. During a recent game day, we tested failing over a Redis cluster by running kill -9 on its primary node, and ended up losing all data in the cluster. We were very surprised by this, but grateful to have found the problem in testing. This result and others from this exercise convinced us that game days like these are quite valuable, and we would highly recommend them for others.Excellent post. Game days are a great idea. Also: massive Redis clustering fail
“Everybody hits the wall, generally between three and five months,” says a former YouTube content moderator I’ll call Rob. “You just think, ‘Holy shit, what am I spending my day doing? This is awful.’”
Perhaps simply by the virtue of being a part of that bundle, the strings utility tries to leverage the common libbfd infrastructure to detect supported executable formats and “optimize” the process by extracting text only from specific sections of the file. Unfortunately, the underlying library can be hardly described as safe: a quick pass with afl (and probably with any other competent fuzzer) quickly reveals a range of troubling and likely exploitable out-of-bounds crashes due to very limited range checking
This Java library can route paths to targets and create paths from targets and params (reverse routing). This library is tiny, without additional dependencies, and is intended for use together with an HTTP server side library. If you want to use with Netty, see netty-router.
classic replication paper, via aphyr: ‘This paper presents an updated version of Viewstamped Replication, a replication technique that handles failures in which nodes crash. It describes how client requests are handled, how the group reorganizes when a replica fails, and how a failed replica is able to rejoin the group. The paper also describes a number of important optimizations and presents a protocol for handling reconfigurations that can change both the group membership and the number of failures the group is able to handle.’
Holy shit we are living in the future.
BioBrick parts are DNA sequences which conform to a restriction-enzyme assembly standard. These Lego-like building blocks are used to design and assemble synthetic biological circuits, which would then be incorporated into living cells such as Escherichia coli cells to construct new biological systems. Examples of BioBrick parts include promoters, ribosomal binding sites (RBS), coding sequences and terminators.(via Soren)
I have to agree with this assessment — there are a lot of loose ends still for production use of Docker in a SOA stack environment:
From my point of view, Docker is probably the best thing I’ve seen in ages to automate a build. It allows to pre build and reuse shared dependencies, ensuring they’re up to date and reducing your build time. It avoids you to either pollute your Jenkins environment or boot a costly and slow Virtualbox virtual machine using Vagrant. But I don’t feel like it’s production ready in a complex environment, because it adds too much complexity. And I’m not even sure that’s what it was designed for.
This is a very solid benchmarking post, examining Kafka in good detail. Nicely done. Bottom line:
I basically spend 2/3 of my work time torture testing and operationalizing distributed systems in production. There’s some that I’m not so pleased with (posts pending in draft forever) and some that have attributes that I really love. Kafka is one of those systems that I pretty much enjoy every bit of, and the fact that it performs predictably well is only a symptom of the reason and not the reason itself: the authors really know what they’re doing. Nothing about this software is an accident. Performance, everything in this post, is only a fraction of what’s important to me and what matters when you run these systems for real. Kafka represents everything I think good distributed systems are about: that thorough and explicit design decisions win.
wow, an actually quite-good cheapo Android tablet from Tesco for UKP65 of Clubcard vouchers, recommended by conoro. Good for the kids
Major improvements for Kafka consistency coming in 0.8.2; replication to multiple in-sync replicas, controlled by a new “min.isr” setting
I have repeatedly been confounded to discover just how many mistakes in both test and application code stem from misunderstandings or misconceptions about time. By this I mean both the interesting way in which computers handle time, and the fundamental gotchas inherent in how we humans have constructed our calendar — daylight savings being just the tip of the iceberg. In fact I have seen so many of these misconceptions crop up in other people’s (and my own) programs that I thought it would be worthwhile to collect a list of the more common problems here.See also the follow-up: http://infiniteundo.com/post/25509354022/more-falsehoods-programmers-believe-about-time-wisdom (via Marc)
What an utter fuckup. Business as usual for Irish Water:
However the spokeswoman said application packs for rented dwellings would be addressed to the landlord, at the landlord’s residence, and it would be the landlord’s responsibility to ensure the tenant received the application pack. Bills are to be issued quarterly, but as Irish Water will have the tenant’s PPS number, the utility firm will be able to pursue the tenant for any arrears and even apply any arrears to new accounts, when the tenant moves to a new address. Last week landlords had expressed concern over potential arrears, the liability for them and the possibility of being used as collection agents by Irish Water.
ugh, what a mess….
* Every rental unit in the State is to get a pack addressed personally to the occupant. If Irish Water does not have details of a tenant, the pack will be addressed to ‘The Occupier’ * Packs will only be issued to individual rental properties in so far as Irish Water is aware of them * Landlords can contact Irish Water to advise they have let a property * Application Packs are issued relative to the information on the Irish Water mailing list. If this is incorrect or out of date, landlords can contact Irish Water to have the information adjusted *Irish Water will contact known landlords after the initial customer application campaign, to advise of properties for which no application has been received * Irish Water said that when a household is occupied the tenant is liable and when vacant the owner is liable. Both should advise Irish Water of change of status to the property – the tenant to cease liability, the landlord to take it up. Either party may take a reading and provide it to Irish Water, alternatively Irish Water will bill on average consumption, based on the date of change.
Like, say, the Christian right, which came together through the social media of its day — little-watched television broadcasts, church bulletins, newsletters—or the Tea Party, which found its way through self-selection on social media and through back channels, Gamergate, in the main, comprises an assortment of agitators who sense which way the winds are blowing and feel left out. It has found a mobilizing event, elicited response from the established press, and run a successful enough public relations campaign that it’s begun attracting visible advocates who agree with the broad talking points and respectful-enough coverage from the mainstream press. If there is a ground war being waged, as the movement’s increasingly militaristic rhetoric suggests, Gamergate is fighting largely unopposed. A more important resemblance to the Tea Party, though, is in the way in which it’s focused the anger of people who realize the world is changing, and not necessarily to their benefit.
There are several reasons that the ID cards have proved so easy to steal: Identity numbers started to be issued in the 1960s and still follow the same pattern. The first few digits are the user’s birth date, followed by either a one for male or two for female; Their usage across different sectors makes them master keys for hackers, say experts; If details are leaked, citizens are unable to change themvia Tony Finch.
looks great, around the corner from Cineworld on King’s Inn St, D1
with Neil McKenzie, Nov 9-16 2014, in the National History Museum in Dublin: ‘These six helmets/viewing devices start off by exploring physical conditions of viewing: if we have two eyes, they why is our vision so limited? Why do we have so little perception of depth? Why don’t our two eyes offer us two different, complementary views of the world around us? Why can’t they extend from our body so we can see over or around things? Why don’t they allow us to look behind and in front at the same time, or sideways in both directions? Why can’t our two eyes simultaneously focus on two different tasks? Looking through Michael Land’s defining work Animal Eyes, we see that nature has indeed explored all of these possibilities: a Hammerhead Shark has hyper-stereo vision; a horse sees 350° around itself; a chameleon has separately rotatable eyes… The series of Meta-Perceptual Helmets do indeed explore these zoological typologies: proposing to humans the hyper-stereo vision of the hammerhead shark; or the wide peripheral vision of the horse; or the backward/forward vision of the chameleon… but they also take us into the unnatural world of mythology and literature: the Cheshire Cat Helmet is so called because of the strange lingering effect of dominating visual information such as a smile or the eyes; the Cyclops allows one large central eye to take in the world around while a second tiny hidden eye focuses on a close up task (why has the creature never evolved that can focus on denitting without constantly having to glance around?).’ (via Emma)
The figures show that, between 2004 and 2013, an average of 71.7 per cent of students at TCD graduated with either a 1st or a 2.1. DCU and UCC had the next highest rate of such awards (64.3 per cent and 64.2 per cent respectively), followed by UCD (55.8 per cent), NUI Galway (54.7 per cent), Maynooth University (53.7 per cent) and University of Limerick (50.2 per cent).
Last year we interviewed Oleg Moskalenko and presented the rfc5766-turn-server project, which is a free open source and extremely popular implementation of TURN and STURN server. A few months later we even discovered Amazon is using this project to power its Mayday service. Since then, a number of features beyond the original RFC 5766 have been defined at the IETF and a new open-source project was born: the coTURN project.
Today we are publishing details of a vulnerability in the design of SSL version 3.0. This vulnerability allows the plaintext of secure connections to be calculated by a network attacker.ouch.
It’s been a while since I wrote a long-form blog post here, but this post on the Swrve Engineering blog is worth a read; it describes how we use SSD caching on our EC2 instances to greatly improve EBS throughput.
“O Cormac, grandson of Conn”, said Carbery, “What is the worst pleading and arguing?” “Not hard to tell”, said Cormac. “Contending against knowledge, contending without proofs, taking refuge in bad language, a stiff delivery, a muttering speech, hair-splitting, uncertain proofs, despising books, turning against custom, shifting one’s pleading, inciting the mob, blowing one’s own trumpet, shouting at the top of one’s voice.”
a simple, lightweight HTTP server for storing and distributing custom Debian packages around your organisation. It is designed to make it as easy as possible to use Debian packages for code deployments and to ease other system administration tasks.
ZDNet’s Steven J. Vaughan-Nichols on the systemd mess (via Kragen)
Criminal complaints have been filed in the UK against Gamma “acting as an accessory to Bahrain’s illegal targeting of activists” using the FinFisher spyware
Meritocracy is a myth. And our belief in it is holding back the tech industry from getting better.
“It’s completely insane. It’s insane that you even have to say out loud that sending death threats to people who disagree with your opinion of video games is wrong. Yet here we are: Apparently, it needs to be said.”
#Gamergate, as they have treated myself and peers in our industry, is a hate group. This word, again, should not lend them any mystique or credence. Rather it should illuminate the fact that even the most nebulous and inconsistent ideas can proliferate wildly if strung onto the organizational framework of the hate group, which additionally gains a startling amount of power online. #Gamergate is a hate group, and they are all the more dismissible for it. And the longer we treat them otherwise, the longer I fear for our industry’s growth.
A group representing frontline emergency staff has warned lives will be lost unless the Government reverses its decision on a new national postcode system due to be rolled out next spring. John Kidd, chairman of the Irish Fire and Emergency Services Association, said the “mainly random nature” of the Eircode system would mean errors by users would go unnoticed, as well as cause confusion and may be “catastrophic” in terms of sending services to the wrong location. [....] Neil McDonnell, general manager of the Freight Transport Association Ireland, said he understood Mr Kidd’s concerns. “Take, for example, two adjacent houses in Glasnevin, Dublin,” said Mr McDonnell. “One could be D11 ZXQ8, the other one D11 67TR. The four-character unique identifier is completely random, with no sequence or algorithm linking one house to the other.”
Two types of people own homes in Vancouver?—?wealthy foreigners who are looking for a place to park their money, and long-time Vancouver residents who have benefited from skyrocketing equity, through no actual effort of their own. There is a simple problem with these people being the primary homeowners in any city?—?they don’t actually create much value for the place they live in. A very large percentage of wealthy foreigners who “park” their money here don’t actually live in Vancouver. Take a drive around most expensive areas and you’ll realize the homes are empty. At most, they send their kids to live in Vancouver, learn english/go to school, and then return to their country (usually to Hong Kong). For some reason this is okay with people who live here. The amount of value added to a city from this sort of activity approaches zero. In fact, I’d argue that these people actually leech off of the system more than anything else.
Welp, that’s the end of my reading The Escapist. this is fucked up. ‘these people say that this is a hate movement, but let’s see what these white supremacists and serial harassers have to say’
hmason: TIL that the phrase software “patch” is from a physical patch applied to Mark 1 paper tape to modify the program.It’s amazing how a term like that can become so divorced from its original meaning so effectively. History!
I analyzed several chunks of The Ultimate Player’s Guide using the Flesch-Kincaid Reading Ease scale, and they scored from grade 8 to grade 11. Yet in my neighborhood they’re being devoured by kids in the early phases of elementary school. Games, it seems, can motivate kids to read—and to read way above their level. This is what Constance Steinkuehler, a games researcher at the University of Wisconsin-Madison, discovered. She asked middle and high school students who were struggling readers (one 11th-grade student read at a 6th-grade level) to choose a game topic they were interested in, and then she picked texts from game sites for them to read—some as difficult as first-year-college language. The kids devoured them with no help and nearly perfect accuracy. How could they do this? “Because they’re really, really motivated,” Steinkuehler tells me. It wasn’t just that the students knew the domain well; there were plenty of unfamiliar words. But they persisted more because they cared about the task. “It’s situated knowledge. They see a piece of language, a turn of phrase, and they figure it out.”When my kids are playing Minecraft, there’s a constant stream of “how do you spell X?” as they craft nametags for their pets. It’s great!
Niall Heery belatedly follows up Small Engine Repair, his 2006 mumblecore critical hit, with a slightly less off-centre comedy that makes imaginative use of a smashing cast. The story skirts tragedy on its leisurely passage from mishap to misadventure, but Gold remains the sort of picture you want to hug indulgently to a welcoming bosom. It gives humanism a good name.Go Niall! it’s a great movie, go see it
web service API for Dublin Bikes data (and other similar bikesharing services run by JCD):
Two kinds of data are delivered by the platform: Static data provides stable information like station position, number of bike stands, payment terminal availability, etc. Dynamic data provides station state, number of available bikes, number of free bike stands, etc. Static data can be downloaded manually in file format or accessed through the API. Dynamic data are refreshed every minute and can be accessed only through the API.Ruby API: https://github.com/oisin/bikes
my coworker JK’s favourite games of 2013: Gone Home, Last Of Us, Proteus, Papers Please etc. I really want to play these, since they’re all totally my bag too.
Amazon has perhaps 1% of the US retail market by value. Should it stop entering new categories and markets and instead take profit, and by extension leave those segments and markets for other companies? Or should it keep investing to sweep them into the platform? Jeff Bezos’s view is pretty clear: keep investing, because to take profit out of the business would be to waste the opportunity. He seems very happy to keep seizing new opportunities, creating new businesses, and using every last penny to do it.
Massive improvement over plain old Hadoop. This blog post goes into really solid techie reasons why, including:
First and foremost, in Spark 1.1 we introduced a new shuffle implementation called sort-based shuffle (SPARK-2045). The previous Spark shuffle implementation was hash-based that required maintaining P (the number of reduce partitions) concurrent buffers in memory. In sort-based shuffle, at any given point only a single buffer is required. This has led to substantial memory overhead reduction during shuffle and can support workloads with hundreds of thousands of tasks in a single stage (our PB sort used 250,000 tasks).Also, use of Timsort, an external shuffle service to offload from the JVM, Netty, and EC2 SR-IOV.
During the 1970s, when Northern Ireland was gripped by near-civil-war, British military intelligence staged the evidence of “black masses” in order to create a Satanism panic among the “superstitious” Irish to discredit the paramilitaries. The secret history of imaginary Irish Satanism is documented in Black Magic and Bogeymen: Fear, Rumour and Popular Belief in the North of Ireland 1972-74, a new book from Sheffield University’s Richard Jenkins, who interviewed Captain Colin Wallace, the former head of British Army “black operations” for Northern Ireland.
Interesting — I hadn’t heard of this being an official practise anywhere before (although we actually did it ourselves this week)…
If a build has made it [past the 'integration test' phase], it is ready to be deployed to one or more internal environments for user-acceptance testing. Users could be UI developers implementing a new feature using the API, UI Testers performing end-to-end testing or automated UI regression tests. As far as possible, we strive to not have user-acceptance tests be a gating factor for our deployments. We do this by wrapping functionality in Feature Flags so that it is turned off in Production while testing is happening in other environments.
Felix says: ‘Like I said, I’d like to move it to a more general / non-personal repo in the future, but haven’t had the time yet. Anyway, you can still browse the code there for now. It is not a big code base so not that hard to wrap one’s mind around it. It is Apache licensed and both Kafka and Voldemort are using it so I would say it is pretty self-contained (although Kafka has not moved to Tehuti proper, it is essentially the same code they’re using, minus a few small fixes missing that we added). Tehuti is a bit lower level than CodaHale (i.e.: you need to choose exactly which stats you want to measure and the boundaries of your histograms), but this is the type of stuff you would build a wrapper for and then re-use within your code base. For example: the Voldemort RequestCounter class.’
Great presentation about Github dev culture and building software without breakage, but still with real progress.
Syncthing is becoming Ind.ie Pulse. Pulse replaces proprietary sync and cloud services with something open, trustworthy and decentralised. Your data is your data alone and you deserve to choose where it is stored, if it is shared with some third party, and how it’s transmitted over the Internet.
This is a harrowing post from Kathy Sierra, full of valid observations:
You’re probably more likely to win the lottery than to get any law enforcement agency in the United States to take action when you are harassed online, no matter how visciously and explicitly. Local agencies lack the resources, federal agencies won’t bother.That to the power of ten in Ireland, too, I’d suspect. Fuck this. Troll culture is way out of control….
An embryonic metrics library for Java/Scala from Felix GV at LinkedIn, extracted from Kafka’s metric implementation and in the new Voldemort release. It fixes the major known problems with the Meter/Timer implementations in Coda-Hale/Dropwizard/Yammer Metrics. ‘Regarding Tehuti: it has been extracted from Kafka’s metric implementation. The code was originally written by Jay Kreps, and then maintained improved by some Kafka and Voldemort devs, so it definitely is not the work of just one person. It is in my repo at the moment but I’d like to put it in a more generally available (git and maven) repo in the future. I just haven’t had the time yet… As for comparing with CodaHale/Yammer, there were a few concerns with it, but the main one was that we didn’t like the exponentially decaying histogram implementation. While that implementation is very appealing in terms of (low) memory usage, it has several misleading characteristics (a lack of incoming data points makes old measurements linger longer than they should, and there’s also a fairly high possiblity of losing interesting outlier data points). This makes the exp decaying implementation robust in high throughput fairly constant workloads, but unreliable in sparse or spiky workloads. The Tehuti implementation provides semantics that we find easier to reason with and with a small code footprint (which we consider a plus in terms of maintainability). Of course, it is still a fairly young project, so it could be improved further.’ More background at the kafka-dev thread: http://mail-archives.apache.org/mod_mbox/kafka-dev/201402.mbox/%3C131A7649-ED57-45CB-B4D6-F34063267664@linkedin.com%3E
‘Chiranjeeb Buragohain and Subhash Suri: “Quantiles on Streams” in Encyclopedia of Database Systems, Springer, pp 2235–2240, 2009. ISBN: 978-0-387-35544-3′, cited by Martin Kleppman in http://mail-archives.apache.org/mod_mbox/kafka-dev/201402.mbox/%3C131A7649-ED57-45CB-B4D6-F34063267664@linkedin.com%3E as a good, short literature survey re estimating percentiles with a small memory footprint.
Many Belkin routers attempt to determine if they’re connected to the internet by pinging ‘heartbeat.belkin.com’, in a classic amateur fail move. Good reason not to run Belkin firmware if that’s the level of code quality to expect
An _extremely_ detailed resource about the bash bug