Let’s rewind to September 2012. It was about then- according to this recently published report (paywall) in The American Journal of Tropical Medicine – that an “otherwise healthy, 36-year-old man” felt a rough patch in his mouth, a scaly little area his right cheek. It didn’t hurt. But then it didn’t stay there either. He started testing for it with his tongue. It traveled. It moved to the back of his mouth, then forward, coiled backwards again. In the language of science: “These rough patches would appear and disappear on a daily basis, giving the patient the indirect sense that there was an organism moving within the oral cavity.”
Wow, this looks excellent. A must-read for people working on systems with high-volume, low-latency phone-to-server communications — and free!
How prepared are you to build fast and efficient web applications? This eloquent book provides what every web developer should know about the network, from fundamental limitations that affect performance to major innovations for building even more powerful browser applications—including HTTP 2.0 and XHR improvements, Server-Sent Events (SSE), WebSocket, and WebRTC. Author Ilya Grigorik, a web performance engineer at Google, demonstrates performance optimization best practices for TCP, UDP, and TLS protocols, and explains unique wireless and mobile network optimization requirements. You’ll then dive into performance characteristics of technologies such as HTTP 2.0, client-side network scripting with XHR, real-time streaming with SSE and WebSocket, and P2P communication with WebRTC. Deliver optimal TCP, UDP, and TLS performance; Optimize network performance over 3G/4G mobile networks; Develop fast and energy-efficient mobile applications; Address bottlenecks in HTTP 1.x and other browser protocols; Plan for and deliver the best HTTP 2.0 performance; Enable efficient real-time streaming in the browser; Create efficient peer-to-peer videoconferencing and low-latency applications with real-time WebRTC transportsVia Eoin Brazil.
3 new Snowden leaks, covering acquisition of Yahoo address books, buddy lists, and email account activity, and how spammer activity required intervention to avoid losing useful data in the noise
slides (lots of slides) from Baron Schwartz’ talk at Velocity in NYC.
Timestamps, as implemented in Riak, Cassandra, et al, are fundamentally unsafe ordering constructs. In order to guarantee consistency you, the user, must ensure locally monotonic and, to some extent, globally monotonic clocks. This is a hard problem, and NTP does not solve it for you. When wall clocks are not properly coupled to the operations in the system, causal constraints can be violated. To ensure safety properties hold all the time, rather than probabilistically, you need logical clocks.
Using the correct User-Agent: string, all auth is bypassed on several released models of D-Link and Planex routers. Horrific fail by D-Link
one of the most obvious inferences from the Snowden revelations published by the Guardian, New York Times and ProPublica recently is that the NSA has indeed been up to the business of inserting covert back doors in networking and other computing kit. The reports say that, in addition to undermining all of the mainstream cryptographic software used to protect online commerce, the NSA has been “collaborating with technology companies in the United States and abroad to build entry points into their products”. These reports have, needless to say, been strenuously denied by the companies, such as Cisco, that make this networking kit. Perhaps the NSA omitted to tell DARPA what it was up to? In the meantime, I hear that some governments have decided that their embassies should no longer use electronic communications at all, and are returning to employing couriers who travel the world handcuffed to locked dispatch cases. We’re back to the future, again.
The mistake came when an electoral commission accidentally published results showing a victory for Ilham Aliyev, the country’s long-standing President, a day before voting. Meydan TV, an online channel critical of the government, released a screenshot from a mobile app for the Azerbaijan Central Election Commission which showed that Mr Aliyev had received 72.76 per cent of the vote compared with 7.4 per cent for the opposition candidate, Jamil Hasanli. The screenshot also indicates that the app displayed information about how many people voted at various times during the day. Polls opened at 8am.
According to EasyDNS:
Any registrar that has taken one of these sites offline that now impedes the registrants of those domains from simply getting their domain names out of there and back online somewhere else will then be subject to the TDRP – Transfer Dispute Resolution Policy and if they lose (which they will) they will be subject to TDRP fees assesed by the registry operator, and to quote the TDRP itself “Transfer dispute resolution fees can be substantial”. This is why it is never a good idea to just react to pressure in the face of obnoxious bluster – in the very act of trying to diffuse any perceived culpability you end up opening yourself to real liability.
interesting discussion in the comments. “Patricia”‘s process is particularly hair-raisingly complex, involving 3 separate machines and a multitude of VMs
The ever-plummeting chances of a PhD finding a faculty job:
Since 1982, almost 800,000 PhDs were awarded in science and engineering fields, whereas only about 100,000 academic faculty positions were created in those fields within the same time frame. The number of S&E PhDs awarded annually has also increased over this time frame, from ~19,000 in 1982 to ~36,000 in 2011. The number of faculty positions created each year, however, has not changed, with roughly 3,000 new positions created annually.(via Javier Omar Garcia)
Sometimes good judgment can compel us to act illegally. Should a self-driving vehicle get to make that same decision?
‘A Ruby gem providing “time travel” and “time freezing” capabilities, making it dead simple to test time-dependent code. It provides a unified method to mock Time.now, Date.today, and DateTime.now in a single call.’ This is about the nicest mock-time library I’ve found so far. (via Ben)
this is brilliant
by Chris Newcombe, an AWS principal engineer. Several Amazonians sharing their results in simulating tricky distributed-systems problems using formal methods
However, the gold standard for database benchmarking is to test the performance of a system on the real production workload, since synthetic benchmarks often don’t exercise systems in the same way. When making decisions about a significant component of Facebook’s infrastructure, we need to understand how a database system will really perform in Facebook’s production workload. [....] LinkBench addresses these needs by replicating the data model, graph structure, and request mix of our MySQL social graph workload.Mentioned in a presentation from Peter Bailis, http://www.hpts.ws/papers/2013/bailis-hpts-2013.pdf
from the Percona toolkit. ‘Conveniently summarizes the status and configuration of a server. It is not a tuning tool or diagnosis tool. It produces a report that is easy to diff and can be pasted into emails without losing the formatting. This tool works well on many types of Unix systems.’ — summarises OOM history, top, netstat connection table, interface stats, network config, RAID, LVM, disks, inodes, disk scheduling, mounts, memory, processors, and CPU.
What they discovered is that when the volunteers cut back from seven-and-a-half to six-and-a-half hours’ sleep a night, genes that are associated with processes like inflammation, immune response and response to stress became more active. The team also saw increases in the activity of genes associated with diabetes and risk of cancer. The reverse happened when the volunteers added an hour of sleep.
some great phone cases from an Irish company, with nifty art by Irish illustrators and artists including Fatti Burke and Chris Judge
Interesting empirical results using JDK 7u21:
Full GC duration depends on the number of objects allocated and the locality of their references. It does not depend that much on actual heap size.Reference locality has a surprisingly high effect.
Occupy.here began two years ago as an experiment for the encampment at Zuccotti Park. It was a wifi router hacked to run OpenWrt Linux (an operating system mostly used for computer networking) and a small “captive portal” website. When users joined the wifi network and attempted to load any URL, they were redirected to http://occupy.here. The web software offered up a simple BBS-style message board providing its users with a space to share messages and files.Nifty project from Dan Phiffer.
Mark Jeftovic is on fire after receiving yet another “take down this domain or else” mail from the City of London police:
We have an obligation to our customers and we are bound by our Registrar Accreditation Agreements not to make arbitrary changes to our customers settings without a valid FOA (Form of Authorization). To supersede that we need a legal basis. To get a legal basis something has to happen in court. [...] What gets me about all of this is that the largest, most egregious perpetrators of online criminal activity right now are our own governments, spying on their own citizens, illegally wiretapping our own private communications and nobody cares, nobody will answer for it, it’s just an out-of-scope conversation that is expected to blend into the overall background malaise of our ever increasing serfdom. If I can’t make various governments and law enforcement agencies get warrants or court orders before they crack my private communications then I can at least require a court order before I takedown my own customer.
The problem with software patents, part XVII.
So you have a situation where even when the original patent holder donated the patent for “the public good,” sooner or later, an obnoxious patent troll like IV comes along and turns it into a weapon. Again: AmEx patented those little numbers on your credit card, and then for the good of the industry and consumer protection donated the patent to a non-profit, who promised not to enforce the patent against banks… and then proceeded to sell the patent to Intellectual Ventures who is now suing banks over it.
holy moly. This is some heavily-optimized mechanical-sympathy Java code. By using a sparse data structure, cache-aligned fields, and wait-free low-level CAS concurrency primitives via sun.misc.Unsafe, a single-producer/single-consumer queue implementation goes pretty damn fast compared to the current state of the art
interesting new distributed atomic transaction algorithm from Peter Bailis
John H. Gass hadn’t had a traffic ticket in years, so the Natick resident was surprised this spring when he received a letter from the Massachusetts Registry of Motor Vehicles informing him to cease driving because his license had been revoked. [...] After frantic calls and a hearing with Registry officials, Gass learned the problem: An antiterrorism computerized facial recognition system that scans a database of millions of state driver’s license images had picked his as a possible fraud. “We send out 1,500 suspension letters every day,” said Registrar Rachel Kaprielian. [...] “There are mistakes that can be made.”
Massachusetts bought the system with a $1.5 million grant from the Department of Homeland Security. At least 34 states use such systems, which law enforcement officials say help prevent identity theft and ID fraud.
In my opinion, this kind of thing — trial by inaccurate, false-positive-prone algorithm, is one of the most worrying things about the post-PRISM world.
When we created SpamAssassin, we were well aware of the risk of automated misclassification. Any machine-learning classifier will always make mistakes. The key is to carefully calibrate the expected false-positive/false-negative ratio so that the negative side-effects of a misclassification corresponds to the expected rate.
These anti-terrorism machine learning systems are calibrated to catch as many potential cases as possible, but by aiming to reduce false negatives to this degree, they become wildly prone to false positives. And when they’re applied as a dragnet across all citizens’ interactions with the state — or even in the case of PRISM, all citizens’ interactions that can be surveilled en masse — it’s going to create buckets of bureaucratic false-positive horror stories, as random innocent citizens are incorrectly tagged as criminals due to software bugs and poor calibration.
Nifty new feature — if a request takes over the 99th percentile for requests to that server, it’ll be repeated against another replica. Unnecessary for Voldemort, of course, which queries all replicas anyway!
As part of the Turmoil system, the NSA places secret servers, codenamed Quantum, at key places on the internet backbone. This placement ensures that they can react faster than other websites can. By exploiting that speed difference, these servers can impersonate a visited website to the target before the legitimate website can respond, thereby tricking the target’s browser to visit a Foxacid server.whoa, I missed this before.
By modifying the User-Agent: header string, each HTTP transaction is “stained” to allow tracking. huh
ugh, this is a mess. Docker, automate this crap
Well-written, comprehensive writeup of the Silk Road takedown, and the libertarian craziness of Ross William Ulbricht, it’s alleged owner and operator
In Kaspersky’s view, patent trolls are no better than the extortionists who cropped up in Russia after the fall of the Soviet Union, when crime ran rampant. Kaspersky saw more and more people becoming victims of various extortion schemes. US patent trolls seemed very similar. “Kaspersky’s view was that paying patent trolls was like paying a protection racket,” said Kniser. He wasn’t going to do it.yay! pity it didn’t manage to establish precedent, though. But go Kaspersky!
good comments from Sergio, particularly about the scalability of the single transactor in the Datomic architecture. I agree it’s a worrying design flaw
Excited! one commenter claims a paperback of the new edition of Luigi Serafini’s masterwork should cost about $75 when it comes out in a couple of months. sign me up, this is an amazing work
When the Guardian offered John Lanchester access to the GCHQ files, the journalist and novelist was initially unconvinced. But what the papers told him was alarming: that Britain is sliding towards an entirely new kind of surveillance society
The enhancement in performance was achieved by providing a fast-path where trades are executed directly by the FPGA under the control of trigger rules processed by the x86 based functions. The latency is reduced further by two additional techniques in the FPGA – inline parsing and pre-emption. As market data enters the switch, the Ethernet frame is parsed serially as bits arrive, allowing partial information to be extracted and matched before the whole frame has been received. Then, instead of waiting until the end of a potential triggering input packet, pre-emption is used to start sending the overhead part of a response which contains the Ethernet, IP, TCP and FIX headers. This allows completion of an outgoing order almost immediately after the end of the triggering market feed packet.Insane stuff. (Via Martin Thompson)
Summary: poor reliability, better latencies, and cheaper (!)
Interviews with 2 New York bike thieves (one bottom feeder, one professional), reviewing the current batch of bicycle locks. Summary: U-locks are good, when used correctly, particularly the Kryptonite New York Lock ($80). On the other hand, Dublin’s recent spate of thefts are largely driven by wide availability of battery-powered angle grinders (thanks Lidl!), which, according to this article, are relatively quiet and extremely fast. :(
I could see some value, perhaps, in a tablet that I share with my wife, where each of us have our own accounts, with independent configurations, apps, and settings. We could each conveniently identify ourselves by our fingerprint. But biometrics cannot, and absolutely must not, be used to authenticate an identity. For authentication, you need a password or passphrase. Something that can be independently chosen, changed, and rotated. [...] Once your fingerprint is compromised (and, yes, it almost certainly already is, if you’ve crossed an international border or registered for a driver’s license in most US states), how do you change it? Are you starting to see why this is a really bad idea?
This is a pretty good summary of the salient points from the criminal complaint against Ross William Ulbricht — I’d say it’s pretty bad news for any users of the dodgy site, particularly given this:
“During the 60-day period from May 24, 2013 to July 23, 2013, there were approximately 1,217,218 communications sent between Silk Road users through Silk Road’s private-message system.”According to the complaint, those are now in the FBI’s hands — likely unencrypted.
ouch. some serious slagging here, along with taco science. (BTW we have the same problem with carne asada in Ireland, our taquerias use the cheater method too, sadly)
Levison lost [in secret court against the government's order]. In a work-around, Levison complied the next day by turning over the private SSL keys as an 11 page printout in 4-point type. The government called the printout “illegible” and the court ordered Levison to provide a more useful electronic copy.Nice try though! Bottom line is they demanded the SSL private key. (via Waxy)
the fantastic French kids’ site is now crowdfunding new work — first off being a German Alphabet part of the site. My kids love their stuff, so — bonne chance!
An amazing hack. ‘Air Miles are awesome, they can be used to score free flights, hotel stays and if you’re really lucky, the scorn and hatred of everyone you come in contact with who has to pay full price when they travel. The king of all virtually free travelers is one David Phillips, a civil engineer who teaches at the University of California, Davis. David came to the attention of the wider media when he managed to convert about 12,150 cups of Healthy Choice chocolate pudding [costing $3000] into over a million Air Miles. Ever since, David and his entire family have been travelling the world for next to nothing.’ (via al3xandru)
An adventure that takes you through several popular Java language features and shows how they compile to bytecode and eventually JIT to assembly code.
Applied formal methods in order to test distributed systems — specifically GlusterFS:
I’ll use an example from my own recent experience. I’m developing a new kind of replication for GlusterFS. To make sure the protocol behaves correctly even across multiple failures, I developed a Murphi model for it. [...] I added a third failure [to the simulated model]. I didn’t expect a three-node system to continue working if more than one of those were concurrent (the model allows the failures to be any mix of sequential and concurrent), but I expected it to fail cleanly without reaching an invalid state. Surprise! It managed to produce a case where a reader can observe values that go back in time. This might not make much sense without knowing the protocol involved, but it might give some idea of the crazy conditions a model checker will find that you couldn’t possibly have considered. [...] So now I have a bug to fix, and that’s a good thing. Clearly, it involves a very specific set of ill-timed reads, writes, and failures. Could I have found it by inspection or ad-hoc analysis? Hell, no. Could I have found it by testing on live systems? Maybe, eventually, but it probably would have taken months for this particular combination to occur on its own. Forcing it to occur would require a lot of extra code, plus an exerciser that would amount to a model checker running 100x slower across machines than Murphi does. With enough real deployments over enough time it would have happened, but the only feasible way to prevent that was with model checking. These are exactly the kinds of bugs that are hardest to fix in the field, and that make users distrust distributed systems, so those of us who build such systems should use every tool at our disposal to avoid them.
ie. “fear of small, clustered holes”. Sounds like it’s not so much a “phobia” as some kind of innate, visceral disgust response; I get it. ‘As for who actually made the word up, that distinction probably belongs to a blogger in Ireland named Louise, Andrews says. According to an archived Geocities page, Louise settled on “trypophobia” (Greek for “boring holes” + “fear”) after corresponding with a representative at the Oxford English Dictionary. Louise, Andrews and trypophobia Facebook group members have petitioned the dictionary to include the word. The term will need to be used for years and have multiple petitions and scholarly references before the dictionary accepts it, Andrews says. I, for one, would prefer to forget about it forever.’
“We think that everyone has trypophobic tendencies even though they may not be aware of it,” said Dr Cole. “We found that people who don’t have the phobia still rate trypophobic images as less comfortable to look at than other images. It backs up the theory that we are set-up to be fearful of things which hurt us in our evolutionary past. We have an innate predisposition to be wary of things that can harm us.”
This is cool. Deploy Docker container images onto a Mesos cluster: key point, in the description of the Redis example: ‘there’s no need to install Redis or its supporting libraries on your Mesos hosts.’
Aphyr takes a look at Kafka 0.8′s replication with the Jepsen test suite. It doesn’t go great. Jay Kreps responds here: http://blog.empathybox.com/post/62279088548/a-few-notes-on-kafka-and-jepsen
A book published during the presidency of Chester A. Arthur has a greater chance of being in print today than one published during the time of Reagan.
This is not a gently sloping downward curve. Publishers seem unwilling to sell their books on Amazon for more than a few years after their initial publication. The data suggest that publishing business models make books disappear fairly shortly after their publication and long before they are scheduled to fall into the public domain. Copyright law then deters their reappearance as long as they are owned. On the left side of the graph before 1920, the decline presents a more gentle time-sensitive downward sloping curve.
Curated dissociated text. That’s great
During the first semester of my daughter’s junior/senior year, she took her first programming class. She knew I’d be thrilled, but she did it anyway. When my daughter got home from the first day of the semester, I asked her about the class. “Well, I’m the only girl in class,” she said. Fortunately, that didn’t bother her, and she even liked joking around with the guys in class. My daughter said that you noticed and apologized to her because she was the only girl in class. And when the lessons started (Visual Basic? Seriously??), my daughter flew through the assigments. After she finished, she’d help classmates who were behind or struggling in class. Over the next few weeks, things went downhill. While I was attending SC ’12 in Salt Lake City last November, my daughter emailed to tell me that the boys in her class were harassing her. “They told me to get in the kitchen and make them sandwiches,” she said. I was painfully reminded of the anonymous men boys who left comments on a Linux Pro Magazine blog post I wrote a few years ago, saying the exact same thing.I am sick to death of this ‘brogrammer’ bullshit.
The bat had the misfortune of being on display in the shop front of Elvery’s store on O’Connell Street, then Sackville Street, during the Easter Rising. J.W. Elvery & Co. was Ireland’s oldest sports store, specialising in sporting goods and waterproofed wear, with branches in Dublin, Cork (Patrick Street) and London (Conduit Street). [...] Its location, about one block from the GPO, meant it was in the middle of the cross-fire and general destruction of the main street.
empirical BigTable and GFS failure numbers from Google are orders of magnitude higher than naïve independent-failure models. (via kragen)
YES. (via Des Traynor)
iOS 7 includes — and uses — multipath TCP, right now for device-to-Siri communications.
MPTCP is a TCP extension that enables the simultaneous use of several IP addresses or interfaces. Existing applications – completely unmodified — see what appears to be a standard TCP interface. But under the covers, MPTCP is spreading the connection’s data across several subflows, sending it over the least congested paths.
‘a client-side database that supports the complete DynamoDB API, but doesn’t manipulate any tables or data in DynamoDB itself. You can write code while sitting in a tree, on the beach, or in the desert. When you are ready to deploy your application, you simply instruct it to connect to the actual DynamoDB endpoint. No other modifications will be needed.’ This is good — an in-memory data store for integration testing is absolutely vital for production usage. (Voldemort does this well, for example.)
‘Fancy algorithms are slow when n is small, and n is usually small.’ — Rob PikeBeen there, bought the t-shirt ;)
good slides explaining the Raft protocol
In case you’re missing the story here, Dual_EC_DRBG (which I wrote about yesterday) is the random number generator voted most likely to be backdoored by the NSA. The story here is that — despite many valid concerns about this generator — RSA went ahead and made it the default generator used for all cryptography in its flagship cryptography library. The implications for RSA and RSA-based products are staggering. In a modestly bad but by no means worst case, the NSA may be able to intercept SSL/TLS connections made by products implemented with BSafe.
This is exactly my problem with Cucumber and similar BDD test frameworks.
When I write a Cucumber feature, I have to write the Gherkin that describes the acceptance criteria, and the Ruby code that implements the step definitions. Since the code to implement the step definitions is just normal RSpec (or whichever testing library you use), if someone else is writing the Gherkin, the amount of setup to create a working test should be about the same. So you’re only breaking even! However, I don’t believe that it would really be breaking even. Cucumber adds another layer of indirection on top of your tests. When I’m trying to see why a specific scenario is failing, first I need to find the step that is failing. Since these steps are defined with regular expressions, I have to grep for the step definition.
This is watching your sharp, witty father start telling old fart jokes as his mind slows down. And as much as the internet is habituated to defending GTA as “satire,” what is it satirizing, if everything is either sad or awful? Where is the “satire” when the awful parts no longer seem edgy or provocative, just attempts at catch-all “offense” that aren’t honed enough to even connect? Here’s a series that has been creating real, meaningful friction with conventional entertainment for as long as I can remember, and rather than push the envelope by creating new kinds of monsters, it’s reciting the same old gangland fantasies, like a college boy who can’t stop staring at the Godfather II poster on his wall, talking about how he’s gonna be a big Hollywood director in between bong rips. You call the trading index BAWSAQ? Oh, bro, you’re so funny, you’re gonna be huge.
“We hope that this finally puts to rest the illusions people have about fingerprint biometrics. It is plain stupid to use something that you can´t change and that you leave everywhere every day as a security token”, said Frank Rieger, spokesperson of the CCC. “The public should no longer be fooled by the biometrics industry with false security claims. Biometrics is fundamentally a technology designed for oppression and control, not for securing everyday device access.” iPhone users should avoid protecting sensitive data with their precious biometric fingerprint not only because it can be easily faked, as demonstrated by the CCC team. Also, you can easily be forced to unlock your phone against your will when being arrested. Forcing you to give up your (hopefully long) passcode is much harder under most jurisdictions than just casually swiping your phone over your handcuffed hands.
OfCom has published a report on online piracy, which found that the practice is becoming less common and that pirates tend to spend more on legitimate content than non-pirates. The research, which was not funded by the entertainment industry, was conducted by Kantar Media among 21,474 participants and took place in 2012 across four separate stages. Over that time, the ratio of legal to illegal content fell — confirming a suspected trend as legal streaming options became more available. It also confirmed another suspicion — that a relatively small number of web users are responsible for most piracy. In OfCom’s data, just two percent of users conducted three quarters of all piracy. Ofcom described piracy as “a minority activity”. Of those surveyed, 58 percent accessed music, movie or TV content online, while 17 percent accessed illegal content sources. Those who admitted pirating content spent on average £26 every three months on legitimate content, set against an average spend of £16 among non-pirates.
The excellent Trouble Brewing are looking for investors
‘A counting Bloom filter (CBF) generalizes a Bloom filter data structure so as to allow membership queries on a set that can be changing dynamically via insertions and deletions. As with a Bloom filter, a CBF obtains space savings by allowing false positives. We provide a simple hashing-based alternative based on d-left hashing called a d-left CBF (dlCBF). The dlCBF offers the same functionality as a CBF, but uses less space, generally saving a factor of two or more. We describe the construction of dlCBFs, provide an analysis, and demonstrate their effectiveness experimentally’
In a talk about a neat software component he designed, Bruce Haddon observed that there is no way that the final structure and algorithmic behavior of this component could have been predicted, designed, or otherwise anticipated. Haddon observed that computer science serves as a source of core ideas: it provides the data structures and algorithms that are the building blocks. Meanwhile, he views software engineering as a useful set of methods to help design reliable software without losing your mind. Yet he points out that neither captures the whole experience. That’s because much of the work is what Haddon calls hacking, but what others would call bricolage. Simply put, there is much trial and error: we put ideas to together and see where it goes.This is a great post, and I agree (broadly). IMO, most software engineering requires little CS, but there are occasional moments where a single significant aspect of a project requires a particular algorithm, and would be kludgy, hacky, or over-complex to solve without it.
I have come around to the view that the real core difficulty of [distributed] systems is operations, not architecture or design. Both are important but good operations can often work around the limitations of bad (or incomplete) software, but good software cannot run reliably with bad operations. This is quite different from the view of unbreakable, self-healing, self-operating systems that I see being pitched by the more enthusiastic NoSQL hypesters. Worse yet, you can’t easily buy good operations in the same way you can buy good software—you might be able to hire good people (if you can find them) but this is more than just people; it is practices, monitoring systems, configuration management, etc.
see also HN comments: https://news.ycombinator.com/item?id=6398650 , particularly davidmr’s great one:
I suppose all of this is to say that the amount of required parallelization of a problem isn’t necessarily related to the size of the problem set as is mentioned most in the article, but also the inherent CPU and IO characteristics of the problem. Some small problems are great for large-scale map-reduce clusters, some huge problems are horrible for even bigger-scale map-reduce clusters (think fluid dynamics or something that requires each subdivision of the problem space to communicate with its neighbors). I’ve had a quote printed on my door for years: Supercomputers are an expensive tool for turning CPU-bound problems into IO-bound problems.I love that quote!
Gilt ran a stress-test of Riak to replace Voldemort (I think) in a shadow stack, with good results:
Riak’s strong performance suggests that, should we pursue implementation, it will withstand our unique traffic needs and prove reliable. As for the Gilt-Basho team’s strong performance: It was amazing that we were able to accomplish so much in just a week’s time! Thanks again to Seth and Steve for making this possible.
wow this looks great.
The Long Dark is a thoughtful, first-person survival simulation that emphasizes quiet exploration in a stark, yet hauntingly beautiful, post-disaster setting. The breathtakingly picturesque Pacific Northwest frames the backdrop for the drama of The Long Dark.
“The key factor is the environment, whether you’re talking about humans or rats,” Dr. Hart said. “The rats that keep pressing the lever for cocaine are the ones who are stressed out because they’ve been raised in solitary conditions and have no other options. But when you enrich their environment, and give them access to sweets and let them play with other rats, they stop pressing the lever.”
featuring some mental pics of the “Information Dominance Center”, the Star Trek bridge which NSA chief Keith Alexander built with taxpayer money
Regardless of how we got here, the NSA can’t reform itself. Change cannot come from within; it has to come from above. It’s the job of government: of Congress, of the courts, and of the president. These are the people who have the ability to investigate how things became so bad, rein in the rogue agency, and establish new systems of transparency, oversight, and accountability. Any solution we devise will make the NSA less efficient at its eavesdropping job. That’s a trade-off we should be willing to make, just as we accept reduced police efficiency caused by requiring warrants for searches and warning suspects that they have the right to an attorney before answering police questions. We do this because we realize that a too-powerful police force is itself a danger, and we need to balance our need for public safety with our aversion of a police state.
Biometrics was rolled out for food distribution in order to cut down on fraud, but it’s now resulting in a subset of users being unable to authenticate:
The biometric authentication system installed at the PDS outlets fails to establish the identity of many genuine beneficiaries, mostly workers, as their daily grind in the agricultural fields, construction sites or as domestic help have eroded the lines on their thumb resulting in distorted impressions.
ha, this is very clever! If you have enough volume, this is a nice estimation algorithm to compute stream quantiles in very little RAM
Spam Arrest is a company that sells an anti-spam service. They attempted to sue some spammers and, as has been widely reported, lost badly. This case emphasizes three points that litigious antispammers seem not to grasp: Under CAN SPAM, a lot of spam is legal. Judges hate plaintiffs who try to be too clever, and hate sloppy preparation even more. Never, ever, file a spam suit in Seattle.
good data points, but could do with latency percentiles
Very interesting research into poverty and scarcity, in the Washington Post:
The scarcity trap captures this notion we see again and again in many domains. When people have very little, they undertake behaviors that maintain or reinforce their future disadvantage. If you have very little, you often behave in such a way so that you’ll have little in the future. In economics, people talk about the poverty trap. We’re generalizing that, saying this happens a lot, and we’ve experienced it.
Yet again, security software fails on packaging and UI. via Tony Finch
At one point, Hayden expressed a distaste for online anonymity, saying “The problem I have with the Internet is that it’s anonymous.” But he noted, there is a struggle over that issue even inside government. The issue came to a head during the Arab Spring movement when the State Department was funding technology [presumably Tor?] to protect the anonymity of activists so governments could not track down or repress their voices. “We have a very difficult time with this,” Hayden said. He then asked, “is our vision of the World Wide Web the global digital commons – at this point you should see butterflies flying here and soft background meadow-like music — or a global free fire zone?” Given that Hayden also compared the Internet to the wild west and Somalia, Hayden clearly leans toward the “global free fire zone” vision of the Internet.well, that’s a good analogy for where we’re going — a global free-fire zone.
When we asked Sallner to quantify the scale of the migration he said, “They’re moving it all. Everything they have. All of the MySQL servers are moving to MariaDB, as far as I understand.” By moving to MariaDB, Google can free itself of any dependence on technology dictated by Oracle – a company whose motivations are unclear, and whose track record for working with the wider technology community is dicey, to say the least. Oracle has controlled MySQL since its acquisition of Sun in 2010, and the key InnoDB storage engine since it got ahold of Innobase in 2005. [...] We asked Cole why Google would shift from MySQL to MariaDB, and what the key technical differences between the systems were. “From my perspective, they’re more or less equivalent other than if you look at specific features and how they implement them,” Cole said, speaking in a personal capacity and not on behalf of Google. “Ideologically there are lots of differences.”So — AWS, when will RDS offer MariaDB as an option?
The code’s behavior, and the command-and-control server’s Virginia placement, is also consistent with what’s known about the FBI’s “computer and internet protocol address verifier,” or CIPAV, the law enforcement spyware first reported by WIRED in 2007. Court documents and FBI files released under the FOIA have described the CIPAV as software the FBI can deliver through a browser exploit to gather information from the target’s machine and send it to an FBI server in Virginia. The FBI has been using the CIPAV since 2002 against hackers, online sexual predators, extortionists, and others, primarily to identify suspects who are disguising their location using proxy servers or anonymity services, like Tor. Prior to the Freedom Hosting attack, the code had been used sparingly, which kept it from leaking out and being analyzed.
lots more detail on the new “Java Mission Control” feature in Hotspot 7u40 JVMs, and how to use it to start and stop profiling in a live, production JVM from a separate “jcmd” command-line client. If the overhead is small, this could be really neat — turn on profiling for 1 minute every hour on a single instance, and collect realtime production profile data on an automated basis for post-facto analysis if required
Modern telecommunications technology implied the development of modern telecommunications surveillance, because it moved the scope of action from the physical world (where intelligence, generally seen as part of the military mission, had acted) to the virtual world—including the scope of those actions that could threaten state power. While the public line may have been, as US Secretary of State Henry Stimson said in 1929, “gentlemen do not open each other’s mail”, you can bet that they always did keep a keen eye on the comings and goings of each other’s shipping traffic. The real reason that surveillance in the context of state intelligence was limited until recently was because it was too expensive, and it was too expensive for everyone. The Westphalian compromise demands equality of agency as tied to territory. As soon as one side gains a significant advantage, the structure of sovereignty itself is threatened at a conceptual level?—?hence Oppenheimer as the death of any hope of international rule of law. Once surveillance became cheap enough, all states were (and will increasingly be) forced to attempt it at scale, as a reaction to this pernicious efficiency. The US may be ahead of the game now, but Moore’s law and productization will work their magic here.
Bit of detail into Twitter’s TSD metric store.
There are separate online clusters for different data sets: application and operating system metrics, performance critical write-time aggregates, long term archives, and temporal indexes. A typical production instance of the time series database is based on four distinct Cassandra clusters, each responsible for a different dimension (real-time, historical, aggregate, index) due to different performance constraints. These clusters are amongst the largest Cassandra clusters deployed in production today and account for over 500 million individual metric writes per minute. Archival data is stored at a lower resolution for trending and long term analysis, whereas higher resolution data is periodically expired. Aggregation is generally performed at write-time to avoid extra storage operations for metrics that are expected to be immediately consumed. Indexing occurs along several dimensions–service, source, and metric names–to give users some flexibility in finding relevant data.
I didn’t clearly explain that there’s an enormous continuum between, on the one hand, a full break of RSA or Diffie-Hellman (which still seems extremely unlikely to me), and on the other, “pure side-channel attacks” involving no new cryptanalytic ideas. Along that continuum, there are many plausible places where the NSA might be. For example, imagine that they had a combination of side-channel attacks, novel algorithmic advances, and sheer computing power that enabled them to factor, let’s say, ten 2048-bit RSA keys every year. In such a case, it would still make perfect sense that they’d want to insert backdoors into software, sneak vulnerabilities into the standards, and do whatever else it took to minimize their need to resort to such expensive attacks. But the possibility of number-theoretic advances well beyond what the open world knows certainly wouldn’t be ruled out. Also, as Schneier has emphasized, the fact that NSA has been aggressively pushing elliptic-curve cryptography in recent years invites the obvious speculation that they know something about ECC that the rest of us don’t.
Built into the HotSpot JVM [in JDK version 7u40] is something called the Java Flight Recorder. It records a lot of information about/from the JVM runtime, and can be thought of as similar to the Data Flight Recorders you find in modern airplanes. You normally use the Flight Recorder to find out what was happening in your JVM when something went wrong, but it is also a pretty awesome tool for production time profiling. Since Mission Control (using the default templates) normally don’t cause more than a per cent overhead, you can use it on your production server.I’m intrigued by the idea of always-on profiling in production. This could be cool.
One of the US agents’ tools is the use of backup files established by smartphones. According to one NSA document, these files contain the kind of information that is of particular interest to analysts, such as lists of contacts, call logs and drafts of text messages. To sort out such data, the analysts don’t even require access to the iPhone itself, the document indicates. The department merely needs to infiltrate the target’s computer, with which the smartphone is synchronized, in advance. Under the heading “iPhone capability,” the NSA specialists list the kinds of data they can analyze in these cases. The document notes that there are small NSA programs, known as “scripts,” that can perform surveillance on 38 different features of the iPhone 3 and 4 operating systems. They include the mapping feature, voicemail and photos, as well as the Google Earth, Facebook and Yahoo Messenger applications.and, of course, the alternative means of backup is iCloud…. wonder how secure those backups are.
Boost ASIO at the front end (!), Kafka 0.8, Storm, and ElasticSearch
The inconvenience of extra passenger screening and added costs at airports after 9/11 cause many short-haul passengers to drive to their destination instead, and, since airline travel is far safer than car travel, this has led to an increase of 500 U.S. traffic fatalities per year. Using DHS-mandated value of statistical life at $6.5 million, this equates to a loss of $3.2 billion per year, or $32 billion over the period 2002 to 2011 (Blalock et al. 2007).
The debate has been stifled in Britain more successfully than anywhere else in the free world and, astonishingly, this has been with the compliance of a media and public that regard their attachment to liberty to be a matter of genetic inheritance. So maybe it is best for me to accept that the BBC, together with most of the newspapers, has moved with society, leaving me behind with a few old privacy-loving codgers, wondering about the cause of this shift in attitudes. Is it simply the fear of terror and paedophiles? Are we so overwhelmed by the power of the surveillance agencies that we feel we can’t do anything? Or is it that we have forgotten how precious and rare truly free societies are in history?
Some great street art from Brighton, via Darach Ennis
In the fight against the unauthorised sharing of copyright protected material, aka piracy, Dutch Internet Service Providers have been summoned by courts to block their subscribers’ access to The Pirate Bay (TPB) and related sites. This paper studies the effectiveness of this approach towards online copyright enforcement, using both a consumer survey and a newly developed non-infringing technology for BitTorrent monitoring. While a small group of respondents download less from illegal sources or claim to have stopped, and a small but significant effect is found on the distribution of Dutch peers, no lasting net impact is found on the percentage of the Dutch population downloading from illegal sources.
Bruce Schneier’s suggestions:
Assuming the hypothetical NSA breakthroughs don’t totally break public-cryptography — and that’s a very reasonable assumption — it’s pretty easy to stay a few steps ahead of the NSA by using ever-longer keys. We’re already trying to phase out 1024-bit RSA keys in favor of 2048-bit keys. Perhaps we need to jump even further ahead and consider 3072-bit keys. And maybe we should be even more paranoid about elliptic curves and use key lengths above 500 bits. One last blue-sky possibility: a quantum computer. Quantum computers are still toys in the academic world, but have the theoretical ability to quickly break common public-key algorithms — regardless of key length — and to effectively halve the key length of any symmetric algorithm. I think it extraordinarily unlikely that the NSA has built a quantum computer capable of performing the magnitude of calculation necessary to do this, but it’s possible. The defense is easy, if annoying: stick with symmetric cryptography based on shared secrets, and use 256-bit keys.
a pretty good description of the process of adding service metrics to a Django webapp using graphite and statsd. Bookmarking mainly for the great real-time graphing hack at the end…
a nifty hack.
Recently I have been banging my head trying to import a ton of OCR acquired data expressed in tabular form. I think I have come up with a neat approach using probabilistic reasoning combined with mixed integer programming. The method is pretty robust to all sorts of real world issues. In particular, the method leverages topological understanding of tables, encodes it declaratively into a mixed integer/linear program, and integrates weak probabilistic signals to classify the whole table in one go (at sub second speeds). This method can be used for any kind of classification where you have strong logical constraints but noisy data.(via proggit)
‘Plugin to make highly interactive graphite graph objects ((i.e. graphs where you can interactively toggle on/off individual series, inspect datapoints, zoom in realtime, etc) Supports Flot (canvas), Rickshaw (svg) and standard graphite png images (in case you’re nostalgic and don’t like interactivity).’
‘The implementation of the concurrency primitive LockSupport.parkNanos(), the function that controls *every* concurrency primitive on the JVM, is flawed, and any NTP sync, or system time change, can potentially break it with unexpected results across the board when running a 64bit JVM on Linux 64bit.’ Basically, LockSupport.parkNanos() calls pthread_cond_timedwait() using a CLOCK_REALTIME instead of CLOCK_MONOTONIC. ‘tinker step 0′ in ntp.conf may be a viable workaround.
The new Snowden revelations are explosive. Basically, the NSA is able to decrypt most of the Internet. They’re doing it primarily by cheating, not by mathematics. It’s joint reporting between the Guardian, the New York Times, and ProPublica. I have been working with Glenn Greenwald on the Snowden documents, and I have seen a lot of them. These are my two essays on today’s revelations. Remember this: The math is good, but math has no agency. Code has agency, and the code has been subverted.
summary – turn off DASH (Dynamic adaptive streaming) using a userscript.
‘This paper and talk was given by the LinkedIn Voldemort Team at the Workshop on Big Data Benchmarking (WBDB May 2012).’
With SSD, we find that garbage collection will become a very significant bottleneck, especially for systems which have little control over the storage layer and rely on Java memory management. Big heapsizes make the cost of garbage collection expensive, especially the single threaded CMS Initial mark. We believe that data systems must revisit their caching strategies with SSDs. In this regard, SSD has provided an efficient solution for handling fragmentation and moving towards predictable multitenancy.
Before Summingbird at Twitter, users that wanted to write production streaming aggregations would typically write their logic using a Hadoop DSL like Pig or Scalding. These tools offered nice distributed system abstractions: Pig resembled familiar SQL, while Scalding, like Summingbird, mimics the Scala collections API. By running these jobs on some regular schedule (typically hourly or daily), users could build time series dashboards with very reliable error bounds at the unfortunate cost of high latency. While using Hadoop for these types of loads is effective, Twitter is about real-time and we needed a general system to deliver data in seconds, not hours. Twitter’s release of Storm made it easy to process data with very low latencies by sacrificing Hadoop’s fault tolerant guarantees. However, we soon realized that running a fully real-time system on Storm was quite difficult for two main reasons: Recomputation over months of historical logs must be coordinated with Hadoop or streamed through Storm with a custom log loading mechanism; Storm is focused on message passing and random-write databases are harder to maintain. The types of aggregations one can perform in Storm are very similar to what’s possible in Hadoop, but the system issues are very different. Summingbird began as an investigation into a hybrid system that could run a streaming aggregation in both Hadoop and Storm, as well as merge automatically without special consideration of the job author. The hybrid model allows most data to be processed by Hadoop and served out of a read-only store. Only data that Hadoop hasn’t yet been able to process (data that falls within the latency window) would be served out of a datastore populated in real-time by Storm. But the error of the real-time layer is bounded, as Hadoop will eventually get around to processing the same data and will smooth out any error introduced. This hybrid model is appealing because you get well understood, transactional behavior from Hadoop, and up to the second additions from Storm. Despite the appeal, the hybrid approach has the following practical problems: Two sets of aggregation logic have to be kept in sync in two different systems; Keys and values must be serialized consistently between each system and the client. The client is responsible for reading from both datastores, performing a final aggregation and serving the combined results Summingbird was developed to provide a general solution to these problems.Very interesting stuff. I’m particularly interested in the design constraints they’ve chosen to impose to achieve this — data formats which require associative merging in particular.
We mentioned above that pop-up spaces have become popular across Europe because they allow developers and city councils to harness urban creativity in order to drive up real estate prices without ceding control of a given site. Those who produce the space through hard work, collaboration and passion move on, making way for property development and speculation. The international research in this area is very clear on this point and it has been documented in places from Lower-East Side Manhattan to Berlin’s Kreuzberg. Most perversely, increased property prices make it even more difficult for creativity to flourish in a given area and end up driving out long-term working class communities, migrants and young people. But what can we do? If every attempt we make to make our city a better place simply ends up being captured in the calculations of real estate players, surely the situation is hopeless? Is it better, then, to do nothing? We don’t think it is better to do nothing and, like Upstart, we still believe we can find a way together through experimentation and collaboration. However, this means questioning, reflecting on and publicly discussing the relationship between our efforts to make a city more after our hearts desire and the process of gentrification. As noted above, this is especially the case with pop-up spaces given their temporary nature. It is really necessary that we think about how to make sure our activities don’t contribute to gentrification in the long term, but instead benefit the city as a whole. We certainly don’t have the solutions, but if we sweep these awkward questions under the carpet we risk contributing to the very forces we want to challenge and alienating those who will perceive us as the ‘front-line’ of gentrification.
Disruptor: decimating P99s since 2011
I love these.
Photographic prints are great because they don’t need power to be displayed. They are more or less permanent. Videos are great because they record a sequence of time which shows reality almost like how we experience. Is it possible to combine the two? And not via long exposure photography where often details are lost from motion. So I played around with the tools of digital photography and post processing to give you this series: Time is a dimension. This series of images are mostly landscapes, seascapes and cityscapes, and they are a single composite made from sequences that span 2-4 hours, mostly of sunrises and sunsets. The basic structure of a landscape is present in every piece. But each panel or concentric layer shows a different slice of time, which is related to the adjacent panel/layer. The transition from daytime to night is gradual and noticeable in every piece, but would not be something you expect to see in a still image.
‘Visualizations that make no sense.’ Some of these are unintentional comedy gold — pie charts feature heavily, of course. (via Des Traynor)
Peter Bailis with an interesting distributed-storage atomicity algorithm for performing multi-record transactional updates
good background on Github’s Elasticsearch scaling efforts. Some rather horrific split-brain problems under load, and crashes due to OpenJDK bugs (sounds like OpenJDK *still* isn’t ready for production). painful
Summarising a very shoddy tale from our paper of record.
I don’t know what happened here. I don’t know whether there ever was a woman who met the description given by the Irish Times who suffered a medical crisis during pregnancy. I don’t know why a group of men in positions of authority in the Irish Times decided that, if there was such a woman, they had any right to tell the rest of the country about her experiences. I don’t know why, when they discovered that a mistake had been made in the one legal fact used to justify that decision they didn’t immediately apologise. And I don’t know what happened between the 23rd August 2013 and 31st August 2013 to prompt them to print a shoulder shrugging ‘acceptance’ that the case ‘hadn’t happened’ and limit the paper’s apology to an institution, as opposed to its readers. But, from what I’ve seen this week, I do know one thing. Whatever questions readers might have, The Irish Times isn’t interested in giving them any answers.
Rackspace’s large-scale TSD storage system, built on Cassandra, Java, ASL2
Amazing reverse engineering.
In a hotel room in Texas, Clive Sinclair had a big problem. He wanted to sell a cheap scientific calculator that would grab the market from expensive calculators such as the popular HP-35. Hewlett-Packard had taken two years, 20 engineers, and a million dollars to design the HP-35, which used 5 complex chips and sold for $395. Sinclair’s partnership with calculator manufacturer Bowmar had gone nowhere. Now Texas Instruments offered him an inexpensive calculator chip that could barely do four-function math. Could he use this chip to build a $100 scientific calculator? Texas Instruments’ engineers said this was impossible – their chip only had 3 storage registers, no subroutine calls, and no storage for constants such as ?. The ROM storage in the calculator held only 320 instructions, just enough for basic arithmetic. How could they possibly squeeze any scientific functions into this chip? Fortunately Clive Sinclair, head of Sinclair Radionics, had a secret weapon – programming whiz and math PhD Nigel Searle. In a few days in Texas, they came up with new algorithms and wrote the code for the world’s first single-chip scientific calculator, somehow programming sine, cosine, tangent, arcsine, arccos, arctan, log, and exponentiation into the chip. The engineers at Texas Instruments were amazed. How did they do it? Up until now it’s been a mystery. But through reverse engineering, I’ve determined the exact algorithms and implemented a simulator that runs the calculator’s actual code. The reverse-engineered code along with my detailed comments is in the window below.
LOL MS. Sadly, this talk of “core competencies” and “visibility” is pretty reminiscent of Amazon’s review season, too:
This illustrated another problem with [stack ranking]: It destroyed trust between individual contributors and management, because the stack rank required that all lower-level managers systematically lie to their reports. Why? Because for years Microsoft did not admit the existence of the stack rank to nonmanagers. Knowledge of the process gradually leaked out, becoming a recurrent complaint on the much-loathed (by Microsoft) Mini-Microsoft blog, where a high-up Microsoft manager bitterly complained about organizational dysfunction and was joined in by a chorus of hundreds of employees. The stack rank finally made it into a Vanity Fair article in 2012, but for many years it was not common knowledge, inside or outside Microsoft. It was presented to the individual contributors as a system of objective assessment of “core competencies,” with each person being judged in isolation. When review time came, and programmers would fill out a short self-assessment talking about their achievements, strengths, and weaknesses, only some of them knew that their ratings had been more or less already foreordained at the stack rank. [...] If you did know about the stack rank, you weren’t supposed to admit it. So you went through the pageantry of the performance review anyway, arguing with your manager in the rhetoric of “core competencies.” The managers would respond in kind. Since the managers had little control over the actual score and attendant bonus and raise (if any), their job was to write a review to justify the stack rank in the language of absolute merit. (“Higher visibility” was always a good catch-all: Sure, you may be a great coder and work 80 hours a week, but not enough people have heard of you!)
This is hilarious. Quid pro quo!
Once he had set up the 0871 line, every time a bank, gas or electricity supplier asked him for his details online, he submitted it as his contact number. He added he was “very honest” and the companies did ask why he had a premium number. He told the programme he replied: “Because I’m getting annoyed with PPI phone calls when I’m trying to watch Coronation Street so I’d rather make 10p a minute.” He said almost all of the companies he dealt with were happy to use it and if they refused he asked them to email.
This is brilliant. Half of the office now wants prints.
Massive congratulations to Edge magazine. The stellar publication has been around for 20 years! To celebrate, their 258th issue comes in 20 different flavours, and one of those flavours includes the earthly overtones of both Minecraft and Dungeons & Dragons. Junkboy drew it, and I [Owen] worded it a few weeks ago.
Forecast.io are doing such a great job of applying modern machine-learning to traditional weather data. “Quicksilver” is their neural-net-adjusted global temperature geodata, and here’s how it’s built
from VLDB 2013:
MillWheel is a framework for building low-latency data-processing applications that is widely used at Google. Users specify a directed computation graph and application code for individual nodes, and the system manages persistent state and the continuous flow of records, all within the envelope of the framework’s fault-tolerance guarantees. This paper describes MillWheel’s programming model as well as its implementation. The case study of a continuous anomaly detector in use at Google serves to motivate how many of MillWheel’s features are used. MillWheel’s programming model provides a notion of logical time, making it simple to write time-based aggregations. MillWheel was designed from the outset with fault tolerance and scalability in mind. In practice, we find that MillWheel’s unique combination of scalability, fault tolerance, and a versatile programming model lends itself to a wide variety of problems at Google.
Süddeutsche Zeitung (SZ) had already revealed in late June that the British had access to the cable TAT-14, which connects Germany with the USA, UK, Denmark, France and the Netherlands. In addition to TAT-14, the other cables that GCHQ has access to include Atlantic Crossing 1, Circe North, Circe South, Flag Atlantic-1, Flag Europa-Asia, SeaMeWe-3 and SeaMeWe-4, Solas, UK France 3, UK Netherlands-14, Ulysses, Yellow and the Pan European Crossing.
This is amazing news. Paying attention, Sean Sherlock?
A major new patent bill, passed in a 117-4 vote by New Zealand’s Parliament after five years of debate, has banned software patents. The relevant clause of the patent bill actually states that a computer program is “not an invention.” Some have suggested that was a way to get around the wording of the TRIPS intellectual property treaty, which requires patents to be “available for any inventions, whether products or processes, in all fields of technology.” [...] One Member of Parliament who was deeply involved in the debate, Clare Curran, quoted several heads of software firms complaining about how the patenting process allowed “obvious things” to get patented and that “in general software patents are counter-productive.” Curran quoted one developer as saying, “It’s near impossible for software to be developed without breaching some of the hundreds of thousands of patents granted around the world for obvious work.” “These are the heavyweights of the new economy in software development,” said Curran. “These are the people that needed to be listened to, and thankfully, they were.”
Docker is to deployment as Git is to development. Developers are able to leverage Git’s performance and flexibility when building applications. Git encourages experiments and doesn’t punish you when things go wrong: start your experiments in a branch, if things fall down, just git rebase or git reset. It’s easy to start a branch and fast to push it. Docker encourages experimentation for operations. Containers start quickly. Building images is a snap. Using another images as a base image is easy. Deploying whole images is fast, and last but not least, it’s not painful to rollback. Fast + flexible = deployments are about to become a lot more enjoyable.
how LI solved a tricky graph-database-query latency problem with a set-cover algorithm
“I have been told that they cannot change your fundamental business practices,” said Callas, who unlike Levison was able to say SilentCircle has received no NSLs or court orders of any kind. “I presume that would mean things like getting SSL keys because that would mean they could impersonate your servers. That would be like setting up a store front that says your business name and putting [government agents] in your company uniforms.” Similarly, he added: “They cannot make changes to existing operating systems. They can’t make you change source code.” To which [Lavabit's] Levison replied: “That was always my understanding, too. That’s why this is so important. Like [Callas] at SilentCircle said, the assumption has been that the government can’t force us to change our business practices like that and compromise that information. Like I said, I don’t hold those beliefs anymore.”
An excellent post from Martin Thompson showing a new JSR166 concurrency primitive, StampedLock, compared against a number of alternatives in a simple microbenchmark. The most interesting thing for me is how much the lock-free, AtomicReference.compareAndSet()-based approach blows away all the lock-based approaches — even in the 1-reader-1-writer case. Its code is extremely simple, too: https://github.com/mjpt777/rw-concurrency/blob/master/src/LockFreeSpaceship.java
This is super-cool. ‘Network engineering no longer should be mundane tasks like conf, set interfaces fe-0/0/0 unit o family inet address 10.1.1.1/24. How does mindless CLI work translate to efficiently spent time ? What if you need to change 300 devices? What if you are writing it by hand? An error-prone waste of time. Juniper today announced Puppet support for their 12.2R3,5 JUNOS code. This is compatible with EX4200, EX4550, and QFX3500 switches. These are top end switches, but this start is directly aimed at their DC and enterprise devices. Initially, the manifest interactions offered are interface, layer 2 interface, vlan, port aggregation groups, and device names.’ Based on what I saw in the Network Automation team in Amazon, this is an amazing leap forward; it’d instantly render obsolete a bunch of horrific SSH-CLI automation cruft.
The future of the AWS command line tools is awscli, a single, unified, consistent command line tool that works with almost all of the AWS services. Here is a quick list of the services that awscli currently supports: Auto Scaling, CloudFormation, CloudSearch, CloudWatch, Data Pipeline, Direct Connect, DynamoDB, EC2, ElastiCache, Elastic Beanstalk, Elastic Transcoder, ELB, EMR, Identity and Access Management, Import/Export, OpsWorks, RDS, Redshift, Route 53, S3, SES, SNS, SQS, Storage Gateway, Security Token Service, Support API, SWF, VPC. Support for the following appears to be planned: CloudFront, Glacier, SimpleDB. The awscli software is being actively developed as an open source project on Github, with a lot of support from Amazon. You’ll note that the biggest contributors to awscli are Amazon employees with Mitch Garnaat leading. Mitch is also the author of boto, the amazing Python library for AWS.