Choco is [FOSS] dedicated to Constraint Programming. It is a Java library written under BSD license. It aims at describing hard combinatorial problems in the form of Constraint Satisfaction Problems and solving them with Constraint Programming techniques. The user models its problem in a declarative way by stating the set of constraints that need to be satisfied in every solution. Then, Choco solves the problem by alternating constraint filtering algorithms with a search mechanism. [...] Choco is among the fastest CP solvers on the market. In 2013 and 2014, Choco has been awarded many medals at the MiniZinc challenge that is the world-wide competition of constraint-programming solvers.
Shaun Of The Dead, Hot Fuzz, and The World’s End are a trilogy. I had no idea! (via David Malone)
Summary: PostgreSQL’s dreaded unpredictable “vacuum” GC
YAS3FS (Yet Another S3-backed File System) is a Filesystem in Userspace (FUSE) interface to Amazon S3. It was inspired by s3fs but rewritten from scratch to implement a distributed cache synchronized by Amazon SNS notifications. A web console is provided to easily monitor the nodes of a cluster.
RunJOP (Run Just Once Please) is a distributed execution framework to run a command (i.e. a job) only once in a group of servers [built using AWS DynamoDB and S3].nifty! Distributed cron is pretty easy when you’ve got Dynamo doing the heavy lifting.
Spot-on points which Docker needs to address. It’s still production-ready, and _should_ be used there, it just has significant rough edges…
The latest hacky workaround to Twitter’s API shortcoming
mocks are the sound of your code crying out, “please structure me differently!”+1
go NewEgg: ‘Newegg went against a company that claimed its patent covered SSL and RC4 encryption, a common encryption system used by many retailers and websites. This particular patent troll has gone against over 100 other companies, and brought in $45 million in settlements before going after Newegg. We won.’
A lovely eulogy for Nóirín Plunkett, from Rich Bowen. RIP Nóirín :(
beautiful visualisation of a decision tree
This is a great post from Kent Beck, putting a lot of recent deployment/rollout patterns in a clear context — that of supporting “reversibility”:
Development servers. Each engineer has their own copy of the entire site. Engineers can make a change, see the consequences, and reverse the change in seconds without affecting anyone else. Code review. Engineers can propose a change, get feedback, and improve or abandon it in minutes or hours, all before affecting any people using Facebook. Internal usage. Engineers can make a change, get feedback from thousands of employees using the change, and roll it back in an hour. Staged rollout. We can begin deploying a change to a billion people and, if the metrics tank, take it back before problems affect most people using Facebook. Dynamic configuration. If an engineer has planned for it in the code, we can turn off an offending feature in production in seconds. Alternatively, we can dial features up and down in tiny increments (i.e. only 0.1% of people see the feature) to discover and avoid non-linear effects. Correlation. Our correlation tools let us easily see the unexpected consequences of features so we know to turn them off even when those consequences aren’t obvious. IRC. We can roll out features potentially affecting our ability to communicate internally via Facebook because we have uncorrelated communication channels like IRC and phones. Right hand side units. We can add a little bit of functionality to the website and turn it on and off in seconds, all without interfering with people’s primary interaction with NewsFeed. Shadow production. We can experiment with new services under real load, from a tiny trickle to the whole flood, without affecting production. Frequent pushes. Reversing some changes require a code change. On the website we never more than eight hours from the next schedule code push (minutes if a fix is urgent and you are willing to compensate Release Engineering). The time frame for code reversibility on the mobile applications is longer, but the downward trend is clear from six weeks to four to (currently) two. Data-informed decisions. (Thanks to Dave Cleal) Data-informed decisions are inherently reversible (with the exceptions noted below). “We expect this feature to affect this metric. If it doesn’t, it’s gone.” Advance countries. We can roll a feature out to a whole country, generate accurate feedback, and roll it back without affecting most of the people using Facebook. Soft launches. When we roll out a feature or application with a minimum of fanfare it can be pulled back with a minimum of public attention. Double write/bulk migrate/double read. Even as fundamental a decision as storage format is reversible if we follow this format: start writing all new data to the new data store, migrate all the old data, then start reading from the new data store in parallel with the old.We do a bunch of these in work, and the rest are on the to-do list. +1 to these!
Walkthrough of debugging connection timeouts in a load test. Nice graphs (using matplotlib)
WB: By the late 80s the IANA [the Internet Assigned Numbers Authority, set up in 1988 to manage global IP address allocations] was trying to get all those countries that were trying to join the internet to use the ISO 3166 standard for country codes. It was used for all sorts of things?—?you see it on cars, “GB” for the UK. [...] At that point, we’re faced with a problem that Jon Postel would like to have changed it to .gb to be consistent with the rest of the world. Whereas .uk had already been established, with a few tens of thousands of domain names with .uk on them. I remember chairing one of the JANET net workshops that were held every year, and the Northern Irish were adamant that they were part of the UK?—?so the consensus was, we’d try and keep .uk, we’d park .gb and not use it. PK: I didn’t particularly want to change to .gb because I was responsible for Northern Ireland as well. And what’s more, there was a certain question as to whether a research group in the US should be allowed to tell the British what to do. So this argy-bargy continued for a little while and, in the meantime, one of my clients was the Ministry of Defence, and they decided they couldn’t wait this long, and they decided I was going to lose the battle, and so bits of MOD went over to .gb?—?I didn’t care, as I was running .gb and .uk in any case.
The solution is for social media sites and the police to take threats or jokes about swatting, doxxing, and organized crime seriously. Tweeting about buying a gun and shooting up a school would be taken seriously, and so should the threat of raping, doxxing, swatting or killing someone. Privacy issues and online harassment are directly linked, and online harassment isn’t going anywhere. My fear is that, in reaction to online harassment, laws will be passed that will break down our civil freedoms and rights online, and that more surveillance will be sold to users under the guise of safety. More surveillance, however, would not have helped me or my mother. A platform that takes harassment and threats seriously instead of treating them like jokes would have.
Deep Dream estranges us from our fears, perhaps, but it doesn’t make them go away. It’s easy to discuss Deep Dream as an independent creature, a foreign intelligence that we interact with for fun. Yet like all kitsch, it comes straight back to its creators.
now this is a Long Read. the inimitable Margaret Atwood on climate change, beautifully illustrated
While someone can certainly make the case that an AK-47, or any other kind of gun or rifle is designed, nothing whose primary purpose is to take away life can be said to be designed well. And that attempting to separate an object from its function in order to appreciate it for purely aesthetic reasons, or to be impressed by its minimal elegance, is a coward’s way of justifying the death they’ve designed into the word, and the money with which they’re lining their pockets.
turns out Ruby has a good set of random-text-generation gems on offer
Amazing story of 1960s detente via Maciej: ‘During the Cold War, Boeing execs got a strange call from the State Department: Would you guys mind trading secrets with the Russians?’
This bill prioritises other peoples’ “alarm or distress” over your communications not just TO them but also ABOUT them. Don’t like what Joan Burton is doing with the water charges? Want to write something on independent media about what you think of that? Better not alarm or distress or harm her! This is the core of my issue with the bill. It’s not just that almost all the agreeable parts of it are already covered by other laws. It’s not just that it’s utterly unenforceable with our current justice system. It’s not just that it’s so vague and fluffy. It’s that it’s so ill-defined and over-reaching that its interpretation will inevitably have to be left to judges. Leaving anything to judges is a bad idea in general. This overly broad and poorly worded bill is a god-send to people who like to bully others into silence. Ironic that eh?!
Lambdas in Java 8 introduce some unpredictable performance implications, due to reliance on escape analysis to eliminate object allocation on every lambda invocation. Peter Lawrey has some details
an excellent followup operational post on CircleCI’s “database is not a queue” outage
(1) players are anonymous, and the possibility of “policing individual behavior is almost impossible”; (2) they only encounter each other a few times in passing — it’s very possible to hurl an expletive at another player, and never “see” him or her again; and (3) finally, and perhaps predictably, the sex-ratio of players is biased pretty heavily toward men. (A 2014 survey of gender ratios on Reddit found that r/halo was over 95 percent male.) [....] In each of these environments, Kasumovic suggests, a recent influx of female participants has disrupted a pre-existing social hierarchy. That’s okay for the guys at the top — but for the guys at the bottom, who stand to lose more status, that’s very threatening. (It’s also in keeping with the evolutionary framework on anti-lady hostility, which suggests sexism is a kind of Neanderthal defense mechanism for low-status, non-dominant men trying to maintain a shaky grip on their particular cave’s supply of women.) “As men often rely on aggression to maintain their dominant social status,” Kasumovic writes, “the increase in hostility towards a woman by lower-status males may be an attempt to disregard a female’s performance and suppress her disturbance on the hierarchy to retain their social rank.”
Most analyses of the market indicate that office parks simply aren’t as appealing or profitable as they were in the 20th century and that Americans just aren’t as keen to cloister themselves in workspaces that are reachable only by car.
Jaysus, this is terrifying.
Miller and Valasek’s full arsenal includes functions that at lower speeds fully kill the engine, abruptly engage the brakes, or disable them altogether. The most disturbing maneuver came when they cut the Jeep’s brakes, leaving me frantically pumping the pedal as the 2-ton SUV slid uncontrollably into a ditch.Avoid any car which supports this staggeringly-badly-conceived Uconnect feature:
All of this is possible only because Chrysler, like practically all carmakers, is doing its best to turn the modern automobile into a smartphone. Uconnect, an Internet-connected computer feature in hundreds of thousands of Fiat Chrysler cars, SUVs, and trucks, controls the vehicle’s entertainment and navigation, enables phone calls, and even offers a Wi-Fi hot spot.:facepalm: Also, Chrysler’s response sucks: “Chrysler’s patch must be manually implemented via a USB stick or by a dealership mechanic.”
the RFC take on ISO-8601. I need to update my mental bookmarks to start referring to this instead
Great turn of phrase from Matthew Green (@matthew_d_green). Emin Gün Sirer adds some detail: “well, an asset with bounded value, and an unbounded liability”
good detail in this presentation
oh look, Google has a flight search engine! I had no idea
TorrentFreak has the story of a UK-producer and songwriter named Lee Adams who took part in an official remix competition of boy band One Direction’s music, put on by the band and its label, Sony Music. The stems for remixing were released on Soundcloud. The rules of the contest required entrants to upload their remixes on Soundcloud… and that’s exactly what Adams did. And yet those works still got taken down via copyright claims from Sony Music as infringing.
Argentina has, as a matter of constitutional law, effectively open borders. There are no caps or quotas or lottery systems. You can move there legally if you have an employer or family member to sponsor you. That’s all you need. If you don’t have a sponsor, and make your way in illegally, you’re recognized as an “irregular migrant.” Discrimination against irregular migrants in health care or education is illegal, and deportation in noncriminal cases is exceptionally rare. Large-scale amnesties are the norm. Obviously Argentina is not nearly as rich as Germany or the US or the UK. But it’s considerably richer than three of its neighbors (Bolivia, Paraguay, and Brazil). And yet it doesn’t try hard to keep their residents out. It welcomes them — as it should. “One could have expected catastrophe—an uncontrollable flow of poorer immigrants streaming into the country coupled with angry public backlash,” Elizabeth Slater writes in the World Policy Journal. “That hasn’t happened.” Angela Merkel clearly expects catastrophe if she lets people like this weeping young Palestinian girl stay in Germany. That catastrophe is simply a myth; it wouldn’t happen. What would happen is that Germany’s economy would grow, its culture would grow richer, and that girl and more like her could see their lives improve immeasurably.
‘A programming language based on the one liners of Arnold Schwarzenegger’. Presenting hello.arnoldc: IT’S SHOWTIME TALK TO THE HAND “hello world” YOU HAVE BEEN TERMINATED (via Robert Walsh)
This is a nifty hack. TIL! ’9-patch uses png transparency to do an advanced form of 9-slice or scale9. The guides are straight, 1-pixel black lines drawn on the edge of your image that define the scaling and fill of your image. By naming your image file name.9.png, Android will recognize the 9.png format and use the black guides to scale and fill your bitmaps.’
Mr Murphy said it was the Government’s objective for Ireland to be a leader on data protection and data-related issues. The members of the forum include Data Protection Commissioner Helen Dixon, John Barron, chief technology officer with the Revenue Commissioners, Seamus Carroll, head of civil law reform division at the Department of Justice and Tim Duggan, assistant secretary with the Department of Social Protection. Gary Davis, director of privacy and law enforcement requests with Apple, is also on the forum. Mr Davis is a former deputy data protection commissioner in Ireland. There are also representatives from Google, Twitter, LinkedIn and Facebook, from the IDA, the Law Society and the National Statistics Board. Chair of Digital Rights Ireland Dr TJ McIntyre and Dr Eoin O’Dell, associate professor, School of Law, Trinity College Dublin are also on the voluntary forum.
From DataDog. See also “How to collect NGINX metrics”: https://www.datadoghq.com/blog/how-to-collect-nginx-metrics/
nicely detailed writeup of how New Relic are dockerizing
a brief howto
a new sketch algorithm from Baron Schwartz and Preetam Jinka of VividCortex; similar to Count-Min but with last-seen timestamp instead of frequency.
The UK Ordnance Survey’s “open data’ free product, free for all uses:
Code-Point Open is FREE to view, download and use for commercial, educational and personal purposes.(via Antoin)
Interestingly, they claim that IPv6 tends to be more reliable and has lower latency now:
Based on our testing, this makes our Happy Eyeballs implementation go from roughly 50/50 IPv4/IPv6 in iOS 8 and Yosemite to ~99% IPv6 in iOS 9 and El Capitan betas. While our previous implementation from four years ago was designed to select the connection with lowest latency no matter what, we agree that the Internet has changed since then and reports indicate that biasing towards IPv6 is now beneficial for our customers: IPv6 is now mainstream instead of being an exception, there are less broken IPv6 tunnels, IPv4 carrier-grade NATs are increasing in numbers, and throughput may even be better on average over IPv6.
lest we forget — this is a 2014-era writeup of OpenPostcode (open), Loc8 and GoCode (proprietary) as alternative options to the Eircode system
handy step-by-step clickthrough guide
Excellent HN thread re automated anomaly detection in production, Q&A with the dev team
A new HTTP client library for Android and Java, with a lot of nice features:
HTTP/2 and SPDY support allows all requests to the same host to share a socket. Connection pooling reduces request latency (if SPDY isn’t available). Transparent GZIP shrinks download sizes. Response caching avoids the network completely for repeat requests. OkHttp perseveres when the network is troublesome: it will silently recover from common connection problems. If your service has multiple IP addresses OkHttp will attempt alternate addresses if the first connect fails. This is necessary for IPv4+IPv6 and for services hosted in redundant data centers. OkHttp initiates new connections with modern TLS features (SNI, ALPN), and falls back to TLS 1.0 if the handshake fails. Using OkHttp is easy. Its 2.0 API is designed with fluent builders and immutability. It supports both synchronous blocking calls and async calls with callbacks.
Reasonably solid white paper
‘Honestly, I don’t think it will take long for other drivers to realize that self-driving cars are “easy targets” in traffic.’ — also, an insurance expert suggests that self-driving cars won’t increase premiums
NZ’s HDC Act gets the EFF thumbs-down
Cloudflare are running a 40-machine, 50TB Kafka cluster, ingesting at 15 Gbps, for log processing. Also: Go producers/consumers, capnproto as wire format, and CitusDB/Postgres to store rolled-up analytics output. Also using Space Saver (top-k) and HLL (counting) estimation algorithms.
a command line tool for JVM diagnostic troubleshooting and profiling.
‘Simplistic interactive filtering tool’ — live incremental-search filtering in a terminal window
Yay. Sensible Docker registry pricing at last. Given the high prices, rough edges and slow performance of the other registry offerings, I’m quite happy to see this.
Google Container Registry helps make it easy for you to store your container images in a private and encrypted registry, built on Cloud Platform. Pricing for storing images in Container Registry is simple: you only pay Google Cloud Storage costs. Pushing images is free, and pulling Docker images within a Google Cloud Platform region is free (Cloud Storage egress cost when outside of a region). Container Registry is now ready for production use: * Encrypted and Authenticated – Your container images are encrypted at rest, and access is authenticated using Cloud Platform OAuth and transmitted over SSL * Fast – Container Registry is fast and can handle the demands of your application, because it is built on Cloud Storage and Cloud Networking. * Simple – If you’re using Docker, just tag your image with a gcr.io tag and push it to the registry to get started. Manage your images in the Google Developers Console. * Local – If your cluster runs in Asia or Europe, you can now store your images in ASIA or EU specific repositories using asia.gcr.io and eu.gcr.io tags.
Pragmatic evolution story, adding Docker as a packaging/deploy format for an existing production Capistrano/Rails fleet
Hystrix-style Circuit Breakers and Bulkheads for Ruby/Rails, from Shopify
GitHub’s statsd replacement in C
‘Want to learn how Facebook scales their load balancing infrastructure to support more than 1.3 billion users? We will be revealing the technologies and methods we use to route and balance Facebook’s traffic. The Traffic team at Facebook has built several systems for managing and balancing our site traffic, including both a DNS load balancer and a software load balancer capable of handling several protocols. This talk will focus on these technologies and how they have helped improve user performance, manage capacity, and increase reliability.’ Can’t find the standalone slides, unfortunately.
a good collection of coding fonts (via Tony Finch)
Finagle Futures ported to C++11
So then they reach inside to one of the layers and spin the knob randomly to fuck it up. Lower layers are edges and curves. Higher layers are faces, eyes and shoggoth ovipositors. [....] But the best part is not when they just glitch an image — which is a fun kind of embossing at one end, and the “extra eyes” filter at the other — but is when they take a net trained on some particular set of objects and feed it static, then zoom in, and feed the output back in repeatedly. That’s when you converge upon the platonic ideal of those objects, which — it turns out — tend to be Giger nightmare landscapes. Who knew. (I knew.)This stuff is still boggling my mind. All those doggy faces! That is one dog-obsessed ANN.
The paper describing the innards of Spark Streaming and its RDD-based recomputation algorithm:
we use a data structure called Resilient Distributed Datasets (RDDs), which keeps data in memory and can recover it without replication by tracking the lineage graph of operations that were used to build it. With RDDs, we show that we can attain sub-second end-to-end latencies. We believe that this is sufficient for many real-world big data applications, where the timescale of the events tracked (e.g., trends in social media) is much higher.
Gor, a very nice-looking tool to log and replay HTTP traffic, specifically designed to “tee” live traffic from production to staging for pre-release testing
Well-written description of the pros and cons. I’m a rebaser, fwiw. (via Darrell)
To sum up, if you want a perfect performance you need to: Ensure traffic is distributed evenly across many RX queues and SO_REUSEPORT processes. In practice, the load usually is well distributed as long as there are a large number of connections (or flows). You need to have enough spare CPU capacity to actually pick up the packets from the kernel. To make the things harder, both RX queues and receiver processes should be on a single NUMA node.
This is amazing, and a little scary.
If we choose higher-level layers, which identify more sophisticated features in images, complex features or even whole objects tend to emerge. Again, we just start with an existing image and give it to our neural net. We ask the network: “Whatever you see there, I want more of it!” This creates a feedback loop: if a cloud looks a little bit like a bird, the network will make it look more like a bird. This in turn will make the network recognize the bird even more strongly on the next pass and so forth, until a highly detailed bird appears, seemingly out of nowhere.An enlightening comment from the G+ thread:
This is the most fun we’ve had in the office in a while. We’ve even made some of those ‘Inceptionistic’ art pieces into giant posters. Beyond the eye candy, there is actually something deeply interesting in this line of work: neural networks have a bad reputation for being strange black boxes that that are opaque to inspection. I have never understood those charges: any other model (GMM, SVM, Random Forests) of any sufficient complexity for a real task is completely opaque for very fundamental reasons: their non-linear structure makes it hard to project back the function they represent into their input space and make sense of it. Not so with backprop, as this blog post shows eloquently: you can query the model and ask what it believes it is seeing or ‘wants’ to see simply by following gradients. This ‘guided hallucination’ technique is very powerful and the gorgeous visualizations it generates are very evocative of what’s really going on in the network.?
This is great news — the current protocol is a binary, proprietary horrorshow, particularly around error reporting. Available “later this year” in production, and Pushy plan to support it.
tl;dr: Code review trumps TDD alone for finding bugs. (Via Mark Dennehy)
Objective: Our objective is to describe how software engineering might benefit from an evidence-based approach and to identify the potential difficulties associated with the approach. Method: We compared the organisation and technical infrastructure supporting evidence-based medicine (EBM) with the situation in software engineering. We considered the impact that factors peculiar to software engineering (i.e. the skill factor and the lifecycle factor) would have on our ability to practice evidence-based software engineering (EBSE). Results: EBSE promises a number of benefits by encouraging integration of research results with a view to supporting the needs of many different stakeholder groups. However, we do not currently have the infrastructure needed for widespread adoption of EBSE. The skill factor means software engineering experiments are vulnerable to subject and experimenter bias. The lifecycle factor means it is difficult to determine how technologies will behave once deployed. Conclusions: Software engineering would benefit from adopting what it can of the evidence approach provided that it deals with the specific problems that arise from the nature of software engineering.(via Mark Dennehy)
curl -s http://checkip.amazonaws.com/
The ruling is terrible through and through. First off, it insists that the comments on the news story were clearly “hate speech” and that, as such, “did not require any linguistic or legal analysis since the remarks were on their face manifestly unlawful.” To the court, this means that it’s obvious such comments should have been censored straight out. That’s troubling for a whole host of reasons at the outset, and highlights the problematic views of expressive freedom in Europe. Even worse, however, the Court then notes that freedom of expression is “interfered with” by this ruling, but it doesn’t seem to care — saying that it is deemed “necessary in a democratic society.”This is going to have massive chilling effects. Terrible ruling from the ECHR.
In the wake of this judgment, the legal situation is complicated. In an e-mail to Ars, T J McIntyre, who is a lecturer in law and Chairman of Digital Rights Ireland, the lead organization that won an important victory against EU data retention in the Court of Justice of the European Union last year, explained where things now stand. “Today’s decision doesn’t have any direct legal effect. It simply finds that Estonia’s laws on site liability aren’t incompatible with the ECHR. It doesn’t directly require any change in national or EU law. Indirectly, however, it may be influential in further development of the law in a way which undermines freedom of expression. As a decision of the Grand Chamber of the ECHR it will be given weight by other courts and by legislative bodies.”
We used three key principles in designing our datacenter networks: We arrange our network around a Clos topology, a network configuration where a collection of smaller (cheaper) switches are arranged to provide the properties of a much larger logical switch. We use a centralized software control stack to manage thousands of switches within the data center, making them effectively act as one large fabric. We build our own software and hardware using silicon from vendors, relying less on standard Internet protocols and more on custom protocols tailored to the data center.
Nice hack. An automated nginx reverse proxy which regenerates as the Docker containers update
Parse on their ditching-Rails story. I haven’t heard a nice thing about Ruby or Rails as an operational, production-quality platform in a long time :(
we are introducing Flow Logs for the Amazon Virtual Private Cloud. Once enabled for a particular VPC, VPC subnet, or Elastic Network Interface (ENI), relevant network traffic will be logged to CloudWatch Logs for storage and analysis by your own applications or third-party tools. You can create alarms that will fire if certain types of traffic are detected; you can also create metrics to help you to identify trends and patterns. The information captured includes information about allowed and denied traffic (based on security group and network ACL rules). It also includes source and destination IP addresses, ports, the IANA protocol number, packet and byte counts, a time interval during which the flow was observed, and an action (ACCEPT or REJECT).
‘[Tim Hunt] said that while he meant to be ironic, he did think it was hard to collaborate with women because they are too emotional – that he was trying to be honest about the problems.’ So much for the “nasty twitter took my jokes seriously” claims then.
Good post, and hard to disagree.
One of the “features” of systemd is that it allows you to boot a system without needing a shell at all. This seems like such a senseless manoeuvre that I can’t help but think of it as a knee-jerk reaction to the perception of Too Much Shell in sysv init scripts. In exactly which universe is it reasonable to assume that you have a running D-Bus service (or kdbus) and a filesystem containing unit files, all the binaries they refer to, all the libraries they link against, and all the configuration files any of them reference, but that you lack that most ubiquitous of UNIX binaries, /bin/sh?
ouch, really sounds like Storm didn’t cut the muster. ‘It’s hard to imagine something more damaging to Apache Storm than this. Having read it through, I’m left with the impression that the paper might as well have been titled “Why Storm Sucks”, which coming from Twitter themselves is quite a statement.’ If I was to summarise the lessons learned, it sounds like: backpressure is required; and multi-tenant architectures suck.
Allied Irish Banks’s web and mobile banking portals are ludicrously insecure. Vast numbers of accounts have easily-guessable registration numbers and are thus ‘protected’ by a level of security that is twice as easy to crack as would be provided by a single password containing only two lowercase letters. A person of malicious intent could easily gain access to hundreds, possibly thousands, of accounts as well as completely overwhelm the branch network by locking an estimated several 100,000s of people out of their online banking. Both AIB and the Irish Financial Services Ombudsman have refused to respond meaningfully to multiple communications each in which these concerns were raised privately.
Nice detailed description of an auto-scaled SQS worker pool
The Daily Beast is scathing re the OPM hack:
Here’s where things start to get scary. Whoever has OPM’s records knows an astonishing amount about millions of federal workers, members of the military, and security clearance holders. They can now target those Americans for recruitment or influence. After all, they know their vices, every last one—the gambling habit, the inability to pay bills on time, the spats with former spouses, the taste for something sexual on the side—since all that is recorded in security clearance paperwork. (To get an idea of how detailed this gets, you can see the form, called an SF86, here.) Speaking as a former counterintelligence officer, it really doesn’t get much worse than this.
‘Easy Skeezy Ruby Date/Time Formatting’ — or indeed anywhere else strftime() is supported
‘a fully-automated solution to build auto-scaling etcd clusters in AWS’
New static analysis goodnews, freshly open-sourced by Facebook:
Facebook Infer uses logic to do reasoning about a program’s execution, but reasoning at this scale — for large applications built from millions of lines of source code — is hard. Theoretically, the number of possibilities that need to be checked is more than the number of estimated atoms in the observable universe. Furthermore, at Facebook our code is not a fixed artifact but an evolving system, updated frequently and concurrently by many developers. It is not unusual to see more than a thousand modifications to our mobile code submitted for review in a given day. The requirements on the program analyzer then become even more challenging because we expect a tool to report quickly on these code modifications — in the region of 10 minutes — to fit in with developers’ workflow. Coping with this scale and velocity requires advanced mathematical techniques. Facebook Infer uses two such techniques: separation logic and bi-abduction. Separation logic is a theory that allows Facebook Infer’s analysis to reason about small, independent parts of the application storage, rather than having to consider the entirety of the memory potentially at every step. That would be a daunting task on modern processors with their large addressable virtual memories. Bi-abduction is a logical inference technique that allows Facebook Infer to discover properties about the behavior of independent parts of the application code. By storing these properties between runs, Facebook Infer needs to analyze only the parts of the software that have changed, reusing the results of its previous analysis where it can. By combining these approaches, our analyzer is able to find complex problems in modifications to an application built from millions of lines of code, in minutes.(via Bryan O’Sullivan)
This is great. the story of how cheesy funk carioca tune “A Minha Amiga Fran” managed to become “Kawo Kawo” and become a massive hit in Thailand
oh dear. I can see how this happened — in many cases they may not still have samples to derive new sums from :(
what’s the export policy for Google’s new Photos service? pretty good, it turns out
the bottom line is as follows: If the time it takes to create a dump, ? < M/2 then use ?opt = ?(2?M) – ? Otherwise (it takes longer than M/2 to create a dump), just use ?opt = M.
Creating a user with our DSL looks like: registrationAPI.createUser(“user”); You might expect this to create a user with the username ‘user’, but then we’d get conflicts between every test that wanted to call their user ‘user’ which would prevent tests from running safely against the same deployment of the exchange. Instead, ‘user’ is just an alias that is only meaningful while this one test is running. The DSL creates a unique username that it uses when talking to the actual system. Typically this is done by adding a postfix so the real username is still reasonably understandable e.g. user-fhoai42lfkf.Nice approach — makes sense.
Orbit Async implements async-await methods in the JVM. It allows programmers to write asynchronous code in a sequential fashion. It was developed by BioWare, a division of Electronic Arts.Open source, BSD-licensed.
Well, well — looks like AWS is about to disrupt PKI, and about time too. If they come up with a Plex-style “provision a cert” API, it’ll be revolutionary
including a lovely set from Tove Jansson
large-scale automated TLS certificate deployment. very impressive and not easy to reproduce, good work Plex! (via Nelson)
Airbnb’s workflow management system; works off a DAG defined in Python code (ugh). Nice UI though, but I think Pinboard’s take is neater
Introducing the chumbox
A high-performance java build tool, from Facebook. Make-like
in favour of a proprietary ground-up rewrite called Heron. Reading between the lines it sounds like Storm had problems with latency, reliability, data loss, and supporting back pressure.
neat substitute for physical-time clocks in synchronization and ordering in a distributed system, based on Lamport’s Logical Clocks and Google’s TrueTime. ‘HLC captures the causality relationship like LC, and enables easy identification of consistent snapshots in distributed systems. Dually, HLC can be used in lieu of PT clocks since it maintains its logical clock to be always close to the PT clock.’
Increasingly bizarre postal address obfuscation with An Post, the Irish postal service. Example:
I have decided to see what you can post [....] My first experiment was a dice [sic] with one line of the address on each side. An Post delivered two days later. They win this roundVia JG
‘Can do ~1M queries to ~3K public DNS servers within ~3 minutes with just a few threads.’ via Trustin Lee. Netty is the business
Good series of blog posts on the LMAX trading platform’s performance testing strategy — they capture live traffic off the wire, then build statistical models simulating its features. See also http://epickrram.blogspot.co.uk/2014/07/performance-testing-at-lmax-part-two.html and http://epickrram.blogspot.co.uk/2014/08/performance-testing-at-lmax-part-three.html .
The modern state system is built on a bargain between governments and citizens. States provide collective social goods, and in turn, via a system of norms, institutions, regulations, and ethics to hold this power accountable, citizens give states legitimacy. This bargain created order and stability out of what was an increasingly chaotic global system. If algorithms represent a new ungoverned space, a hidden and potentially ever-evolving unknowable public good, then they are an affront to our democratic system, one that requires transparency and accountability in order to function. A node of power that exists outside of these bounds is a threat to the notion of collective governance itself. This, at its core, is a profoundly undemocratic notion—one that states will have to engage with seriously if they are going to remain relevant and legitimate to their digital citizenry who give them their power.
great example of how Minecraft solved the problem the easy way — by simply not making an MMO, the whole problem effectively goes away
Changes which server-side developers will need to start considering as HTTP/2 rolls out. Remove domain sharding; stop concatenating resources; stop inlining resources; use server push.
Without switching to chronyd, ntpd -x sounds not too suboptimal:
With ntpd, the kernel backward step is used by default. With ntpd versions before 4.2.6, or 4.2.6 and later patched for this bug, the -x option (added to /etc/sysconfig/ntpd) can be used to disable the kernel leap second correction and ignore the leap second as far as the local clock is concerned. The one-second error gained after the leap second will be measured and corrected later by slewing in normal operation using NTP servers which already corrected their local clocks.It’s all pretty messy though :(
Russia’s troll farms. Ladies and gentlemen — the future