Links for 2014-07-06

  • Layered Glass Table Concept Creates a Cross-Section of the Ocean

    beautiful stuff — and a snip at only UKP 5,800 ex VAT. it’d make a good DIY project though ;)

    (tags: art tables glass layering 3d cross-sections water ocean sea mapping cartography layers this-is-colossal design furniture)

  • Two traps in iostat: %util and svctm

    Marc Brooker:

    As a measure of general IO busyness %util is fairly handy, but as an indication of how much the system is doing compared to what it can do, it’s terrible. Iostat’s svctm has even fewer redeeming strengths. It’s just extremely misleading for most modern storage systems and workloads. Both of these fields are likely to mislead more than inform on modern SSD-based storage systems, and their use should be treated with extreme care.

    (tags: ioutil iostat svctm ops ssd disks hardware metrics stats linux)

  • New AWS Web Services region: eu-central-1 (soon)

    Iiiinteresting. Sounds like new anti-NSA-snooping privacy laws will be driving a lot of new mini-regions in AWS. Hope Amazon have their new-region-standup process a little more streamlined by now than when I was there ;)

    (tags: aws germany privacy ec2 eu-central-1 nsa snooping)

  • How A Spam Newsletter Caused a Bank Run in Bulgaria

    According to the Bulgarian National Security Agency (see here, for a reporting in English), an investment company that “built a network of associated companies for marketing services” that was used to diffuse panic by means of an alert, uncomfortably titled “Information Bulletin of on the Risk of Deposits in Bulgarian Banks”. The “bulletin” claimed – Bloomberg reports – KTB was undergoing a liquidity shortage. The message apparently also said that the government deposit guarantee fund was under-capitalised to meet possible repayments, that banks could go bankrupt and that the peg of the currency with the euro could be broken. Allegedly, the alert was diffused by text, email and even Facebook messages, thus ensuring a very widespread outreach. In a country that in 1997 underwent a very serious banking crisis featuring all these characteristics – whose memory is still fresh – this was enough to spur panic.

    (tags: spam banking bulgaria banks euro panic facebook social-media)

  • New Russian Law To Forbid Storing Russians’ Data Outside the Country – Slashdot

    On Friday Russia’s parliament passed a law “which bans online businesses from storing personal data of Russian citizens on servers located abroad[.] … According to ITAR-TASS, the changes to existing legislation will come into effect in September 2016, and apply to email services, social networks and search engines, including the likes of Facebook and Google. Domain names or net addresses not complying with regulations will be put on a blacklist maintained by Roskomnadzor (the Federal Supervision Agency for Information Technologies and Communications), the organisation which already has the powers to take down websites suspected of copyright infringement without a court order. In the case of non-compliance, Roskomnadzor will be able to impose ‘sanctions,’ and even instruct local Internet Service Providers (ISPs) to cut off access to the offending resource.”

    (tags: russia privacy nsa censorship protectionism internet web)

Posted in Uncategorized | Comments closed

Links for 2014-07-04

  • Irish parliament pressing ahead with increased access to retained telecoms data

    While much of the new bill is concerned with the dissolution of the Competition Authority and the National Consumer Agency and the formation of a new merged Competition and Consumer Protection Commission (CCPC) the new bill also proposed to extend the powers of the new CCPC to help it investigate serious anticompetitive behaviour. Strikingly the new bill proposes to give members of the CCPC the power to access data retained under the Communications (Retention of Data) Act 2011. As readers will recall this act implements Directive 2006/24/EC which obliges telecommunications companies to archive traffic and location data for a period of up to two years to facilitate the investigation of serious crime. Ireland chose to implement the maximum two year retention period and provided access to An Garda Siochana, The Defence Forces and the Revenue Commissioners. The current reform of Irish competition law now proposes to extend data access powers to the members of the CCPC for the purposes of investigating cartel offences.

    (tags: data-retention privacy surveillance competition ccpc ireland law dri)

  • NSA: Linux Journal is an “extremist forum” and its readers get flagged for extra surveillance has published the relevant XKEYSCORE source code, and if you look closely at the rule definitions, you will see* listed alongside Tails and Tor. According to an article on, the NSA considers Linux Journal an “extremist forum”. This means that merely looking for any Linux content on Linux Journal, not just content about anonymizing software or encryption, is considered suspicious and means your Internet traffic may be stored indefinitely.
    This is, sadly, entirely predictable — that’s what happens when you optimize the system for over-sampling, with poor oversight.

    (tags: false-positives linuxjournal linux terrorism tor tails nsa surveillance snooping xkeyscore selectors oversight)

  • stout

    a C++ library adding some modern language features like Option, Try, Stopwatch, and other Guava-ish things (via @cscotta)

    (tags: c++ library stout option try guava coding)

Posted in Uncategorized | Comments closed

Links for 2014-07-03

Posted in Uncategorized | Comments closed

Links for 2014-07-02

Posted in Uncategorized | Comments closed

Links for 2014-07-01

Posted in Uncategorized | Comments closed

Links for 2014-06-30

  • Facebook Doesn’t Understand The Fuss About Its Emotion Manipulation Study

    This is quite unethical, and I’m amazed it was published at all. Kashmir Hill at Forbes nails it:

    While many users may already expect and be willing to have their behavior studied — and while that may be warranted with “research” being one of the 9,045 words in the data use policy — they don’t expect that Facebook will actively manipulate their environment in order to see how they react. That’s a new level of experimentation, turning Facebook from a fishbowl into a petri dish, and it’s why people are flipping out about this.
    Shocking stuff. We need a new social publishing platform, built on ethical, open systems.

    (tags: ethics facebook privacy academia depression feelings emotion social-publishing social experimentation papers)

  • Building a Smarter Application Stack – DevOps Ireland

    This sounds like a very interesting Dublin meetup — Engine Yard on thursday night:

    This month, we’ll have Tomas Doran from Yelp talking about Docker, service discovery, and deployments. ‘There are many advantages to a container based, microservices architecture – however, as always, there is no silver bullet. Any serious deployment will involve multiple host machines, and will have a pressing need to migrate containers between hosts at some point. In such a dynamic world hard coding IP addresses, or even host names is not a viable solution. This talk will take a journey through how Yelp has solved the discovery problems using Airbnb’s SmartStack to dynamically discover service dependencies, and how this is helping unify our architecture, from traditional metal to EC2 ‘immutable’ SOA images, to Docker containers.’

    (tags: meetups talks dublin deployment smartstack ec2 docker yelp service-discovery)

  • Smart Integration Testing with Dropwizard, Flyway and Retrofit

    Retrofit in particular looks neat. Mind you having worked with in-memory SQL databases before for integration testing, I’d never do that again — too many interop glitches compared to “real world” MySQL/Postgres

    (tags: testing integration-testing retrofit flyway dropwizard logentries)

  • Twitter’s TSAR

    TSAR = “Time Series AggregatoR”. Twitter’s new event processor-style architecture for internal metrics. It’s notable that now Twitter and Google are both apparently moving towards this idea of a model of code which is designed to run equally in realtime streaming and batch modes (Summingbird, Millwheel, Flume).

    (tags: analytics architecture twitter tsar aggregation event-processing metrics streaming hadoop batch)

  • ‘Robust De-anonymization of Large Sparse Datasets’ [pdf]

    paper by Arvind Narayanan and Vitaly Shmatikov, 2008. ‘We present a new class of statistical de- anonymization attacks against high-dimensional micro-data, such as individual preferences, recommendations, transaction records and so on. Our techniques are robust to perturbation in the data and tolerate some mistakes in the adversary’s background knowledge. We apply our de-anonymization methodology to the Netflix Prize dataset, which contains anonymous movie ratings of 500,000 subscribers of Netflix, the world’s largest online movie rental service. We demonstrate that an adversary who knows only a little bit about an individual subscriber can easily identify this subscriber’s record in the dataset. Using the Internet Movie Database as the source of background knowledge, we successfully identified the Netflix records of known users, uncovering their apparent political preferences and other potentially sensitive information.’

    (tags: anonymisation anonymization sanitisation databases data-dumps privacy security papers)

  • HSE data releases may be de-anonymisable

    Although the data has been kept anonymous, the increasing sophistication of computer-driven data-mining techniques has led to fears patients could be identified. A HSE spokesman confirmed yesterday that the office responded to requests for data from a variety of sources, including researchers, the universities, GPs, the media, health insurers and pharmaceutical companies. An average of about two requests a week was received. [...] The information provided by the HPO has significant patient identifiers removed, such as name and date of birth. According to the HSE spokesman, individual patient information is not provided and, where information is sought for a small group of patients, this is not provided where the number involved is under five. “In such circumstances, it is highly unlikely that anyone could be identified. Nevertheless, we will have another look at data releases from the office,” he said.
    I’d say this could be readily reversible, from the sounds of it.

    (tags: anonymisation sanitisation data-dumps hse health privacy via:tjmcintyre)

  • Beautiful algorithm visualisations from Mike Bostock

    This is a few days old, but unmissable. I swear, the ‘Wilson’s algorithm transformed into a tidy tree layout’ viz brought tears to my eyes ;)

    (tags: dataviz algorithms visualization visualisation mazes trees sorting animation mike-bostock)

  • ByteArrayOutputStream is really, really slow sometimes in JDK6

    This leads us to the bug. The size of the array is determined by Math.max(buf.length < < 1, newcount). Ordinarily, buf.length << 1 returns double buf.length, which would always be much larger than newcount for a 2 byte write. Why was it not? The problem is that for all integers larger than Integer.MAX_INTEGER / 2, shifting left by one place causes overflow, setting the sign bit. The result is a negative integer, which is always less than newcount. So for all byte arrays larger than 1073741824 bytes (i.e. one GB), any write will cause the array to resize, and only to exactly the size required.

    (tags: bugs java jdk6 bytearrayoutputstream impala performance overflow)

  • Cory Doctorow on Thomas Piketty’s ‘Capital in the 21st Century’

    quite a leftie analysis

    (tags: history capitalism economics piketty capital finance taxation growth money cory-doctorow thomas-piketty)

  • ThreadSanitizer

    Google’s purify/valgrind-like concurrency checking tool: ‘As a bonus, ThreadSanitizer finds some other types of bugs: thread leaks, deadlocks, incorrect uses of mutexes, malloc calls in signal handlers, and more. It also natively understands atomic operations and thus can find bugs in lock-free algorithms. [...] The tool is supported by both Clang and GCC compilers (only on Linux/Intel64). Using it is very simple: you just need to add a -fsanitize=thread flag during compilation and linking. For Go programs, you simply need to add a -race flag to the go tool (supported on Linux, Mac and Windows).’

    (tags: concurrency bugs valgrind threadsanitizer threading deadlocks mutexes locking synchronization coding testing)

Posted in Uncategorized | Comments closed

Links for 2014-06-27

  • Sandymount Repair Cafe

    ‘A repair café brings together people with things that need fixin’ with people who have the skills to fix them in a social cafe style environment. It is an effort to move away from the throwaway culture that prevailed at the end of the twentieth century and move towards a more sustainable and enlightened approach to our relationship with consumer goods. Repair cafes are self organising events at a community level run by local volunteers with the support of local community groups, local agencies and other interested organisations. They are not-for-profit but not anti-profit and an important part of their goal is to promote local repair businesses and initiatives. is the online hub of a network of repair cafés across Ireland.’ Sounds interesting: says they’ll be doing it tomorrow from 2-5pm in Sandymount in Dublin.

    (tags: dublin sandymount repair fixing diy frugality repaircafe hardware)

  • Chef Vault

    A way to securely store secrets (auth details, API keys, etc.) in Chef

    (tags: chef storage knife authorisation api-keys security encryption)

  • Amazon EC2 Service Limits Report Now Available

    ‘designed to make it easier for you to view and manage your limits for Amazon EC2 by providing the latest information on service limits and links to quickly request limit increases. EC2 Service Limits Report displays all your service limit information in one place to help you avoid encountering limits on future EC2, EBS, Auto Scaling, and VPC usage.’

    (tags: aws ec2 vpc ebs autoscaling limits ops)

  • Delivery Notifications for Simple Email Service

    Today we are enhancing SES with the addition of delivery notifications. You can now elect to receive an Amazon SNS notification each time SES successfully delivers a message to a recipient’s email server. These notifications give you increased visibility into the mail delivery process. With today’s release, you can now track deliveries, bounces, and complaints, all via notification to the SNS topic or topics of your choice.

    (tags: delivery email smtp ses aws sns notifications ops)

  • How Emoji Get Lost In Translation

    I recently texted a friend to say how I was excited to meet her new boyfriend, and, because “excited” doesn’t look so exciting on an iPhone screen, I editorialized with what seemed then like an innocent “[dancer]“. (Translation: Can’t wait for the fun night out!) On an Android phone, I realized later, that panache would have been a put-down: The dancers become “[playboy bunny].” (Translation: You’re a Playboy bunny who gets around!)

    (tags: emoji icons graphics text speech phones)

Posted in Uncategorized | Comments closed

Links for 2014-06-26

Posted in Uncategorized | Comments closed

Links for 2014-06-25

Posted in Uncategorized | Comments closed

Links for 2014-06-24

Posted in Uncategorized | Comments closed

Links for 2014-06-23

  • Startup equity gotcha

    ‘Two months ago, an early Uber employee thought that he had found a buyer for his vested stock, at $200 per share. But when his agent tried to seal the deal, Uber refused to sign off on the transfer. Instead, it offered to buy back the shares for around $135 a piece, which is within the same price range that Google Ventures and TPG Capital had paid to invest in Uber the previous July. Take it or hold it.’ As rbranson on Twitter put it: ‘reminder that startup equity is basically worthless unless you’re a founder or investor, OR the company goes public.’

    (tags: startups uber stock stock-options shares share-option equity via:rbranson work)

Posted in Uncategorized | Comments closed

Links for 2014-06-20

Posted in Uncategorized | Comments closed

Links for 2014-06-19

Posted in Uncategorized | Comments closed

Links for 2014-06-18

Posted in Uncategorized | Comments closed

Links for 2014-06-17

  • FlatBuffers: Main Page

    A new serialization format from Google’s Android gaming team, supporting C++ and Java, open source under the ASL v2. Reasons to use it:

    Access to serialized data without parsing/unpacking – What sets FlatBuffers apart is that it represents hierarchical data in a flat binary buffer in such a way that it can still be accessed directly without parsing/unpacking, while also still supporting data structure evolution (forwards/backwards compatibility). Memory efficiency and speed – The only memory needed to access your data is that of the buffer. It requires 0 additional allocations. FlatBuffers is also very suitable for use with mmap (or streaming), requiring only part of the buffer to be in memory. Access is close to the speed of raw struct access with only one extra indirection (a kind of vtable) to allow for format evolution and optional fields. It is aimed at projects where spending time and space (many memory allocations) to be able to access or construct serialized data is undesirable, such as in games or any other performance sensitive applications. See the benchmarks for details. Flexible – Optional fields means not only do you get great forwards and backwards compatibility (increasingly important for long-lived games: don’t have to update all data with each new version!). It also means you have a lot of choice in what data you write and what data you don’t, and how you design data structures. Tiny code footprint – Small amounts of generated code, and just a single small header as the minimum dependency, which is very easy to integrate. Again, see the benchmark section for details. Strongly typed – Errors happen at compile time rather than manually having to write repetitive and error prone run-time checks. Useful code can be generated for you. Convenient to use – Generated C++ code allows for terse access & construction code. Then there’s optional functionality for parsing schemas and JSON-like text representations at runtime efficiently if needed (faster and more memory efficient than other JSON parsers).
    Looks nice, but it misses the language coverage of protobuf. Definitely more practical than capnproto.

    (tags: c++ google java serialization json formats protobuf capnproto storage flatbuffers)

  • AWS SDK for Java Client Configuration

    turns out the AWS SDK has lots of tuning knobs: region selection, socket buffer sizes, and debug logging (including wire logging).

    (tags: aws sdk java logging ec2 s3 dynamodb sockets tuning)

  • Behind the loom band

    The simple woven multicoloured bracelet has made Cheong Choon Ng, a Malaysian immigrant to the US, a dollar millionaire. He invented the “Rainbow Loom” after watching his daughters making bracelets with rubber bands.
    So, really, it’s his daughters that invented it. ;) My kids are massive fans. This is a 100% legit, Rubik’s-Cube-style craze. (via Conor O’Neill)

    (tags: via:conoro loom-bands rubber-bands toys crazes)

  • lookout/ngx_borderpatrol

    BorderPatrol is an nginx module to perform authentication and session management at the border of your network. BorderPatrol makes the assumption that you have some set of services that require authentication and a service that hands out tokens to clients to access that service. You may not want those tokens to be sent across the internet, even over SSL, for a variety of reasons. To this end, BorderPatrol maintains a lookup table of session-id to auth token in memcached.

    (tags: borderpatrol nginx modules authentication session-management web-services http web authorization)

  • Use of Formal Methods at Amazon Web Services

    Chris Newcombe, Marc Brooker, et al. writing about their experience using formal specification and model-checking languages (TLA+) in production in AWS:

    The success with DynamoDB gave us enough evidence to present TLA+ to the broader engineering community at Amazon. This raised a challenge; how to convey the purpose and benefits of formal methods to an audience of software engineers? Engineers think in terms of debugging rather than ‘verification’, so we called the presentation “Debugging Designs”. Continuing that metaphor, we have found that software engineers more readily grasp the concept and practical value of TLA+ if we dub it ‘Exhaustively-testable pseudo-code’. We initially avoid the words ‘formal’, ‘verification’, and ‘proof’, due to the widespread view that formal methods are impractical. We also initially avoid mentioning what the acronym ‘TLA’ stands for, as doing so would give an incorrect impression of complexity.
    More slides at ; proggit discussion at

    (tags: formal-methods model-checking tla tla+ programming distsys distcomp ebs s3 dynamodb aws ec2 marc-brooker chris-newcombe)

  • Call me maybe: RabbitMQ

    We used Knossos and Jepsen to prove the obvious: RabbitMQ is not a lock service. That investigation led to a discovery hinted at by the documentation: in the presence of partitions, RabbitMQ clustering will not only deliver duplicate messages, but will also drop huge volumes of acknowledged messages on the floor. This is not a new result, but it may be surprising if you haven’t read the docs closely–especially if you interpreted the phrase “chooses Consistency and Partition Tolerance” to mean, well, either of those things.

    (tags: rabbitmq network partitions failure cap-theorem consistency ops reliability distcomp jepsen)

  • Jump Consistent Hash: A Fast, Minimal Memory, Consistent Hash Algorithm

    ‘a fast, minimal memory, consistent hash algorithm that can be expressed in about 5 lines of code. In comparison to the algorithm of Karger et al., jump consistent hash requires no storage, is faster, and does a better job of evenly dividing the key space among the buckets and of evenly dividing the workload when the number of buckets changes. Its main limitation is that the buckets must be numbered sequentially, which makes it more suitable for data storage applications than for distributed web caching.’ Implemented in Guava. This is also noteworthy: ‘Google has not applied for patent protection for this algorithm, and, as of this writing, has no plans to. Rather, it wishes to contribute this algorithm to the community.’

    (tags: hashing consistent-hashing google guava memory algorithms sharding)

  • Bike Wheel Spoke ABS Safety Reflective Tube Reflector

    Available in blue, orange, and grey for $2.84 from the insanely-cheap China-based Also available: rim-based reflective stickers

    (tags: bikes cycling reflective safety dealextreme tat)

Posted in Uncategorized | Comments closed

Links for 2014-06-16

Posted in Uncategorized | Comments closed

Links for 2014-06-14

Posted in Uncategorized | Comments closed

Links for 2014-05-29

  • Tracedump

    a single application IP packet sniffer that captures all TCP and UDP packets of a single Linux process. It consists of the following elements: * ptrace monitor – tracks bind(), connect() and sendto() syscalls and extracts local port numbers that the traced application uses; * pcap sniffer – using information from the previous module, it captures IP packets on an AF_PACKET socket (with an appropriate BPF filter attached); * garbage collector – periodically reads /proc/net/{tcp,udp} files in order to detect the sockets that the application no longer uses. As the output, tracedump generates a PCAP file with SLL-encapsulated IP packets – readable by eg. Wireshark. This file can be later used for detailed analysis of the networking operations made by the application. For instance, it might be useful for IP traffic classification systems.

    (tags: debugging networking linux strace ptrace tracedump tracing tcp udp sniffer ip tcpdump)

  • You Are Not a Digital Native: Privacy in the Age of the Internet

    an open letter from Cory Doctorow to teen readers re privacy. ‘The problem with being a “digital native” is that it transforms all of your screw-ups into revealed deep truths about how humans are supposed to use the Internet. So if you make mistakes with your Internet privacy, not only do the companies who set the stage for those mistakes (and profited from them) get off Scot-free, but everyone else who raises privacy concerns is dismissed out of hand. After all, if the “digital natives” supposedly don’t care about their privacy, then anyone who does is a laughable, dinosauric idiot, who isn’t Down With the Kids.’

    (tags: children privacy kids teens digital-natives surveillance cory-doctorow danah-boyd)

  • Shutterbits replacing hardware load balancers with local BGP daemons and anycast

    Interesting approach. Potentially risky, though — heavy use of anycast on a large-scale datacenter network could increase the scale of the OSPF graph, which scales exponentially. This can have major side effects on OSPF reconvergence time, which creates an interesting class of network outage in the event of OSPF flapping. Having said that, an active/passive failover LB pair will already announce a single anycast virtual IP anyway, so, assuming there are a similar number of anycast IPs in the end, it may not have any negative side effects. There’s also the inherent limitation noted in the second-to-last paragraph; ‘It comes down to what your hardware router can handle for ECMP. I know a Juniper MX240 can handle 16 next-hops, and have heard rumors that a software update will bump this to 64, but again this is something to keep in mind’. Taking a leaf from the LB design, and using BGP to load-balance across a smaller set of haproxy instances, would seem like a good approach to scale up.

    (tags: scalability networking performance load-balancing bgp exabgp ospf anycast routing datacenters scaling vips juniper haproxy shutterstock)

  • Tron: Legacy Encom Boardroom Visualization

    this is great. lovely, silly, HTML5 dataviz, with lots of spinning globes and wobbling sines on a black background

    (tags: demo github wikipedia dataviz visualisation mapping globes rob-scanlan graphics html5 animation tron-legacy tron movies)

  • CockroachDB

    a distributed key/value datastore which supports ACID transactional semantics and versioned values as first-class features. The primary design goal is global consistency and survivability, hence the name. Cockroach aims to tolerate disk, machine, rack, and even datacenter failures with minimal latency disruption and no manual intervention. Cockroach nodes are symmetric; a design goal is one binary with minimal configuration and no required auxiliary services. Cockroach implements a single, monolithic sorted map from key to value where both keys and values are byte strings (not unicode). Cockroach scales linearly (theoretically up to 4 exabytes (4E) of logical data). The map is composed of one or more ranges and each range is backed by data stored in RocksDB (a variant of LevelDB), and is replicated to a total of three or more cockroach servers. Ranges are defined by start and end keys. Ranges are merged and split to maintain total byte size within a globally configurable min/max size interval. Range sizes default to target 64M in order to facilitate quick splits and merges and to distribute load at hotspots within a key range. Range replicas are intended to be located in disparate datacenters for survivability (e.g. { US-East, US-West, Japan }, { Ireland, US-East, US-West}, { Ireland, US-East, US-West, Japan, Australia }). Single mutations to ranges are mediated via an instance of a distributed consensus algorithm to ensure consistency. We’ve chosen to use the Raft consensus algorithm. All consensus state is stored in RocksDB. A single logical mutation may affect multiple key/value pairs. Logical mutations have ACID transactional semantics. If all keys affected by a logical mutation fall within the same range, atomicity and consistency are guaranteed by Raft; this is the fast commit path. Otherwise, a non-locking distributed commit protocol is employed between affected ranges. Cockroach provides snapshot isolation (SI) and serializable snapshot isolation (SSI) semantics, allowing externally consistent, lock-free reads and writes–both from an historical snapshot timestamp and from the current wall clock time. SI provides lock-free reads and writes but still allows write skew. SSI eliminates write skew, but introduces a performance hit in the case of a contentious system. SSI is the default isolation; clients must consciously decide to trade correctness for performance. Cockroach implements a limited form of linearalizability, providing ordering for any observer or chain of observers.
    This looks nifty. One to watch.

    (tags: cockroachdb databases storage georeplication raft consensus acid go key-value-stores rocksdb)

  • Tuning LevelDB

    good docs from Riak

    (tags: leveldb tuning performance ops riak)

  • Proof of burn – Bitcoin

    method for bootstrapping one cryptocurrency off of another. The idea is that miners should show proof that they burned some coins – that is, sent them to a verifiably unspendable address. This is expensive from their individual point of view, just like proof of work; but it consumes no resources other than the burned underlying asset. To date, all proof of burn cryptocurrencies work by burning proof-of-work-mined cryptocurrencies, so the ultimate source of scarcity remains the proof-of-work-mined “fuel”.

    (tags: bitcoin proof money mining cryptocurrency)

  • The programming error that cost Mt Gox 2609 bitcoins

    Digging into broken Bitcoin scripts in the blockchain. Fascinating:

    While analyzing coinbase transactions, I came across another interesting bug that lost bitcoins. Some transactions have the meaningless and unredeemable script: OP_IFDUP OP_IF OP_2SWAP OP_VERIFY OP_2OVER OP_DEPTH That script turns out to be the ASCII text script. Instead of putting the redemption script into the transaction, the P2Pool miners accidentally put in the literal word “script”. The associated bitcoins are lost forever due to this error.
    (via Nelson)

    (tags: programming script coding bitcoin mtgox via:nelson scripting dsls)

  • Moquette MQTT

    a Java implementation of an MQTT 3.1 broker. Its code base is small. At its core, Moquette is an events processor; this lets the code base be simple, avoiding thread sharing issues. The Moquette broker is lightweight and easy to understand so it could be embedded in other projects.

    (tags: mqtt moquette netty messaging queueing push-notifications iot internet push eclipse)

  • “Taking the hotdog”

    aka. lock acquisition. ex-Amazon-Dublin lingo, observed in the wild ;)

    (tags: language hotdog archie-mcphee amazon dublin intercom coding locks synchronization)

Posted in Uncategorized | Comments closed

Links for 2014-05-27

Posted in Uncategorized | Comments closed

Links for 2014-05-26

Posted in Uncategorized | Comments closed

Links for 2014-05-23

  • BPF – the forgotten bytecode

    ‘In essence Tcpdump asks the kernel to execute a BPF program within the kernel context. This might sound risky, but actually isn’t. Before executing the BPF bytecode kernel ensures that it’s safe: * All the jumps are only forward, which guarantees that there aren’t any loops in the BPF program. Therefore it must terminate. * All instructions, especially memory reads are valid and within range. * The single BPF program has less than 4096 instructions. All this guarantees that the BPF programs executed within kernel context will run fast and will never infinitely loop. That means the BPF programs are not Turing complete, but in practice they are expressive enough for the job and deal with packet filtering very well.’ Good example of a carefully-designed DSL allowing safe “programs” to be written and executed in a privileged context without security risk, or risk of running out of control.

    (tags: coding dsl security via:oisin linux tcpdump bpf bsd kernel turing-complete configuration languages)

  • Handmade Kitchen Goods from Makers & Brothers – Cool Hunting

    lovely kitchen-gear design from local-boys-made-good Makers & Brothers

    (tags: makers-and-brothers design crafts kitchen nyc terrazo chopping-boards)

Posted in Uncategorized | Comments closed

Links for 2014-05-22

  • ‘Monitoring and detecting causes of failures of network paths’, US patent 8,661,295 (B1)

    The first software patent in my name — couldn’t avoid it forever :(

    Systems and methods are provided for monitoring and detecting causes of failures of network paths. The system collects performance information from a plurality of nodes and links in a network, aggregates the collected performance information across paths in the network, processes the aggregated performance information for detecting failures on the paths, analyzes each of the detected failures to determine at least one root cause, and initiates a remedial workflow for the at least one root cause determined. In some aspects, processing the aggregated information may include performing a statistical regression analysis or otherwise solving a set of equations for the performance indications on each of a plurality of paths. In another aspect, the system may also include an interface which makes available for display one or more of the network topology, the collected and aggregated performance information, and indications of the detected failures in the topology.
    The patent describes an early version of Pimms, the network failure detection and remediation system we built for Amazon.

    (tags: amazon pimms swpats patents networking ospf autoremediation outage-detection)

Posted in Uncategorized | Comments closed

Links for 2014-05-20

Posted in Uncategorized | Comments closed

Links for 2014-05-19

Posted in Uncategorized | Comments closed

Links for 2014-05-16

Posted in Uncategorized | Comments closed

Links for 2014-05-15

Posted in Uncategorized | Comments closed

Links for 2014-05-14

Posted in Uncategorized | Comments closed

Links for 2014-05-13

Posted in Uncategorized | Comments closed

Links for 2014-05-12

Posted in Uncategorized | Comments closed

Links for 2014-05-10

Posted in Uncategorized | Comments closed

Links for 2014-05-09

Posted in Uncategorized | Comments closed

Links for 2014-05-08

Posted in Uncategorized | Comments closed

Links for 2014-05-07

Posted in Uncategorized | Comments closed

Links for 2014-05-06

  • Minimum Viable Block Chain

    Ilya Grigorik describes the design of the Bitcoin/altcoin block chain algorithm. Illuminating writeup

    (tags: algorithms bitcoin security crypto blockchain ilya-grigorik)

  • Docker Plugin for Jenkins

    The aim of the docker plugin is to be able to use a docker host to dynamically provision a slave, run a single build, then tear-down that slave. Optionally, the container can be committed, so that (for example) manual QA could be performed by the container being imported into a local docker provider, and run from there.
    The holy grail of Jenkins/Docker integration. How cool is that…

    (tags: jenkins docker ops testing ec2 hosting scaling elastic-scaling system-testing)

  • Simple Binary Encoding

    an OSI layer 6 presentation for encoding/decoding messages in binary format to support low-latency applications. [...] SBE follows a number of design principles to achieve this goal. By adhering to these design principles sometimes means features available in other codecs will not being offered. For example, many codecs allow strings to be encoded at any field position in a message; SBE only allows variable length fields, such as strings, as fields grouped at the end of a message. The SBE reference implementation consists of a compiler that takes a message schema as input and then generates language specific stubs. The stubs are used to directly encode and decode messages from buffers. The SBE tool can also generate a binary representation of the schema that can be used for the on-the-fly decoding of messages in a dynamic environment, such as for a log viewer or network sniffer. The design principles drive the implementation of a codec that ensures messages are streamed through memory without backtracking, copying, or unnecessary allocation. Memory access patterns should not be underestimated in the design of a high-performance application. Low-latency systems in any language especially need to consider all allocation to avoid the resulting issues in reclamation. This applies for both managed runtime and native languages. SBE is totally allocation free in all three language implementations. The end result of applying these design principles is a codec that has ~25X greater throughput than Google Protocol Buffers (GPB) with very low and predictable latency. This has been observed in micro-benchmarks and real-world application use. A typical market data message can be encoded, or decoded, in ~25ns compared to ~1000ns for the same message with GPB on the same hardware. XML and FIX tag value messages are orders of magnitude slower again. The sweet spot for SBE is as a codec for structured data that is mostly fixed size fields which are numbers, bitsets, enums, and arrays. While it does work for strings and blobs, many my find some of the restrictions a usability issue. These users would be better off with another codec more suited to string encoding.

    (tags: sbe encoding protobuf protocol-buffers json messages messaging binary formats low-latency martin-thompson xml)

  • Observations of an Internet Middleman

    That leaves the remaining six [consumer ISPs peering with Level3] with congestion on almost all of the interconnect ports between us. Congestion that is permanent, has been in place for well over a year and where our peer refuses to augment capacity. They are deliberately harming the service they deliver to their paying customers. They are not allowing us to fulfil the requests their customers make for content. Five of those congested peers are in the United States and one is in Europe. There are none in any other part of the world. All six are large Broadband consumer networks with a dominant or exclusive market share in their local market. In countries or markets where consumers have multiple Broadband choices (like the UK) there are no congested peers.
    Amazing that L3 are happy to publish this — that’s where big monopoly ISPs have led their industry.

    (tags: net-neutrality networking internet level3 congestion isps us-politics)

  • interview with Google VP of SRE Ben Treynor

    interviewed by Niall Murphy, no less ;). Some good info on what Google deems important from an ops/SRE perspective

    (tags: sre ops devops google monitoring interviews ben-treynor)

Posted in Uncategorized | Comments closed

Links for 2014-05-02

  • Faster BAM Sorting with SAMtools and RocksDB

    Now this is really really clever. Heap-merging a heavyweight genomics format, using RocksDB to speed it up.

    There’s a problem with the single-pass merge described above when the number of intermediate files, N/R, is large. Merging the sorted intermediate files in limited memory requires constantly reading little bits from all those files, incurring a lot of disk seeks on rotating drives. In fact, at some point, samtools sort performance becomes effectively bound to disk seeking. [...] In this scenario, samtools rocksort can sort the same data in much less time, using no more memory, by invoking RocksDB’s background compaction capabilities. With a few extra lines of code we configure RocksDB so that, while we’re still in the process of loading the BAM data, it runs additional background threads to merge batches of existing sorted temporary files into fewer, larger, sorted files. Just like the final merge, each background compaction requires only a modest amount of working memory.
    (via the RocksDB facebook group)

    (tags: rocksdb algorithms sorting leveldb bam samtools merging heaps compaction)

  • Coding For Life (Battery Life, That Is)

    great presentation on Android mobile battery life, and what to avoid

    (tags: presentations via:sergio android mobile battery battery-life 3g wifi gprs hardware)

  • Oisin’s mobile app release checklist

    ‘This form is to document the testing that has been done on each app version before submitting to the App Store. For each item, indicate Yes if the testing has been done, Not Applicable if the testing does not apply (eg testing audio for an app that doesn’t play any), or No if the testing has not been done for another reason.’

    (tags: apps checklists release coding ios android mobile ohurley)

  • “A New Data Structure For Cumulative Frequency Tables”

    paper by Peter M Fenwick, 1993. ‘A new method (the ‘binary indexed tree’) is presented for maintaining the cumulative frequencies which are needed to support dynamic arithmetic data compression. It is based on a decomposition of the cumulative frequencies into portions which parallel the binary representation of the index of the table element (or symbol). The operations to traverse the data structure are based on the binary coding of the index. In comparison with previous methods, the binary indexed tree is faster, using more compact data and simpler code. The access time for all operations is either constant or proportional to the logarithm of the table size. In conjunction with the compact data structure, this makes the new method particularly suitable for large symbol alphabets.’ via Jakob Buchgraber, who’s implementing it right now in Netty ;)

    (tags: netty frequency-tables data-structures algorithms coding binary-tree indexing compression symbol-alphabets)

Posted in Uncategorized | Comments closed

Links for 2014-05-01

Posted in Uncategorized | Comments closed

Links for 2014-04-30

Posted in Uncategorized | Comments closed

Links for 2014-04-29

  • ‘Pickles & Spores: Improving Support for Distributed Programming in Scala

    ‘Spores are “small units of possibly mobile functional behavior”. They’re a closure-like abstraction meant for use in distributed or concurrent environments. Spores provide a guarantee that the environment is effectively immutable, and safe to ship over the wire. Spores aim to give library authors some confidence in exposing functions (or, rather, spores) in public APIs for safe consumption in a distributed or concurrent environment. The first part of the talk covers a simpler variant of spores as they are proposed for inclusion in Scala 2.11. The second part of the talk briefly introduces a current research project ongoing at EPFL which leverages Scala’s type system to provide type constraints that give authors finer-grained control over spore capturing semantics. What’s more, these type constraints can be composed during spore composition, so library authors are effectively able to propagate expert knowledge via these composable constraints. The last part of the talk briefly covers Scala/Pickling, a fast new, open serialization framework.’

    (tags: pickling scala presentations spores closures fp immutability coding distributed distcomp serialization formats network)

  • BBC News – Microsoft ‘must release’ data held on Dublin server

    Messy. I can’t see this lasting beyond an appeal.

    Law enforcement efforts would be seriously impeded and the burden on the government would be substantial if they had to co-ordinate with foreign governments to obtain this sort of information from internet service providers such as Microsoft and Google, Judge Francis said. In a blog post, Microsoft’s deputy general counsel, David Howard, said: “A US prosecutor cannot obtain a US warrant to search someone’s home located in another country, just as another country’s prosecutor cannot obtain a court order in her home country to conduct a search in the United States. “We think the same rules should apply in the online world, but the government disagrees.”

    (tags: microsoft regions law us-law privacy google cloud international-law surveillance)

  • Russia passes bill requiring bloggers to register with government

    A bill passed by the Russian parliament on Tuesday says that any blogger read by at least 3,000 people a day has to register with the government telecom watchdog and follow the same rules as those imposed by Russian law on mass media. These include privacy safeguards, the obligation to check all facts, silent days before elections and loose but threatening injunctions against “abetting terrorism” and “extremism.”
    Russian blogging platforms have responded by changing view-counter tickers to display “2500+” as a max.

    (tags: russia blogs blogging terrorism extremism internet regulation chilling-effects censorship)

Posted in Uncategorized | Comments closed

Links for 2014-04-28

Posted in Uncategorized | Comments closed

Links for 2014-04-25

Posted in Uncategorized | Comments closed