DataSift Architecture: Realtime Datamining at 120,000 Tweets Per Second
250 million tweets per day, 30-node HBase cluster, 400TB of storage, Kafka and 0mq. This is from 2011, hence this dated line: ‘for a distributed application they thought AWS was too limited, especially in the network. AWS doesn’t do well when nodes are connected together and they need to talk to each other. Not low enough latency network. Their customers care about latency.’ (Nowadays, it would be damn hard to build a lower-latency network than that attached to a cc2.8xlarge instance.)
(tags: datasift architecture scalability data twitter firehose hbase kafka zeromq)
Breaking the 1000 ms Time to Glass Mobile Barrier [slides]
Great presentation from Google on HTML5 CSS+JS render speed, 3G/4G network latency, etc. (via John G)
(tags: google slides 3g 4g lte networking telcos telecom css js html5 web via:jg)
Lucene 4 – Revisiting Problems For Speed [slides]
a Presentation from Simon Willnauer on optimization work performed on Lucene in 2011. The most interesting stuff here is the work done to replace an O(n^2) FuzzyQuery fuzzy-match algorithm with a FSM trie is extremely cool — benchmarked at 214 times faster!
(tags: benchmarks slides lucene search fuzzy-matching text-matching strings algorithms coding fsm tries)
Microsoft Code Digger extension
Miguel de Icaza says it’s witchcraft — I’m inclined to agree:
Code Digger analyzes possible execution paths through your .NET code. The result is a table where each row shows a unique behavior of your code. The table helps you understand the behavior of the code, and it may also uncover hidden bugs. Through the new context menu item “Generate Inputs / Outputs Table” in the Visual Studio editor, you can invoke Code Digger to analyze your code. Code Digger computes and displays input-output pairs. Code Digger systematically hunts for bugs, exceptions, and assertion failures.
(tags: testing constraint-solving solver witchcraft magic dot-net coding tests code-digger microsoft)