Distributed Streams Algorithms for Sliding Windows [PDF]
‘Massive data sets often arise as physically distributed, parallel data streams, and it is important to estimate various aggregates and statistics on the union of these streams. This paper presents algorithms for estimating aggregate functions over a “sliding window” of the N most recent data items in one or more streams. […] Our results are obtained using a novel family of synopsis data structures called waves.’
(tags: waves papers streaming algorithms percentiles histogram distcomp distributed aggregation statistics estimation streams)
good blog post on histogram-estimation stream processing algorithms
After reviewing several dozen papers, a score or so in depth, I identified two data structures that appear to enable us to answer these recency and frequency queries: exponential histograms (from “Maintaining Stream Statistics Over Sliding Windows” by Datar et al.) and waves (from “Distributed Streams Algorithms for Sliding Windows” by Gibbons and Tirthapura). Both of these data structures are used to solve the so-called counting problem, the problem of determining, with a bound on the relative error, the number of 1s in the last N units of time. In other words, the data structures are able to answer the question: how many 1s appeared in the last n units of time within a factor of Error (e.g., 50%). The algorithms are neat, so I’ll present them briefly.
(tags: streams streaming stream-processing histograms percentiles estimation waves statistics algorithms)
Timelike 2: everything fails all the time
Fantastic post on large-scale distributed load balancing strategies from @aphyr. Random and least-conns routing comes out on top in his simulation (although he hasn’t yet tried Marc Brooker’s two-randoms routing strategy)
(tags: via:hn routing distributed least-conns load-balancing round-robin distcomp networking scaling)
Marc Brooker’s “two-randoms” load balancing approach
Marc Brooker on this interesting load-balancing algorithm, including simulation results:
Using stale data for load balancing leads to a herd behavior, where requests will herd toward a previously quiet host for much longer than it takes to make that host very busy indeed. The next refresh of the cached load data will put the server high up the load list, and it will become quiet again. Then busy again as the next herd sees that it’s quiet. Busy. Quiet. Busy. Quiet. And so on. One possible solution would be to give up on load balancing entirely, and just pick a host at random. Depending on the load factor, that can be a good approach. With many typical loads, though, picking a random host degrades latency and reduces throughput by wasting resources on servers which end up unlucky and quiet. The approach taken by the studies surveyed by Mitzenmacher is to try two hosts, and pick the one with the least load. This can be done directly (by querying the hosts) but also works surprisingly well on cached load data. […] Best of 2 is good because it combines the best of both worlds: it uses real information about load to pick a host (unlike random), but rejects herd behavior much more strongly than the other two approaches.
Having seen what Marc has worked on, and written, inside Amazon, I’d take this very seriously… cool to see he is blogging externally too.(tags: algorithm load-balancing distcomp distributed two-randoms marc-brooker least-conns)
Can regular expressions parse HTML?
‘a summary of the main points: The “regular expressions” used by programmers have very little in common with the original notion of regularity in the context of formal language theory. Regular expressions (at least PCRE) can match all context-free languages. As such they can also match well-formed HTML and pretty much all other programming languages. Regular expressions can match at least some context-sensitive languages. Matching of regular expressions is NP-complete. As such you can solve any other NP problem using regular expressions.’
(tags: compsci regexps regular-expressions programming np-complete chomsky-grammar context-free languages)