The log/event processing pipeline you can’t have – apenwarr
So good. Apenwarr knows how to design a system.
Simple things don’t break. Our friends on the “let’s use structured events to make metrics” team streamed those events straight into a database, and it broke all the time, because databases have configuration options and you inevitably set those options wrong, and it’ll fall over under heavy load, and you won’t find out until you’re right in the middle of an emergency and you really want to see those logs. Or events.
(tags: logging scalability klog kernel log-processing events embedded ops)
[1902.04023] Computing Extremely Accurate Quantiles Using t-Digests
‘We present on-line algorithms for computing approximations of rank-based statistics that give high accuracy, particularly near the tails of a distribution, with very small sketches. Notably, the method allows a quantile q to be computed with an accuracy relative to max(q,1?q) rather than absolute accuracy as with most other methods. This new algorithm is robust with respect to skewed distributions or ordered datasets and allows separately computed summaries to be combined with no loss in accuracy. An open-source Java implementation of this algorithm is available from the author. Independent implementations in Go and Python are also available.’ (via Tony Finch)
(tags: java go python open-source quantiles percentiles approximation statistics sketching algorithms via:fanf)
81 Megapixel image of the moon
I took nearly 50,000 images of the night sky to make an 81 Megapixel image of Tuesday’s moon. Uncompressed image linked in the comments. [OC]
via Elliot(tags: via:elliot art moon astronomy photography hd)