Druid | How We Scaled HyperLogLog: Three Real-World Optimizations
3 optimizations Druid.io have made to the HLL algorithm to scale it up for production use in Metamarkets: compacting registers (fixes a bug with unions of multiple HLLs); a sparse storage format (to optimize space); faster lookups using a lookup table.
(tags: druid.io metamarkets scaling hyperloglog hll algorithms performance optimization counting estimation)
HyperLogLog – Intersection Arithmetic
‘In general HLL intersection in StreamLib works. |A INTERSECT B| = |A| + |B| – |A UNION B|. Timon’s article on intersection is important to read though. The usefulness of HLL intersection depends on the features of the HLLs you are intersecting.’
(tags: hyperloglog hll hyperloglogplus streamlib intersections sets estimation algorithms)
Structural Integrity | 99% Invisible
‘The student (who has since been lost to history) was studying Citicorp Center as part of his thesis and had found that the building was particularly vulnerable to quartering winds (winds that strike the building at its corners). Normally, buildings are strongest at their corners, and it’s the perpendicular winds (winds that strike the building at its face) that cause the greatest strain. But this was not a normal building. LeMessurier had accounted for the perpendicular winds, but not the quartering winds. He checked the math, and found that the student was right. He compared what velocity winds the building could withstand with weather data, and found that a storm strong enough to topple Citicorp Center hits New York City every 55 years. But that’s only if the tuned mass damper, which keeps the building stable, is running. LeMessurier realized that a major storm could cause a blackout and render the tuned mass damper inoperable. Without the tuned mass damper, LeMessurier calculated that a storm powerful enough to take out the building his New York every sixteen years.’
(tags: william-lemessurier architecture danger risk buildings nyc citicorp-center wind mass-dampers physics)
Linode announces new instance specs
‘TL;DR: SSDs + Insane network + Faster processors + Double the RAM + Hourly Billing’
(tags: hosting linode ssd performance linux ops datacenters)
-
Fcron is a scheduler. It aims at replacing Vixie Cron, so it implements most of its functionalities. But contrary to Vixie Cron, fcron does not need your system to be up 7 days a week, 24 hours a day : it also works well with systems which are running only occasionnally (contrary to anacrontab). In other words, fcron does both the job of Vixie Cron and anacron, but does even more and better :)) …
Thanks Craig!(tags: via:chughes cron fcron unix linux ops scheduler automation scripts)
-
They’ve done the classic website-redesign screwup — omitted redirects from the old URLs.
Sam Silverwood-Cope, director of Intelligent Positioning, said: “They’ve ignored the legacy of the old Ryanair.com. It’s quite startling. They are doing it just before their busiest time of the year.” A change in [URLs] without proper redirects means many results found by Google now simply return error pages, he added. “Unless redirects get put in pretty soon, the position is going to get worse and worse.”
(tags: ryanair inept fail funny via:christinebohan web google search redirects)
-
Scarfolk is a town in North West England that did not progress beyond 1979. Instead, the entire decade of the 1970s loops ad infinitum. Here in Scarfolk, pagan rituals blend seamlessly with science; hauntology is a compulsory subject at school, and everyone must be in bed by 8pm because they are perpetually running a slight fever. “Visit Scarfolk today. Our number one priority is keeping rabies at bay.” For more information please reread.
(tags: scarfolk 1970s england history funny humour public-information pagan morbid)
-
OpenBSD are going wild ripping out “arcane VMS hacks” in an attempt to render OpenSSL’s source code comprehensible, and finding amazing horrors like this: ‘Well, even if time() isn’t random, your RSA private key is probably pretty random. Do not feed RSA private key information to the random subsystem as entropy. It might be fed to a pluggable random subsystem…. What were they thinking?!’
(tags: random security openssl openbsd coding horror rsa private-keys entropy)