Inside DataDog’s $5M Outage (Real-World Engineering Challenges #8)
Reading between the lines: Ubuntu unattended-upgrades were left enabled, and as a result a fix which required a full reboot was rolled out swiftly and globally, including a key fleet of network control hosts, regardless of any normal deployment phasing rules. This broke all regions and AZs within a 1 hour period. whoopsie
(tags: postmortem datadog ubuntu fail unattended-upgrades systemd bugs availability outages)