we determined that calls to time.Now() or time.Since(time.Time) in Golang were taking about 200 times longer on new servers than they were on old servers. We found similar impacts in Python, which lead us to the clocksource. It turns out that in our Grub boot parameters for AL2, we had added clocksource=hpet about two years ago, but all of our servers launched prior to 6/17 had a clocksource of kvm-clock, apparently ignoring the Grub config. Servers launched after 6/17 (with the same AMI, Grub config, everything) were honoring the specified clocksource, which caused our performance issue.
Amazing level of detail on how the Apollo mission control room operated! (plus more from Ken Shirriff at https://twitter.com/kenshirriff/status/1409241533757345792 )
It seems that ‘symptoms vary slightly based on whether you’re fully vaccinated, half vaccinated or unvaccinated.’