What goes bump in the middle of the night? If it is a cloud service or database server this week, the administrator could truly have a scary Halloween. This week, in honor of our favorite holiday, The New Stack offers not only one but two posts that compile some of the most frightening stories from the cloud native computing community.
One of the posts was contributed by Adam LaGreca, the founder of 10KMedia — a boutique PR agency for B2B DevOps. He culled his Rolodex to unearth six scary outage stories from chief technology officers. “Full-on site outages are horrible — but they don’t make your skin crawl the same way that the random, unpredictable failures really can,” observed Tina Huang, CTO of Transposit. She recalled a tale from her time at Twitter, where a newly installed library failed to parse an individual character within a cookie string. “So every time you logged back in, you were rolling the dice on getting bit by this pesky bug, and you couldn’t be cured without the wizardly powers to reset your cookies on a phone,” she said. Imagine trying to debug this error, on a platform as large as Twitter’s? Charity Majors of Honeycomb, Matthew Fornaciari of Gremlin, Liran Haimovitch of Rookout, Daniel “Spoons” Spoonhower of Lightstep, Lee Liu of LogDNA also weigh in with their own tales of woe, in this highly-entertaining and informative post.
One of our sponsors, Thundra, also compiled for us a highly illuminating post of cloud horror stories as well, this one focusing on Kubernetes. Here we read about resource-hungry pods that crash a production web site, a failed upgrade from unresponsive webhooks, Bitcoin zombies that consume all the available CPU cycles, and ghost pods that block further expansion of a cluster. As fun as these tales are (safely in hindsight), “It’s essential to learn from these stories and to analyze your Kubernetes landscapes. If you avoid these mistakes, you’ll have more reliable, scalable and robust Kubernetes setups,” wrote Thundra co-founder Serkan Özal.
So who this year will dress up as K8s? If you do, send pix for the newsletter.