How We Get Almost Bankrupted Because Of A Documentation Typo
Picture this: it's September 2022, and we at Qovery are happily chugging along, serving nearly 20,000 developers around the globe with our self-service infrastructure platform. Everything's coming up roses until one day, a tiny typo in our API documentation threatened to turn our world upside down.
Romaric PhilogèneMay 13, 2023 · 3 min read
CEO and co-founder of Qovery. Romaric has 10+ years of experience in R&D. From the Ad-Tech to the financial industry, he has deep expertise in highly-reliable and performant systems.See all articles
Here's what happened: one of our customers was working on an API integration and needed to provision an AWS RDS instance. They turned to our documentation for guidance, as you do. In our guide, we had a little section about disk space.
This is where things went sideways. Due to a pesky misprint, our documentation said that the disk unit was in MB (megabytes) instead of GB (gigabytes).
[...] the smallest details, like the wording in our documentation, can have a huge negative impact
So, our customer, blissfully unaware of the mistake, put in '10000', thinking they were setting up a modest 10GB of disk space. But oh boy, were they in for a surprise! They'd actually commissioned a mammoth 10TB.
The aftermath was a bill that would make anyone's eyes water: $3,713.76. We goofed up, so we owned up and footed the bill. But the money wasn't the biggest issue. What really made us gulp was thinking about what might have happened if other customers had made the same mistake.
Think about it: we had almost 20,000 developers using Qovery at that time. Sure, most of them stick with the default settings, but what if they'd decided to switch things up? If they'd run into the same error, we could have been looking at a financial catastrophe, maybe even bankruptcy.
This little blunder was a wake-up call. It made us realize that even the smallest details, like the wording in our documentation, can have a huge impact. It also reminded us that every change, no matter how trivial it seems, needs to be reviewed and double-checked.
So, what did we do about it? We took action. We've introduced a thorough review process for any updates to our API and user documentation. Sure, it's partly about avoiding any more 'Oops' moments that hit us in the wallet, but it's also about making sure our customers get the most accurate information.
Reflections and Learnings
Looking back, September 2022 was a wild ride for us at Qovery. It was a bit scary, a bit humbling, but above all, it was a learning experience. It reminded us to slow down, pay attention to the little things, and always hold ourselves accountable.
Our story is a gentle reminder to everyone in the tech industry. Always pay close attention to details, stay vigilant, and put your customers first. At the end of the day, these aren't just good practices—they're what keep us in business and out of trouble.
In the end, our little 'Oops' moment taught us a big lesson. And who knows, it might just save someone else from making the same mistake. So here's to learning, growing, and always finding the humor in our human errors, no matter how colossal they might seem.
Deliver Self-Service Infrastructure, Faster!
Qovery turns your existing IaC infrastructure and Kubernetes manifests into repeatable blueprints for complete environments.Try it out now!