I recently wrote an article for InfoWorld entitled 7 Signs You're Doing DevOps Wrong. It was the first time I've written about something that didn't have PowerShell in the article anywhere. It was a fun article to write and hit on concepts that I'm passionate about. It hit a nerve to a lot of people and I recently got an email from someone about one of the topics I touched on; not considering failure unacceptable.
The question came from a someone at a highly-regarded financial institution. He was concerned that in a financial environment lots of money can be lost if something goes wrong in production. He was polite but I could sense a tone of disagreement when I basically said it's OK to fail. Here was my reply to him:
I made it clear in the article that not all companies can embrace all DevOps methodologies.? Failure in production does mean more to some companies than it does others. I can definitely see the the impact to a company like Acme Financial Firm. However, humans are fallible. We make mistakes. There inevitably will be a time when someone makes a mistake that brings down some core component of production. A core part of LEAN which DevOps borrowed a lot of principles from is that it's important not to blame individuals for systemic problems.
If a developer, for example, pushes faulty code to production and breaks something that's not the fault of the developer unless his actions were malicious. Why? It's because automated tests should have been introduced and ran beforehand. If any code didn't pass those tests, the system itself would prevent it. If the tests were ran and the code still got into production then this is a clear indicator of more extensive tests that are needed. That is a systemic problem. I'm not saying that the developer should be spoken to in a casual manner about being more careful but, in some sense, he should also be thanked for finding a potential problem in the system.
Join the Jar Tippers on Patreon
It takes a lot of time to write detailed blog posts like this one. In a single-income family, this blog is one way I depend on to keep the lights on. I'd be eternally grateful if you could become a Patreon patron today!Become a Patron!