SRE Wisdom From the Trenches
Key takeaways from Ryan Kitchens talk "How Did Things Go Right? Learning More from Incidents" based on his experience at Netflix: Failure is ever-present in modern software systems Success isn’t necessarily the absence of failure, and having 99.999% uptime is practically meaningless if the users are unable to use the system as they intend. Safety, great performance, and sources of resilience do not come from the absence of failure but rather the presence of adaptive capacity . Moving from "Why did things go wrong?" ask "How did things go right?" is a challenging but valuable exercise. Find out - What's going on when it seems like nothing is happening? When failure does occur, what's going to keep it from being worse? How do teams adapt successfully when preventative techniques fail? How should we prioritize the effort to develop systems that help us safely manage the consequences of failure? Recovery is better than prevention. An incident occurs w...