Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- I can't tell if the article is describing how complex systems fail or if they are using failure characteristics to define complex systems.
- It’s more about failure. It’s right there on top.by shash
- Economists would be wise to read this. The world economy is now a rather complex system, and many of the safeguards from previous generations have been removed in the interests of "efficiency".by RachelF
- This is very important work. If it interests you, and you want to dig further, it's broadly called "Safety II". You probably want to read Erik Hollnagel [1] or Sydney Dekker. I put together a presentation once about how this translates to IT security [2]. But broadly:
Safety I Learn from Errors, Safety defined by absence,Reactive approach, Understanding what goes wrong, Accident causation models, Avoidance of errors, Reducing losses
Safety II Learn from successes, Safety defined by presence, Proactive approach, Understanding what goes right, Repeat what goes right, Enforce successful behaviours, Create new processes based on successful behaviour
If anything, think long and hard about this: "Post-accident attribution to a ‘root cause’ is fundamentally wrong." It's a true statement. If you truly understand this sentence, you have mastered Safety II to the level that a light bulb will come on, and you won't be able to un-see it. This light bulb is a curse, a beautiful curse.
[1] https://safety4sea.com/cm-safety-i-vs-safety-ii-an-overview [2] https://www.msoos.org/largefiles/safety2.pdf
by zero_k - "Catastrophe requires multiple failures – single point failures are not enough."
I disagree with this point. From what I have seen and experienced, there are so many SPOFs in the real-world complex systems that do not have adequate defense or even any defense at all. The only reason catastrophe didn't occur (often) is only because the natural failure rate is extremely low. Like the probability of air traffic controller making a critical mistake is very low, maybe once every few years, but when it happens, the SPOF will almost certainly cause a disaster, and the system (ie. the way air traffic is controlled today) has been around for decades and is so outdated but the cost of changing the entire process around the world is simply way too high so we just take this as "normal".
by elisbce - The definitive work on this topic is Normal Accidents, with a modern retelling in Meltdown.
https://en.wikipedia.org/wiki/Normal_Accidents
https://en.wikipedia.org/wiki/Meltdown_(Clearfield_and_Tilcs...
by squirrel - Whenever I see this [excellent] paper, I am always struck by the first sentence, in the first section:
> All of the interesting systems (e.g. transportation, healthcare, power generation) are inherently and unavoidably hazardous by THE own nature.
(Emphasis mine)
I am not sure if that's a typo, or some writing convention that my uneducated mind can't grok.
- Is it not merely an example of such failure?by DJBunnies
- The operators know complex systems work. Words I've spoken, heard, or both: "Just rerun it, works the second time." "That's been failing for a while now. Just ignore the alert." "This procedure is old. We've added a step 64b in pencil. I can't remember why." "Is it meant to sound like that?" "What's the tape for?" "Careful you don't bump this button." "The database name has changed, the post-it has the new one. That's a '1' by the way." "Accept the failure, keep going." "Because you're the person touching the system the most, you're the most likely person to break it." "I know it says call on failure, but they never answer."by stAInley
- I may have shared this before on a different submission: John Gall’s books are really good on this topic: General Systemantics [https://en.wikipedia.org/wiki/Systemantics]by feyman_r
- Love Systemantics. That was one of the first systems books I read and where I felt like I found someone practiced who understood how systems actually function (a realistic, honest view that systems always fail, it's just a question of when, why, and how).by rglover
- Gall's law is amongst my favorite ones and with decades of experience in software development, I have to say it holds absolutely true:
> “A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.” — John Gall, Systemantics (1975)
- > Failure free operations require experience with failure.
This is why we created Chaos Engineering. By constantly forcing failure, it made us always create systems in defense of that failure, and gave us great data on where the tipping point is for different systems within a particular failure mode.
by jedberg - I've always struggled to apply this to the systems I work on. If the system fails, someone potentially dies, though in practice they've never been more than hospitalized. To avoid that, huge amounts of effort are expended on failure modelling and testing, but that doesn't eliminate unknown unknowns. That discrepancy has made front page news a couple times.
- It's much more universal and complicated than that. One of the big issues schools and pedagogy in general struggle with is that our environment is far too safe in too many ways. For kids there is too few ways to learn from failures. Attempts to solve the issue look often like "Hey, kids, let's fall over now all at once in safest way possible and learn from it!". But it doesn't work at all. Real failures have to be unexpected, related to your decisions and really hurt so that you can learn from them.
PS. Btw, I am certain that this is the main cause of the mental health crisis amongst young people.
by obscurette - "The system continues to function because it contains so many redundancies and because people can make it function, despite the presence of many flaws. After accident reviews nearly always note that the system has a history of prior ‘proto-accidents’ that nearly generated catastrophe. Arguments that these degraded conditions should have been recognized before the overt accident are usually predicated on naïve notions of system performance. System operations are dynamic, with components (organizational, human, technical) failing and being replaced continuously."
This very much resembles Admiral Cloudberg's write-up of the National Airport collision:
"No human being can look at a complex system and predict with any degree of accuracy how exactly it will fail. But with sufficient data, prediction becomes possible because of something known in the occupational safety field as Heinrich’s Law, which states that there are approximately 300 “near misses” for every serious accident; or as the latest version of the law puts it, each fatal accident is accompanied by about 3,000 near misses and about 30,000 “at-risk behaviors.”[39] Statistically, hundreds or thousands of at-risk behaviors and near misses are likely to occur prior to the first fatal accident, providing an opportunity to identify the risk before lives are lost."
https://admiralcloudberg.medium.com/reaping-the-whirlwind-in...
by anonymars - > After accident reviews nearly always note that the system has a history of prior ‘proto-accidents’ that nearly generated catastrophe. Arguments that these degraded conditions should have been recognized before the overt accident are usually predicated on naïve notions of system performance.
What does that mean? If you knew that the precursors were why did you not set alerts?
by esafak - I'm a broken record on how important I think this document is, and that it's hard to appreciate it until you've had extended experience with complex systems actually failing.
The most commonly cited subtext or thrust of it is that "root cause analysis", at least on complex systems, is a fools errand. Something goes wrong, say, in a distributed lock system, and your whole deployment system enters a metastable failure state. Naturally, the "root cause" seems like lock system resiliency. But definitionally a metastable failure is one that persists after the inciting condition is resolved. Now you have two "root causes", the lock failure and the metastability of the deployment system fault. Keep looking and you'll find more.
But to me the biggest brick to the forehead in this piece is further observation that random things are failing all the time in any complex system. "Complex systems run in degraded mode". Resilient components are good, but it's the resiliency of the overall process that orchestrates the whole system that determines whether things are going to blow up.
All practitioner actions are gambles. I should have that inked somewhere.
by tptacek - “Root cause analysis”, like “single point of failure”, is often misunderstood. A root cause analysis often finds multiple root causes. And a system can have multiple single points of failure.by khuston
- Check out the book called, The Logic of Failure. I think you’ll like itby _boffin_
- Root Cause Analysis is an organizational ritual that makes people feel like they're doing something. It's more about politics than about engineering. Not only is the process not useful- more often than not the correct root cause isn't even correctly identified.
Basically a poorly designed system is going to fail somewhere. I like to think about it either as the part of the iceberg that's visible or my other analogy is lighting doesn't strike twice at the same place. A robust/well designed system is just very different than a brittle/poorly designed system. The brittle system will exhibit random failures and trying to chase them is counterproductive.
We do have systems that are fairly complex and reliable. Take an internal combustion engine as one example or an entire car or an airplane (well, one of the well designed ones). Then we build bigger systems on top of that, a city's transportation system, where we do get to scales where things "fail" all the time. Yes, some bus somewhere or some train somewhere doesn't get to a station at the correct time, because a bus breaks down or heavy traffic or whatnot. But still a well designed system is robust to those. The parts we build the systems out of are well understood and so the resulting reliability can also be understood.
With software specifically we tend to not design and build that well. We throw stuff together. Then we're surprised when they fail. And we make excuses by saying "complex systems". I often feel it's our lack of discipline and skill vs. other domains and not really that our systems are that much more complex.
EDIT: Some might argue that mechanical systems like engines are just fundamentally simpler than software. But look at something like a bearing. Just one bearing is an incredibly complex system. Bearing failure is a statistical event, some bearings fail sooner, some fail later. There are centuries of know-how in the bearings used in said engine. Mechanical engineers generally use well understood components/designs and build in margin to meet the requirements of the engine they're designing. In software we often re-invent the "bearing" or we'll pick some new unproven design for a "bearing" and we'll generally build without a good understanding of the performance or failure modes of our "bearings" and without a known pattern as to how to use them and when to use them. And then we're surprised when our "car" fails in weird ways. We also don't test to the same standard that a mechanical engineer would use for a new design. They would have a room full of widgets doing ten million cycles before they accept the design as part of a larger system. We "yolo" it...
by YZF - Same, I think this document is fundamental to our industry and understanding complex systems. We have it as a quarterly reminder to review in our Slack. I give it to everyone that makes it past the first couple of interviews when applying and tout its usefulness for thinking through systems design all the time. RIP Dr. Cook.by asenchi
- > I'm a broken record on how important I think this document is, and that it's hard to appreciate it until you've had extended experience with complex systems actually failing.
If you want a bit of cheese to go with that wine, this article pairs nicely with The Grug-Brained Developer: https://grugbrain.dev/
- > "root cause analysis", at least on complex systems, is a fools errand.
Not when properly understood--which, to be fair, it often isn't. The proper purpose of root cause analysis is to answer the question: does this failure mean I need to change something about this system? If the answer turns out to be yes, the thing that you need to change gets called the "root cause" of the failure. Of course that's a misnomer, strictly speaking, but that doesn't mean you answered the actual question wrong.
by pdonis - >The most commonly cited subtext or thrust of it is that "root cause analysis", at least on complex systems, is a fools errand.
It's only an issue when people take that phrase very literally. People have common sense to understand that things have multiple causes and a chain of events. NASA has "Root Cause Analysis" (singular) all over various official documentation and it doesn't stop them from understanding that the failed O-rings were not the single root cause of the Challenger explosion. Another cause was management normalizing the deviations of previous unsafe datapoints of prior launches which let them greenlight the launch in freezing temperatures. Another cause was the unrealistic flight schedules which can subconsciously pressure management into normalizing dangerous deviations. It wasn't The Rogers Commission that found the multiple causes; it was NASA engineers and management themselves explaining the multiple causes as they were interviewed by the Rogers Commission members.
For whatever reason, alternative jargon such as "Root Causes Analysis" (plural) or "Proximate and Distal Causes Analysis" isn't as widely used.
by jasode