Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- "Okay, everyone is overworked, and we need you to hire more people for this team. We have too much tech debt, and we can't predict which bit of it will explode next week. The change needs to start with the attitudes of our executive leadership."
Oops, now I'm fired.
by dualvariable - There's still some problem with that approach. OK, I will tell what I'm going change to prevent recurrence of the issue. How does that makes sense to the audience? Unless they just want hear that "some" change will be there and don't care about how that change would make any sense.
It boils down to what exactly is the ownership or accountability of your SVP around the issue. Why are they even bothered to ensure that there will be some change? If they are responsible for ensuring that the issue doesn't happen again, then they do need to say "I want the details".
by zkmon - One thing that's often left out that tends to grind organizations to a halt is that continuous improvement processes tend to be additive.
You always add rules, and alerts, and so on. You need to revisit existing processes to and see what can be removed or replaced.
My other nit with these is a term that's abused a lot "Root Cause", most complex issues have many contributing factors, not a single root cause, and if you force the teams to find one, they will.
I dislike RCA (Root Cause Analysis) it puts you in the wrong mindset, CFA would be better (Contributing Factor Analysis).
by juancn - > To drive change in your organization, don't ask "why did this happen?".
> Instead, ask:
> What are we changing so that the same class of failure is less likely next time?
Why are these framed as being almost mutually exclusive? I don't get it. "How/why did this happen?" informs how you answer "What are we changing?" Even the following section, "Reasonable people", functions in this way with a "Why did this happen?" followed by "What are we changing?".
Just feels like this article has some internal conflict with itself in order to achieve a "don't do that, do this" type of style.
by scsh - I'm here for the sentiment expressed by the SVP in this article: "I already believe we reached this point through rational choices. Let's talk about how to change the system so it doesn't happen again."
I do think his choice of words was a bit suboptimal. But, it got the point across.
by cushychicken - I sympathise with this, and a lot of teams do operate that way, but I fundamentally disagree with it.
Part of the reason Amazon's CoE-driven engineer culture had such operation excellence is that the responsibility was driven the whole way up the management chain. If your manager didn't dig down to the root cause of a sev2, the director was damn well going to, and if the director didn't, Andy Jassy was going to call them on it.
That sounds like a lot of busy work, and a bunch of it undoubtedly is, but the flip side of it is that the actual root causes were addressed, and if the root cause needed serious cross-functional resources to address, the escalation would get you what you needed. Need approval to bounce 3 months of work off your roadmap to address? You have a VP on the line to make that call. Need a couple of other teams to fix their shit? Here's a principal engineer who outranks their directors to drive the work. And so on...
by swiftcoder - > "What happens when requirements change inside the launch window?" > "The alert fired but..."
In the running example I think it is clear that you could stop at any point and address the problem from that point rather than continuing. Questions such as "Why do we permit last minute changes at all?" are given no space.
In a complex system sometimes there is no root cause (cf. aviation crash investigations and the Swiss Cheese nature of risk). By allowing one person to shoulder the entire context of the incident and for that person be the one to suggest lasting corrective actions it leaves little need for the VP to even be involved. How would they know if the proposed corrections are worth the time and money to implement? I don't see any leadership in the example described in the article - I just see a knee-jerk reaction rather than any kind of proactive reflection.
> "I thought we fixed this last time!?"
> "This time was different in a way that our corrective actions didn't address."
by zenoprax - > Then I realised that "I don't want the details" wasn't being dismissive. The executive assumed that we were competent, and was saying "I already believe you. Now let's talk about what happens next".
Something about this feels wrong:
- If you trust them completely, you don't need to know what happens next either. Just trust them to do the right things, your leadership isn't required.
- If you don't trust them completely, how can you know if "what happens next" is appropriate without knowing the details?
Isn't this just reinforcing the idea that leadership doesn't need to have their feet on the ground?
The best leaders I've worked with were paying attention to things from the bottom-up as well as top-down.