Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- I suspect the author still misunderstands the SVP. If I were the SVP of engineering saying this to someone, especially someone who is in a non-core engineering role like product, I probably recognize that they have a tendency to verbally spew and I want them to focus on the risk mitigation and response plan. I’ve already talked to my engineering leader because they informed me immediately, and I’ve already read the incident details.
But also the post reads like it comes from an inexperienced/immature service org, so maybe they’re just sorting their operational response process out.
by rhyperior - There's still some problem with that approach. OK, I will tell what I'm going change to prevent recurrence of the issue. How does that makes sense to the audience? Unless they just want hear that "some" change will be there and don't care about how that change would make any sense.
It boils down to what exactly is the ownership or accountability of your SVP around the issue. Why are they even bothered to ensure that there will be some change? If they are responsible for ensuring that the issue doesn't happen again, then they do need to say "I want the details".
by zkmon - > "What happens when requirements change inside the launch window?" > "The alert fired but..."
In the running example I think it is clear that you could stop at any point and address the problem from that point rather than continuing. Questions such as "Why do we permit last minute changes at all?" are given no space.
In a complex system sometimes there is no root cause (cf. aviation crash investigations and the Swiss Cheese nature of risk). By allowing one person to shoulder the entire context of the incident and for that person be the one to suggest lasting corrective actions it leaves little need for the VP to even be involved. How would they know if the proposed corrections are worth the time and money to implement? I don't see any leadership in the example described in the article - I just see a knee-jerk reaction rather than any kind of proactive reflection.
> "I thought we fixed this last time!?"
> "This time was different in a way that our corrective actions didn't address."
by zenoprax - One thing that's often left out that tends to grind organizations to a halt is that continuous improvement processes tend to be additive.
You always add rules, and alerts, and so on. You need to revisit existing processes to and see what can be removed or replaced.
My other nit with these is a term that's abused a lot "Root Cause", most complex issues have many contributing factors, not a single root cause, and if you force the teams to find one, they will.
I dislike RCA (Root Cause Analysis) it puts you in the wrong mindset, CFA would be better (Contributing Factor Analysis).
by juancn - > To drive change in your organization, don't ask "why did this happen?".
> Instead, ask:
> What are we changing so that the same class of failure is less likely next time?
Why are these framed as being almost mutually exclusive? I don't get it. "How/why did this happen?" informs how you answer "What are we changing?" Even the following section, "Reasonable people", functions in this way with a "Why did this happen?" followed by "What are we changing?".
Just feels like this article has some internal conflict with itself in order to achieve a "don't do that, do this" type of style.
by scsh - I'm here for the sentiment expressed by the SVP in this article: "I already believe we reached this point through rational choices. Let's talk about how to change the system so it doesn't happen again."
I do think his choice of words was a bit suboptimal. But, it got the point across.
by cushychicken - I sympathise with this, and a lot of teams do operate that way, but I fundamentally disagree with it.
Part of the reason Amazon's CoE-driven engineer culture had such operation excellence is that the responsibility was driven the whole way up the management chain. If your manager didn't dig down to the root cause of a sev2, the director was damn well going to, and if the director didn't, Andy Jassy was going to call them on it.
That sounds like a lot of busy work, and a bunch of it undoubtedly is, but the flip side of it is that the actual root causes were addressed, and if the root cause needed serious cross-functional resources to address, the escalation would get you what you needed. Need approval to bounce 3 months of work off your roadmap to address? You have a VP on the line to make that call. Need a couple of other teams to fix their shit? Here's a principal engineer who outranks their directors to drive the work. And so on...
by swiftcoder - > Then I realised that "I don't want the details" wasn't being dismissive. The executive assumed that we were competent, and was saying "I already believe you. Now let's talk about what happens next".
Something about this feels wrong:
- If you trust them completely, you don't need to know what happens next either. Just trust them to do the right things, your leadership isn't required.
- If you don't trust them completely, how can you know if "what happens next" is appropriate without knowing the details?
Isn't this just reinforcing the idea that leadership doesn't need to have their feet on the ground?
The best leaders I've worked with were paying attention to things from the bottom-up as well as top-down.