Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Been doing a lot of interviewing lately.
Despite not having written code for about 6 months due to using Claude etc, I've surprised myself with how much the "muscle memory" of leetcode etc has come back to me.
I say this b/c I think the veterans with years of coding, debugging etc experience can just pick back up the skills they need even if they're a bit rusty.
The junior folks on the other hand, I'm not so sure. Friends of mine that hire straight out of college are saying that a combination of going to college during COVID + LLMs + "cloud is all you need" has resulted in juniors who don't know how to solve problems nor architect basic software.
I guess we are all on some kind of chart where the slope of losing the experienced people who know what's going on and AI getting so good we don't need people are going to intersect at some point. And that intersection may be later than is good for society and engineering overall.
by alexpotato - This is the same problem as the "AI drives the car until it can't" situation. Feel free to doze off so long as you can wake up and instantly have world class racecar "save the situation" reflexes. Hope isn't a strategy but that's what all of this feels like.by msandford
- Anyone who's worked in tech in a large company will probably have experienced having an ops team who use RPA tools to do repetitive tasks that tech teams get the blame for when things break. AI will make this so much worse. Things will break, everyone will assume 'tech knows the system', but really it's a new process outside of the tech teams that someone vibe coded but got it wrong.
Audit trails, logs, and tight data governance where things can only be accessed with proper roles is the only possible solution.
If an RPA team ever gets direct access to a production database in your company, look for a new job.
by onion2k - > Robotic process automation is a type of business process automation that automates tasks within business and IT processes using scripts that mimic human interaction with application user interfaces.
For anyone else wondering what RPA means. Never heard that abbreviation before.
- This is why the paradigm for AI use should not be automation but rather the cyborg. Under automation, people are less active and engaged and become mere operators of automated processes. They become slaves of the machines. Under the cyborg model, they arrange the machines in a way to make people masters of a universe that includes the machines helping them be that.by ChiMan
- For more in this vein, look up "Automation should be like Iron Man, not like Ultron". Sad to see so many people let go of their agency.by touisteur
- If capability increase continues as it has, then an incident that cannot be resolved by AI will stump humans no matter the practice.
I like the plane example from the article,but I think in reality it will be like code. 1.5 years ago engineers would routinely say that they still write code by hand here or there to keep their skills sharp, and that's just not something you hear much if at all.
If an SRE is faced with a situation an AI can't solve, then said SRE will use the AI systems to triage further, point it to different places and so on.
This works for SREs with pre-AI experience and intuition, possibly less so with new recruits coming in post-AI. I don't know what the solution to this is, maybe practice drills is it, but I have a hunch the entire field will be subsumed, same as many other engineering fields.
There is only so much need for taste and judgement, before even that has been incorporated into the models.
by danielbln - Nah, LLM models are already the new compilers. A commodity only engineers know how to use (in the context of software engineering in production environments)by sdevonoes
- > If capability increase continues as it has, then an incident that cannot be resolved by AI will stump humans no matter the practice.
I disagree with this. Whatever the AI produces must be embodied in some kind of information system. The moment the output is on disk, it's fish in a barrel for any competent operator.
I've worked in environments that are beyond the pale with regard to complexity. It will take AI another 10 years to product something as complicated and coherent as a semiconductor manufacturing operating system, which is clearly feasible for humans to manage today.
by bob1029 - If you are in a situation where you dont know what happened when something goes wrong, the business incentives will not accept “its too complex” as an answer.
Firms aren’t just selling products, they are selling reliability and taking on liability.
by intended - Like the fact that software "engineering" is mostly nothing like real engineering (and it’s further regressing now due to LLM coding!), the general lack of drilling is again one of the things that make software-related stuff look really naive and amateurish from the perspective of those dealing with the real world. Imagine if the military, police, fire service, and so on did not drill and rehearse incident response?by Sharlin
- It’s tough. The models can at this point very quickly identify issues in a Kubernetes cluster, for example. This because these systems give you a TON of observability, and it can easily see all the different moving parts.
That doesn’t mean the proposed solution is always right, but it is absolutely landing on the root issue faster than most humans would be able to, even pre-AI. Just because it can remember and run through a bunch of commands more quickly that I can.
There are lots of incidents where the symptom doesn’t always clearly point to the issue, so having something that can fairly exhaustively check a lot of different things very quickly is pretty useful!
But I at least partly agree, I think the more complicated and obtuse a system is, the harder it is for AI to do this. But we’ve invested time in making systems much more straightforward to understand and operate from one place (Kubernetes in general), and AI thrives on that.
And yeah, it means your debugging skills wane a bit because, yeah, if the bot can diagnose the issue in 5 minutes, it’d be irresponsible to not use it.
And I’m not really happy about it, and personally I’ve always been able to figure out a tricky bug given enough time. I don’t want to loose that skill. But everyone is under a lot of time pressure these days.
by anon7000 - I see aviation sentiment raised from time to time. In aviation failure is catastrophic, and systems you operate do not change on the fly (pun intended).
You can probably drill SREs that way but you are only teaching them how to _react_ efficiently, not how to _fix_ unique unicorn root causes of these failures. Aviation analogy would be an airplane engineer that tries to drill himself for in-flight engine failures AND engine debugging / engine redesign both at the same time.
It never made sense in software engineering, and never will. Minute spent on drills is a minute better spent on reworking codebase to reduce changes of future incidents. This post is very SRE centric.
- Pilots aren't expected to be design engineers and SREs aren't expected to be software developers.
At least in my experience, operators and on-call engineers aren't necessarily responsible for remediation and validation.
Before you take me too literally, there is, of course, an overlap of skill and trade, but I believe the pilot analogy is more apt than you argue. I do agree that the difference in consequence and stakes is meaningfully important, however.
by thomascountz - In aviation all code and doc are available to inspectors when a catastrophic failure or near miss happens.
It's not the case for other industries, and in particular car industry is able to hide about everything software related from inspectors.
by guerby - I've seen a variation of this where random engineers are pulled into production incident calls and engineers are not expected to be familiar with the system.
They were asked to "just use AI" and understand the component, triage the issue, build a fix etc. The engineer was forced to choose between accepting a potentially mediocre fix AI has suggested or risk being coming across as an incompetent resource who doesn't know how to leverage AI.
You can guess what the engineer chose. The fix wasn't bad but it was suboptimal for some edge cases. We had to later revise it. Have enough of these situations, engineers will eventually definitely give up understanding the system in detail.
by devsda - Code too.
I work with programmers and it's not uncommon that they can remember with shocking detail about code they've written in the past.
Someone might mention an issue that has cropped up and they'll stare off into space for a few moments and actually remember where that issue stems from in the code, because they remember writing it like 8 months ago.
This skill will be lost when AI is generating all code, we'll be stuck in a perpetual loop of having AI keep track of the state of the code in order for AI to extend and maintain it.
by INTPenis - Will AI remember ? Or rather how will we make AI remember ?by iamgopal
- I mean I think this is just how AI already operates. I’ve seen multiple models go digging in the git blame or past PRs, and even unmerged PRs. When agents work on a project that has well-defined in-repo docs, agents normally update them without prompting. So I doubt this will be a big issue for AI, but agreed we’re losing a skillby anon7000
- The more code writes autonomously, the less intuition the human owners have about that code. Loss of intuition is a seed of technical debt that grows with time. Over a long enough horizon, it can make looking at your own codebase feel like the first day on the job (sometimes at a company you started).
Luckily, there are ways to mitigate this and essentially translate those human intuition of how the codebase “should” be into guardrails for the agents. But without that, your setting your sails in a stochastic sea where each wave looks nothing like the last.
by jtfrench - I've been thinking about this lately - is it like using 3rd party libs to achieve stuff faster? As much as I would lovr to hand craft the datetime logic in my app, I might as well use luxon and invest this time somewhere else. Only now with llms, you get virtually infinite 3rd party libs you can use, you create them on the fly. So if you have strong engineering values, I would say simply it boils down to "contracts over programs", you can still be in touch with the logic that glues it all together and treat some logic as a blackbox the same way we do with 3rd party libs?by yard2010
- I think this has as much to do with how hard software became to understand as with the new shortcut to refusing to understand it and the shortsightedness in willing to take it.
We lost a lot of traction in the name of ease of staffing and speed. Using LLMs to generate more code that is harder to understand it catalyzes it but the root of the problem, in my opinion, was letting go of great design and deep understanding for short term profit.
by huurtehoog - I find the use of AI like quicksand.
The more I use it, the more I have to rely on it to make changes/ fix things in the same system. In the end, I come out feeling empty; no intuitive knowledge of the system "I" built or fixed.
Code review is important but it does not replace the mental model I am able to build when I do all the steps of software development manually without AI.
by krtkush - >The more I use it, the more I have to rely on it to make changes/ fix things in the same system. In the end, I come out feeling empty; no intuitive knowledge of the system "I" built or fixed.
Most of us build a system to deliver a product to make money. We don't feel empty for not knowing how the system works, since we don't really care: it's just a means to an end.
by bsoqk - I find that it helps a little to maintain an iron grip on the design. Take the time for really detailed change planning. Be pedantic and push back against every little thing that seems wrong or even a little odd in any plan document the LLM spews out. Likewise, watch for the for the moment when the LLM eventually stumbles and starts to make excuses for deviations from the plan. Interrupt immediately and force it back onto the plan (or your vision), potentially with pointers to resolve immediate problems. Completely unattended agents just don't work for important code.by gmueckl
- I put on AGENTS.MD a prompt that says: teach me something small something about the changes you did if the changes were < 3 tokens if it was more than that teach me more. That's helped me somwhat. So now everytime it changes something it teaches me something about the codebaseby anothereng
- It's the uncanny valley of AI. It's still not quite good enough yet that you can trust it blindly on a big codebase, so you still have to read and understand everything - which is often harder than just writing it up yourself.by konschubert
- No worries, at some point your hit the wall with it and the reality will force you to look at the code. It won't be nice, but until that point delulu land is sustainable enough to fall forwardby Muromec
- Author has a good head on their shoulders, but few if any companies are going to spend time on incident simulations for their SREs.
Why not? Because even pre-AI, very few companies spend time practicing restoring their backups, or disaster recovery, or picking infrequently-used runbooks to practice, or seeing whether they can easily rotate secrets without downtime, or trying to redploy the system onto another vendor's cloud/platform, or, or, or... It is the least-sexy operations work that exists. No executive cares about this. Ops organizations push for flashy work, same as everybody else: new infrastructure for new projects, cool chatbots, new flashy dashboards, make charts go up and to the right, etc.
Airline pilots go through disaster simulation training because the government mandates that training. If it wasn't a condition of holding a pilot's license, no company would pay for it.
Want SREs to spend time training for disasters? Take a step back. Support professional licensure. Make it a condition of holding a license. You won't get industry-wide professional behavior until you professionalize the work. It won't happen without licensing because every corner cut that is not immediately visible to consumers translates to additional profit, and increasing competition eventually requires these corners to be cut in order to keep up with competition and stay in business. Forcing all players to submit to licensing requires all players to pay these costs and thus forbids them from cutting them to become more competitive.
by solatic - >Why not? Because even pre-AI, very few companies spend time practicing restoring their backups, or disaster recovery, or picking infrequently-used runbooks to practice, or seeing whether they can easily rotate secrets without downtime, or trying to redploy the system onto another vendor's cloud/platform, or, or, or...
Exact. Let insurance cover it, say sorry to your customers twice and shwoop never happened.
by sevenzero