Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- I've been spending a lot of time iterating on feature implementation plans with Claude (at work) and GPT (for personal projects).
I then make sure they write a lot of tests and that if functionality is changed or added/removed then tests are updated alongside.
Generally I try to review changes myself and ask it to explain the changes.
It's not perfect but I feel like the planning step being iterative isn't all too different from my old research -> think about implementation -> implement loop.
At that point, reviewing any code changes goes hand in hand with how I coded before LLMs.
The major difference is I usually have it make any changes for things I don't like.
by spike021 - It's not too dissimilar to life as a Principal level engineer pre-AI. You technically oversee the work of many engineers on their own systems, but you couldn't possibly review all the code yourself. You appreciate that human code review reduces, but does not eliminate, the number and severity of bugs shipped to production. You don't have the low-level details in your head, just the high-level ones. You design how all these engineers interact with each other, without stepping on any toes.
Time will tell whether we were right to move from a system where we trusted humans to understand the low-level system, to one where we trust fancy calculators. But so far, it's working.
by solatic - This is how every product manager feels every day.
If you're an LLM prompter shipping products, you're outputs are more akin to a product managers, not an software engineer's.
by 4ndrewl - I’m slowly coming around to the idea that if it works, I’m good with that. I never used to personally check all my colleagues work, either, I just trusted that they’d done a good job.
That’s not to say I blindly trust an LLM - my project is an autovectorising compiler. I have it write a bazillion tests, and I have real, non-trivial programs that it compiled. I have it run a ‘gate’ before releases that takes literally hours on 5 different architectures. To say this code is tested is … masterful understatement.
I think what helps is a long planning session, going into details, working out corner cases, designing test-cases, you know work and effort. Once that’s done it’s easier to let it work through the night and do a major production feature.
I still review the code, but hand on heart I couldn’t say I’d reviewed everything, sometimes I miss something in a complicated check-in. I try to keep checkins small and contained, but sometimes that’s not easy to manage alongside the “don’t break stuff” rule as well.
So it comes down to: I’ve been working on this for 6 months or so, and so far this has worked out for me. You know, kind of like getting used to a colleague’s idiosyncrasies over a period of time.
I will say that in my experience the code quality is fairly high. Maybe that’s down to the effort put in ahead of time in the planning stages, I don’t know, but I don’t seem to suffer from the ‘slop’ that people complain of. YMMV.
by spacedcowboy - Technically it's just a role change. You're now the project manager or something, not the developer. It isn't a project manager's job to understand the code, only to ensure it does what it should and doesn't do what it shouldn't. And to ensure the developers, etc str unblocked and doing well. Embrace the new role.by skeledrew
- Testing.
Lots and lots of testing. I usually interactively test with the LLM. Now that I'm spending less time, coding, I get to spend more time, testing.
In my experience, I test more, with an LLM, than I did, with my own code. I don't find coding errors, but I do find implementation errors.
On another note: I ran an LLM on a bunch of really mature code that I wrote, and could swear was "perfect," and it found a number of issues.
Nobody's perfect. Trust; but verify.
- 1) Need to triple down on infrastructure (safeguards, tests, sandboxes, etc.).
2) Ask agents to do more than just code.
They need to fully demonstrate and prove that their PRs are rock solid. Ask for screenshots, videos, code reviews, detailed architecture diagrams, etc.
It would be an insane amount of work for humans, but AI can do it easily for each PR.
Merging any PR from an agent should be a no-brainer if there's real proof that the feature is built correctly and working.
If there's still doubt, then the agent needs to do a better job of proving that the feature is built correctly, doesn't introduce regressions, etc.
by d0m - I’m surprised I haven’t see anyone say this yet, but I don’t let the LLM design the code at all, I let it implement a design I have created.
By that I mean that, rather than let the LLM do the entire structure, I take the time to create the full hierarchy of abstractions and actors that I want my code to do. I decide, at the UML level, how abstractions will sit and interact — and then I make the LLM implement those gaps.
The key thing here is that, by doing this, I DO know how the code works. I may not know how each unit is implemented, but I know each unit’s responsibilities and how they interact with one another. I’ve found that LLMs are HORRIBLE at writing decent abstractions, regardless of how much prompting and skills I’ve given it. It will always get the job done, but it will often be unmaintainable and require bigger and heavier refactors as I introduce more and more requirement changes.
But by flexing my basic design skills, I don’t have those issues. Future refactoring fit into easy abstractions that I’ve designed. And by documenting it up front, I also have what becomes a living document on how this work will turn out.
It takes a few hours up front to plan ahead, but it makes the LLMs job easier and use less tokens, and it makes the code better long term. I don’t care if one unit is suboptimal, as long as the abstraction is right — because at worst I can change an implementation for a better one later.
by bitwizeshift