Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- I've recently started experimenting with grounding LLM driven implementation/verification on RFC based normative specifications, to avoid having to manually steer the LLM during implementation and dealing with reviewing sloppy pull requests.
It works quite well, as it puts your entire focus on writing (hopefully) unambiguous specifications vs. having to discuss unwanted changes with an LLM during code-review. One flaw is that this only works great if you know exactly what you want, which is not always the case.
by mrbnprck - normative specifications can help, but the thesis here is that specs that define behavior of the product or even architecture are helpful but there's MORE that can be done and even though "program design" feels too in the weeds it's still essential if you care about maintainabilityby dhorthy
- I'm curious how that works - are you actually hand writing specs with RFC level detail yourself completely or is it more of an LLM assisted effort? It seems like writing a comprehensive spec manually would almost be as much (if not more) work than the implementation.by boredtofears
- But do you actually gain anything if you need to write detailed specifications? That seems just as time consuming as writing code, but less gratifying.
Code is just detailed specifications on how things should operate.
by Lindby - I find it amusing that people who are talking about Dark Software Factories, are talking about productivity in terms of number of pull requests or commits as a unit.
If we are going in the Dark Software Factory route, why aren't we calling the code units as bos (bunch of shit) yet.
by orsenthil - No one serious about the idea can afford either vanity metrics nor ignorance of the code. The bar is higher, not lower, to operate with this much automation in the water supply. It's mostly a lot more math and a lot more work.
It's an extreme form of any startup: you trade off capital for years of your life.
- Surely "pile of...?"by dboreham
- its the optimizing utilization instead of overall throughput all over again. eli goldratt talked about this in the 1970s[1]. we still haven't learnedby dhorthy
- My radical opinion is that LLMs are harmful for software development - they are the ultimate "goto" operator. All actual code should be written by a human developer.
Instead, use them in adversarial mode - run QA scenarios using LLM agent as a substitute for end user to do bug discovery.
by d_silin - why?by 0xblacklight
- > All actual code should be written by a human developer.
This seems arbitrary. Why don’t you say the same thing about machine code? Developers use tools so they can avoid writing machine code. What is causing you to draw a line in the sand about use of tools?
The obvious answer is it’s just a function of the time period you grew up in, and a lack of willingness to adapt to change.
by antonvs - I love how we need to preface such an opinion as being "radical" nowadays.by arm32
- I have mixed feelings about software factories!
On one hand, our core product is just simply not fit for them at its scale. We've tried but the project is large enough to require human input for every change. But we have AI automations for light code refactoring, writing tests, UI changes etc. and they work.
On the other hand, I started a number of small experiments to see how far software factories can be pushed and while the code produced so far is nothing spectacular I could easily imagine how this can be extended in the near future. Perhaps if you start from the ground up with the idea that the code will be written that way then you can come up with strategies and architectures that accommodate it. At least this is my thinking right now.
Anyway, it is all open source and documented here https://relentless.works/ I am not sure for long I will keep this running. I provide zero direction to where this is going. I have no idea what it will happen next. It is a fun experiment.
I have another such experiment with a trading agent. I thought it will loose all of the money in short time. For a while it was stuck with no open positions after it lost a bit. I decided not to intervene and just observe the behaviour. Recently it opened new positions which was an interesting development. It is still loosing money (~ -3%) but it has not lost all of them and given the current market circumstances I would say this ain't bad at all. It just shows that perhaps we might be a bit impatient when it comes to AI.
So I think it is probably possible to build software factories but we need new concepts and a bit of change of mindset.
I hope this helps.
by _pdp_ - I've been building and running my software factory for 8 months now. Granted, there's no automated pulling down tasks and pushing PR right now (soon!). But after specifying what I want, it mostly goes to shipping on its own. On occasion, it does raise issues that I have to make a decision on. After doing systems evals on review, I've stopped looking at code during review for 4 months now.
I do spend a lot of time up front specifying what I want. My prompts aren't one-liners, but rather an interview process where we work through all the open questions and ambiguity.
I haven't hit the wall that the OP talked about (when agents just can't seem to make the right changes, and it's impossible for me to go in and change things manually). I used a lot of guardrails such as plan reviews, browser-based QA, adversarial reviews, unit tests, linters, typecheckers, post-commit hooks, and formal method traces. I also specified engineering principles that steers the code base to minimize state and side-effects: functional core; imperative shell, make impossible states impossible, use pure functional style, etc.
There are times, when I can feel a part of the code base is messy without looking at it, because the agent will make recurring mistakes in the same part of the code base over time. What I found the agent was doing over time is that it's been layering state variables as requirements were discovered. So what helps is to ask it to refactor all these state variables into a single sum type. And if the state machine for it is complicated, I'll ask it to write a formal model of the state in Quint. Then I'll generate traces that get run as unit tests, and ask it to write the code against that.
So while the code base isn't exactly Brownfield, it's over a year old now. As for the code base, there's a backend and a frontend. I think it helps that I established a clear pattern I wanted. You code are like memes: agents will just copy patterns they see in the code base. When it does have to create a new part of the system, I found Sonnet-level models tend to draw system boundaries in all the wrong places. Opus is better. I don't yet know about Fable.
Happy to answer any questions about my workflow.
by iamwil - I'd be interested in seeing more of your setup, eg if you published a long blog post and/or repo.by chrisweekly
- I’m most curious about what you’ve built with the code factory. Is it just building itself or do you have any outputs that validate it for building something other than AI coding tools?by eloisius
- This has been my experience as well. While not a full "factory", I've got several mature vibed software projects.
More often than not you can "feel" when the quality has degraded. Many of these pieces leave you hanging on the assumption that once it's slopped up that's the game, it's over, you're cooked, straight to jail.
But I've yet to reach that position and be unable to pop the hood, identify the issue, and agentic-ly clean it up.
The post explicitly says it's not a skill issue, but I've yet to see any situation where an average engineer can't unslop their codebase.
- I think there is a fundamental issue here of what building software even means
If you think you can just assign Github tickets to AI agents and go drink daiquiris on the beach I think you'll find that you end up with more and more towers of abstraction and indirection. There are 'points of view' that emerge during coding I think. And at some point you as a human have to be like "wait... what if we use Redis here". "Wait.. the API is already returning the data we need". "Wait... let's not add customers to the report who have not been active in the past year". Stuff like that
by firasd - yeah I 100% agree - and I think the most popular coding agent workflows / skill kits are designed to pull those insights and intuition out of humans in a way that optimizes for the developer's experience building the plans or building the code, e.g.
- claude code plan mode - mattpocock/skills - obra/superpowers - research/plan/implement
etc etc
by dhorthy - If you're working with the 1% of human developers who think like that, sure.
My experience has been that current models are way beyond all but the most highly competent developers I've encountered in my several decades career.
by dboreham - The machine gives you what you ask for even when that thing doesn’t exist yet.
Rather than “lights off,” utilizing information theory, decision-making theory and creativity theory makes me better at asking for the right things.
Memory is not transcribed to weights like when humans sleep. Memory is notes handed to someone on groundhog’s day who can’t remember yesterday. We hope they believe us. Don’t be too surprised when a highly entropic system introduces entropy to a project over time.
by AmericanOP - > "Wait.. the API is already returning the data we need"
To me that's a huge one. I witnessed this firsthand more than once and the fact that you make that comment too just shows there's a problem: I've seen top frontier models go wild, adding shitload of code to spout something that, eventually (after burning shitload of tokens), worked.
While it was basically a one-line change because the data was already there on the client-side.
The thing thought it noticed a pattern and went ballistic, mimicking some existing client-server communication, to "solve" my problem.
The worst is: all vibe-coded projects are full of such nonsense and the "prompt engineers" (such engineers, much wow btw!) who wrote their magical incantations have zero clue about it.
I use these tools daily, I love the time they save me. But sheesh...
- > There are 'points of view' that emerge during coding
I know it’s a bit cliche at this point, but this harkens to “programming as theory building”[0] which I agree is easy to lose out on when embracing agentic coding today.
by claytonjy - Wait these arent “software factories” they are strung together ai rube goldburg machines
Its crazy to me people write these articles and create standards like this is some kind of engineering standard with years of research and experience
This is like calling these folks the experts on aviation: https://youtu.be/M9Yww9LG3gw?is=xgtA-xMpNy-09Asu
Its still so early in the game for de facto standards - engineering teams need to experiment and see what works for their own quality metrics not just parrot “standards and methodologies”
This is still the very early days of AI and AI engineering
by AIorNot - interesting - i'd say my main goal is to put the current "agentic software factory" hype in the historical context of "we've actually been rube-goldberging software deploys for a while now"by dhorthy
- It is comforting to find other people experiencing the exact same reality as me, since I see so much in this post that matches my own experience.
It reminds me of all the recent talk about "taste". Architecture "quality" may not be objective in a right/wrong sense, in the same way that fashion isn't right/wrong. It's like we are all going to have to relinquish reason/rationality to the machine and start to study up on aesthetics.
Even historically, my big struggles have usually been deciding between two nearly-equivalent options. I get this a lot now with LLMs because there is no break in-between these decisions that implementation used to force. I feel I'm constantly making "taste" calls between tradeoffs that have no clear objective criteria, and it is as exhausting as the code review this post (and my experience) suggests are still necessary, even with Fable/GPT-5.6 level models.
In many cases, I do what I've done with junior engineers whose code I reviewed pre-agent: make on-the-spot judgement calls. When I see a broken window, I call it out. But when I see minor issues, I sometimes just let it pass, note it in memory and tackle it wholesale once an accumulation of similar minor issues get to a certain size.
As a tangential aside, I consider two dev shops from pre-agent days. One decides to hire 7 extremely talented engineers and gets them to work closely together. The other decides to outsource to 100 decent engineers and tries to silo them into modules. I think we are facing a similar choice with agents. You can either work extremely closely with a handful of agents, collaborating on design, review, etc. Or you can spin up a fleet of sub-agents and YOLO, then try to separate the wheat from the chaff in some automated way. My taste is the former, small highly coordinated shop. But time will tell if I am right or wrong.
- yeah my best articulation of taste is something i got from Jake Nations[1] while he was still at netflix -
"you know a bad pattern when you see it because at some point you were up at 2am debugging it"
taste is the hard-earned intuition about every anti-pattern and landmine that has blown up in your face since you started doing software
by dhorthy - Either you need to understand how your codebase works or you don’t.
Claude can write the code for you but it can’t understand it for you. That part has to happen at human speeds.
There are cases where you don’t have to understand everything, but I think that’s a more nuanced question.
All of the above is true even if Claude writes perfect code.
by janalsncm - > Either you need to understand how your codebase works or you don’t.
It's an interesting point. We can also think about it perhaps as a non binary thing - you need X amount of understanding in a specific codebase to be effective. Even before LLMs in large codebases no one understood it all; but we at least mostly understood our own PRs and our own areas of expertise in the codebase.
by weatherlite - My experience conflicts with this assertion. I've used Claude to achieve an understanding of two large codebases (that I mostly wrote, and certainly came up with most of the concepts therein) to the point that it's far superior to my understanding. I now get it to explain things to me that I have long forgotten.by dboreham