Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- A marvelous article that explains everything. It should be mandatory reading for everyone in the industry.by akiarie
- My observation is that right now AI coding agents lean towards re-implementing everything from the ground up. This of course will not scale. It will scale beyond what humans writing code would scale to, but we will ask the AI for more, so it will even things out.
1. We will develop new programming languages that are AI first, in that they are aimed at readability and comprehensibility by machines
2. We will adapt our working style (e.g. how we prompt and train the AIs) to factor in maintenance cost.
i.e. AI first implementations will go through the evolution we had with human based code development.
- by PaulHoule
- I am not really comforted by this argument against the singularity. It seems to rely on the fact that the evolving intelligence doesn't have access to the real world. Okay, but I wasn't really worried about the singularity inside a datacenter. I was worried about it once we wired up all the robots and sensors and power switches to AI controlled systems. We are still doing that, right?
- Great term/concept to have at disposal!
Btw, this is why I believe current models with their 128k - 1M context size limits end up creating decoherent systems. They don't have the capacity to see the big picture above some complexity level, and end up doing lot of local optimizations that decohere the system.
by tomaskafka - “Decoherence” denotes a process (losing coherence), not a state, but the article uses it like a state: “Well, for one thing, it creates decoherence.” The state would be “incoherence“ (the lack of coherence).
> the more decoherent a system is,
This should be either “incoherent” (lacking coherence) or “decohered” (having lost coherence).
That being said, yes, incoherence in a system is a form of technical debt, and use of AI for software development comes with a heightened risk of the system becoming incoherent.
by layer8 - This points back to a more general problem. I love to haul out the quote "All models are wrong, but some models are useful", and it's actually attributed to a statistician talking about statistical models. The quote was written well before AI, from a mathematician working in a time even before computer science dug into data science/eng.
As presented the Decoherence Trap is AI related, but as someone with a background in physical simulation and modeling, and applying math models & system controls algorithms, software in general has always had a connection-to-reality issue. It's always been able to go faster in building models faster than reality can be checked. People are always eager for the results and to take action on them, when the long run limiting factor is often, can I observe enough real data signal to make a good decision by any set of optimizing model rules. It's often harder and slower to collect real-world data, ideas, and samples than it is to rapidly process it. With AI, I think we can even more quickly build models (not in an llm sense, but in a operational sense) that give beautiful, precise results which may or may not be useful in a connected-to-reality way.
If the model operates on bad data, then the risk is always the temptation of having a precise-looking technical answer to apply. The practices need to be updated, but I think the mindset of systems engineering validation & verification, and software QA are going to be very needed going forward.
There is actually a related issue, in that I don't think we want an overly coherent of set of models running through our economy - it's a strong risk of central planning failings vs competitive/experimental markets, but that's a tangential discussion.
by digikata - Great article. I feel most people, even great software engineers, seem to lack the intuitive understanding that building software is essentially modeling a system, and that every model faces the same limit: the number of interactions between its part grows combinatorially as the system grows. If some of those parts or interactions produce no value (bloat), then complexity is added for no gain. This is true regardless of who created the model or wrote the code.
Instead they seem to think that AI can someone work around what is essentially laws of nature.
by rnd33