Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- It is obvious that super intelligence comes from more working memory.
It is the scary thing actually. Cause once AI makes arguments that require a working memory of hundred items, then we as humans will have no way of understanding the arguments…
We can decompose and write things but only up to a point. when Ai can have a working memory that spans hundreds of books, we are necessarily going to have to trust the system.
by d--b - “is obvious” -- that’s what my Russian math professor said in college before skipping the rest of a proof.
But was it?
by tired-turtle - We offload working memory to paper if we want to understand something that does not fit into the regular meat bits.by jacquesm
- >Cause once AI makes arguments that require a working memory of hundred items, then we as humans will have no way of understanding the arguments…
That doesn't follow. We could still understand it just by studying it and committing it all to long-term memory, it just takes longer. And there's a hard cap on the working memory of LLMs, due to the quadratic scaling cost of the full attention layers that have proved unescapable for all SOTA LLMs.
by logicchains - This is why we have hierarchies of abstraction. Pretty much every field of mathematics relies on constructing notations, models, and other tools to simplify things in a way that is verifiable. LLMs rely on the same basic technique, they can just pull from a wide variety of these abstractions at once. So far we've been able to understand their proofs just fine. Computer-assisted proofs in the past that relied on brute-force is where we have run into trouble. We cannot reason about millions of possibilities at once, and we had to trust that the computer program that analyzed them was correct, which is a really hard problem and leaves humans fairly unsatisfied. I think we are actually progressing in terms of understandability in computerized proofs.by flatline
- There are plenty of high value endeavors where being a superhuman knowledge remixer is right on target. But even capturing all of the knowledge is proving elusive.
I use coding agents. I think they're pretty good overall. They save me a lot of tedious coding. For example I probably wouldn't spend the time to implement native splash screens for all the build targets of a Flutter app, but I'll have the coding agent do it.
Nevertheless, for all the time that we've had coding agents, it's still trivially easy to find the jagged edges of their training. For example, Gemini evidently doesn't know if the Xcode part of a Flutter tool chain is misconfigured. That's not exactly a Millennium Prize problem. But it is shaped wrong for a training set for a coding agent.
by Zigurd - A superhuman working memory is exactly how I've been describing the LLM advantage. Paired with the unreliability of its reasoning and judgment, it is what makes AI a supplement to human intelligence, not a replacement.
On the other side, LLMs make random mistakes and wrong choices and they have a bias toward writing more code instead of less. You can make up for this to some degree by running another LLM against their output, but with very diminishing returns. Even if they were perfect, there will be an ongoing cost to little or no human awareness and understanding of the codebase.
It may take some time for people to recognize the cost of AI code generation and their value for virtually everything else, but I believe we'll get there.
by doginasuit - I also find them bad at what I call "abstraction compression." They're really bad at noticing when a helper function is needed, when some structure they repeated five times slightly differently can become a struct, when a whole section of code can be encapsulated in a simpler design.
I'm lucky that for my side project (an interpreter) I've written all the code myself, so I've built up its design in my mind over the past year, and so as I mull over what I'm writing I start coming up with simpler designs.
Interestingly using Opus 5 (and LLMs in general) has made me worse at this, since I don't feel the pain of writing something over and over again. On the other hand, I don't really want to implement a whole stdlib, so I have it write more of the auxiliary code. The hardest thing is that I have to manually manage the context, which is painful when I personally remember every helper function and why. I have to remember to keep the list of helper functions updated, which is irritating.
by smj-edison - Sounds like another attempt to frame AI in a way that makes them feel better about themselves.
The simpler explanation is that a working memory is a requirement for intelligence, and a larger working memory will make you more intelligent. Hence the AI can in fact be more intelligent than the mathematician.
by amelius - Depends how you define intelligence. I struggle to consider LLMs actually intelligent, at least for any definition of intelligence that I would be content by.by slopinthebag
- While TFA itself makes sense I disagree with the title and the conclusion. I would not consider referencing working memory during thinking as “remembering” but as a part of thinking itself. Working memory is the RAM to the much larger but higher latency indexed database that is our long-term memory. As such I would say AI is out-thinking us, even if in a brute force sort of way.
I think where you could say it is out-remembering us is when it can contemplate the vast universe of patterns, gleaned from essentially all human disciplines, encoded in its weights, that may let it draw connections that a human could not, unless they just happen to be familiar with multiple disciplines.
Which is why I think the analogy with Von Neumann / Einstein is also a bit off. From TFA it seems Von Neumann was more akin to what AI does, than Einstein. I don’t get the impression that it was Einstein’s memory but his ability to look at things from a radically different perspective. So far I don’t know that we can categorically say that LLMs can or cannot do that.
by keeda - Yes. That's how LLMs do programming, mostly. It's also why LLMs don't need abstractions or parsimony as much as humans. They can work on something complicated without simplifying it first.
This has major implications that haven't been fully realized yet. On the math side, there are long machine generated proofs. On the code side, there are high volumes of code with similar code not being folded into functions.
by Animats - LLMs use abstractions a ton in code though: standard library functions, popular libraries, etc. They just dont always make their own abstractions. At least not particularly good ones. LLMs work really well when they have well abstracted pieces to put together.by xboxnolifes
- This is the exact opposite of what I’ve been dealing with for awhile. LLMs absolute cannot work on something without an understanding unless they can outsource the understanding to a verifier. If you’ve got an easy to check function to measure progress then “keep going” is all the prompt you need. But if you need it to figure out “I pushed the up button and it moved up and left” then it’ll find the same bug five ways without realizing it’s just one bug in the underlying math.by cgearhart
- I've been working on generating a large code base for the last couple of weeks. Finally got around to generating a sort of code-duplication report and have spent the last week just having it de-duplicating logic that had been strewn all over the place (eg 11 different functions all doing date math to add x days to a date). dozens of items that had each been similar functions duplicated numerous times. crazy. (opus-5-utracode)
- For greenfield projects LLMs don't need abstractions, but as the project gets more complex, the right abstractions save a pot on input tokens (less code to read) and reasoning tokens (less work to do to figure out the code), so they free the context window for higher purposes
Also I suspect that, apart from that, the results on smaller, cleaner codebases are better. LLMs degrade when following more than N instructions (where N depends on the model) even if the context window is not full yet; I suspect they also degrade when code has too many unnecessary concepts and details
- It was duuring my PhD I realised that thinking is essentially
1. remembering all the different information to remember all the tricks
2. trying all the different tricks in the problem
3. optimizing deciding which to try based on different information
4. trying random things to discover patterns (and hence new tricks)
5. explaining your tricks to others so that they can do the first 4 steps independently and come up with even better tricsk
6. refactoring tricks into common and special parts to create a well organized theory
7. documenting for future generation in a language they can understand
that's all thinking is.
by whateverboat - > That's all thinking is
That's more because a PhD (especially in this socioeconomic era where academia is also a microcosm of capitalism) does not automatically teach or focus on deep inquiry, rather narrow research programmes. Case in point, fallaciously reducing it all to a superficial theory of tricks means that by the same light, general relativity or any other major scientific result is merely just a trick. That framing ought to be absurd on the face of it but the fact is many PhD students do still graduate with very unexamined understanding of science, cognition, philosophy, etc. (Another way to see this is to note how such an argument is letting the notion of "trick"/"heuristic" do all the heavy lifting--something an advanced education should have nurtured the critical thinking skills to avoid that very metacognitive pitfall in the first place, which is ironic.) (Yet another way is to see how increasingly experts of one field or another insist that one must stay on their lane when speaking about a complex topic.) There are professors and thinkers who have written/spoken about this but they are a minority. Even Einstein himself when he complained that more and more scientists were trained/structured to miss the forest for the trees, and that was almost 100 years ago.
by calf - LLMs are enormously good 1. and 2. But humans are much better at the rest.
I don't want to read another LLM documentation ever again. The information to text ratio is way too small.
by vatsachak - I thought this was fairly obvious. I don't consider any of the AI models I've worked with particularly smart, but they've read orders of magnitude more than I ever could in my entire lifetime. They have far better book knowledge than I have, so that's how I use them. I use them for things that I suspect other people out there would know, but I don't. But when I work on something that I suspect is truly new, the models rarely understand what I'm doing, and I've got to do it myself. Although I still poll them for basic principles, best practices, and other advice.by mcv
- I can't help but think of Michael Nielsen's essay "Augmenting Long-Term Memory" [1].
> Many people's model of accomplished mathematicians is that they are astoundingly bright, with very high IQs, and the ability to deal with very complex ideas in their mind. A common perception is that their smartness gives them the ability to deal with very complex ideas. Basically, they have a higher horsepower engine.
> It's true that top mathematicians are usually very bright. But here's a different explanation of what's going on. It's that, per Simon, many top mathematicians have, through hard work, internalized many more complex mathematical chunks than ordinary humans. And what this means is that mathematical situations which seem very complex to the rest of us seem very simple to them. So it's not that they have a higher horsepower mind, in the sense of being able to deal with more complexity. Rather, their prior learning has given them better chunking abilities, and so situations most people would see as complex they see as simple, and they find it much easier to reason about.
I once tried out his Anki approach during a math lecture. Whenever I reiterated a card, say about some lemma, I noticed something interesting about it. This was delightful and many lemmas became much more streamlined over time. It's not a "solution" to mathematics, but I found it delightful while it lasted (before akrasia or lack of time kicked in and I stopped doing it).
by re-framer - Compression is intelligence, no?
- > many top mathematicians have, through hard work, internalized many more complex mathematical chunks than ordinary humans
Do you really think an average person can internalize complex math? Them compressing it effectively and then remembering it is a sign of (very) high intelligence.
- One thing about human mathematicians is that they only publish positive results. Professors etc might have file drawers full of "negative results", but the incentives and bandwidth of human mathematicians makes publishing these useful results impossible.
But AI agents have no such limitations and can publish and re-use negative traces easily. There have been some recent projects (https://www.theoremdb.org) aimed at exploiting this fact. https://news.ycombinator.com/item?id=49227505
In general though, LLMs do not have the same limitations and incentives as human mathematicians, and the next year's tsunami of change will make this abundantly. clear.
by philipfweiss - What is next years tsunami of change? Asics?by tomrod
- Yeah, LLMs are great at generating negative results for math-related prompts "We scanned values {a,b,c} from 0-100 and no results" Great..too bad journals will not publish this. But good job, I guess. A negative result is only truly useful if it can be bounded, requiring an actual proof.by paulpauper
- A great scientist once said, that the scientific method also includes stating, why what one "found out" could be wrong, and that one might need to retry experiments, whose results one relies upon. I think it was in "Cargo Cult Science" by Feynman.
In a way what they are doing, only publishing positive results is no longer good science.
by zelphirkalt - The bandwidth is absolutely there ("we tried this and it didn't work" is totally the stuff of conference discussions).
The incentives are not.
The incentives are skewed towards "a magician never reveals her secrets". The results are presented as if a rabbit got pulled out of a hat, with a maximum ta-da! effect, and little backstory of how the hell did we get there.
Don't get me wrong, these things are discussed, often over beers (you better drink it you want to make a career in the field).
But not published.
The younger mathematicians are trying to change that with the blogging culture. But the professional incentives aren't there. (In corp-speak: can't put blogging on perf). They burn out.
That's why math blogs usually come from either the top dogs in the field, like Terrence Tao, who don't need to care about perf, or people outside academia.
That's one thing that I hope the disruptive/destructive effects of LLMs will force mathematicians to face.
As one of my fellow mathematicians sarcastically wrote¹, we've reached a point where we should become a cult because we're acting like one anyway.
The other possibility is, of course, that the shake-up will take us precisely into that direction.
My point here is that the real problem here is not mathematical; it's a social one: incentives and politics, organizational structures, policies, allocation of jobs and funding.
All of this directly impacts how we do mathematics, who we do it with and teach it to, how we teach and communicate, and, of course, what math we even do and look at.
Given that, I'm neither too worried about humans vs. AI standoff, nor hyped about the Glorious New Future full of AI-assisted discoveries.
AI or not, the organizational issues in the field are still there, as are the incentive structures (including the infamous publish-or-perish).
We are doomed, yes, but by our own hands and committees. And it's up to us, not the AI, to get us out of there.
The little shove from the AI might be just the thing we need.
____
¹ https://www.mcsweeneys.net/articles/an-open-letter-to-the-ma...
by alterom - What is a "negative result" in maths? "I tried something and it didn't work" isn't really a result. It's experience maybe. But it's not like science where finding only evidence for the null hypothesis is considered boring. Even unsurprising results are published and even celebrated in mathematics, like Fermat's last theorem.
- It's also "out-brute forcing them." It just never gets tired. If a mathematician picks a research direction and spends a whole week on it and it doesn't pan out, they will likely be annoyed, need a break for a while, etc. This thing just does not ever get tired or discouraged or care; it's just onto the next thing until something ends up working.