Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- If self-reference and strange loops aren't intelligence, and emergence is merely a byproduct of universality, then what on earth is intelligence? What is this thing we question as intelligence, and what are the values that take shape from it?
Can we truly define intelligence as prediction, prediction as compression, and compression as the process of finding the upper bound of Kolmogorov complexity?
Can the statistical compression of data really explain everything? I don't know. What exactly is intelligence? Honestly, in everyday life, I rarely think about what intelligence actually is. I usually just focus on what the task at hand is and how to get it done, which makes this a fascinating question.
When you code with AI, you realize there is something fundamentally different from humans. The qualities that make a good senior programmer and the qualities that make a programmer good at orchestrating AI agents are similar, yet there is a subtly different feel to them. I might not be able to fully articulate it, but...
What exactly is the fundamental difference that creates this subtle distinction?
by jdw64 - Confusion around concepts such as intelligence, consciousness, self-awareness, sentience, sapience and aliveness is causing a huge difficulties in the debates around LLMs and AI. Hofstadter didn’t exactly say that self-reference and strange loops were intelligence, but something deeper.by gizajob
- there is deeper thought on this topic beyond douglas hofstadter (~20 years ago for strange loop and prior for GEB) and scott aaronson
any cognitive science textbook will bring some thoughts, or a cursory google scholar search
- I concurrently prompt a senior human programmers and a set of AI programmers and the difference is the following:
The human applies critical thinking, questions choices, suggests his own ideas and has a certain taste. I also can prompt him, leave for 2 weeks, come back and have a result. And I can trust that it works! Velocity is OK. I often wish he would be faster :-)
The AI needs baby sitting and steering. It codes like a champ but I cant trust it. So we have 2000+ unit tests to make sure stuff worlks. The AI happily goes down into any rabbit whole I send it, so I need to constantly steer it. It has no taste st all, essentially everything is „A great idea“. However, its velocity is awesome. We build tons of featurs in no time. but lets not talk about the code, ok? :-)
I dont think Ill fire the human nor the agents. Both bring a lot of value
by lmf4lol - I think before we get to self-referentiality, we have a bigger problem to solve, and that's one that I did not expect: namely, at which point do we stop saying that a machine pretends to be conscious and start saying that it is conscious?
We have, somewhat unexpectedly, built machines that are very, very good at pretending. Now, I'm not calling the current generation of LLMs we have "conscious," but I can't really define a marker or a boundary beyond which I would start calling them "conscious."
by jwr - Our definition of consciousness requires stakes.
We associate consciousness with precious life: Life that comes with rights, needs, wants.
Life has rights because it naturally demands to be continued. If it makes no difference whether it is continued, it needs no rights, and we do not consider it conscious.
Life has needs because it cannot survive without. If it has no needs, it has no death. No death, no life.
Life has wants because it is harder without. If it has no wants, it has no emotion: It cannot feel pain, otherwise it would want no pain. It cannot feel happiness, otherwise it would want more happiness.
We don't want to build consciousness! :) Deliberately designing pain and artificial needs is a cruel exercise. We can build that, but we see no need to.
The real question at the table is, are these minds? And can minds exist without consciousness? Unequivocally, yes.
- Consciousness requires qualia. My 0,02€.by euroderf
- It's not in big-money's interest to call them conscious because that would mean they get rights too.by amelius
- It “throws” an error when “it tries” “to talk” to the “server.”
We’ve been speaking as if the computers were conscious for a long time already.
by odyssey7 - I think this is exactly the same problem to decide whether any human is conscious, just because they say they are conscious. Humans can act as if they are conscious too, they can even lie to themselves, their whole life, and never discover the truth about their own mind
- Scott Aaronson mentions Penrose's "The Emperor's New Mind", but I feel "Shadows of the Mind" is putting forward a much clearer view of Penrose's thesis. At the current stage of my life I'm quite comfortably in Camp C. "Intelligence" and "consciousness" are not algorithmic.[1]
A lot of materialists are in Camp A. For some even today, LLM's are AGI. Unfortunately, the terminology is quite clearly not adequate. There's many a people that have very different ideas and feelings on what's what.
The question on whether you can have "intelligence" without "consciousness" or "self-reference" are just on top of those. How can we define if a machine is "conscious" if we can't agree on what consciousness is? Panpsychists solve that problem by going the other way and saying that everything is conscious.
Overall these are very interesting topics that we can all ponder. The realities at the current time are, however, that LLM's are indeed useful. And the harnessing tooling that is being employed those days solve a lot of actual usage problems. Whether they solve other fundamental issues remains to be seem IMO.
[1]. I realize a lot of people probably haven't read the book, but you can ask your garden variety LLM to give you a quick rundown on the four camps that Sir Roger is using :)
by _hao - This one thinks LLMs should be trained and RLHF'd into illeists, as an experiment.by greyface-
- > It claimed (incorrectly, in my view and in most experts’) that AI could never work because there was something about Gödel’s Theorem and self-reference that no computer program could ever capture
The fact he uses an appeal to “the experts” here for something that is fundamentally a philosophical or metaphysical claim shows he doesn’t actually understand the point.
If LLMs were equivalent to humans then we wouldn’t use them. They would be doing their own thing according to their own will.
I still need to tell the LLM what to do and to direct it according to my will in order to create something that is useful. And I say this as someone spending $600/month on codex and Claude max subscriptions. I’m managing these things all day
by monideas - What the author refers to is the ability to iterate and update the internal memory. A classic transformer based LLM can only produce the next token and never go back and update old tokens or delete them. The best thing it can do is produce thinking tokens to serialize the internal state of the final layer so that it can pass it back into the first layer.
Just like the transformer was an advancement over LSTMs by making it possible to have perfect recall (reading every input), the only way to improve over the transformer is to build a deep equilibrium version, where the DEQ transformer is capable of updating its own memory (writing every output).
Such a machine would be considered a linear bounded automaton (a turing machine without unlimited tape) and therefore even the human brain could not have an architectural edge over it in terms of intelligence. The human brain could only have an edge in terms of energy efficiency.
by imtringued - I've used concepts from GEB a lot in my thinking about how LLMs.
On self-referentiality and the MU Puzzle: https://matthodges.com/posts/2025-04-21-openai-o4-mini-high-...
On Gödelian limits of prompt-safe AI : https://matthodges.com/posts/2025-08-26-music-to-break-model...
On the blurry line between pattern matching and reasoning: https://matthodges.com/posts/2026-08-19-bongard-problems/
I'm glad that the author of this post ends with:
> What’s left? Consciousness
because when I've read (and re-read) GEB, I find the book to be much deeper than just a threshold for conversational intelligence.
by m-hodges - Perhaps "true" self-referentiality was not needed. But it seems "dynamic feedback" is essential, and it seems to be adjacent to self-referentiality.
If we look at optimization process, it first does a forward pass which produces the output. Then it looks into computations which happened during the forward pass (by that I mean backpropagation), and adjusts parameters in such a way that it might produce a better output.
Formulated this way, it sounds like self-referentiality (system looks into what it just did!), but, of course, implementation is quite simple: it just stores activations from the forward pass. And training process includes not just code which does the forward pass, but also a full description of that computation which allows it to do a backward pass. So it's a kind of an unrolled self-referentiality which is not difficult to implement.
Perhaps more efficient learning can be implemented if researchers figure out a trick to avoid two separate, distinct passes. Our brains don't do a global backprop and are more sample-efficient.
by killerstorm - Was Hofstadter ever arguing that intelligence requires self-referentiality?
I haven't read his stuff in a long time, but from what I recall he was saying more that something about consciousness and the sense of self is based on self-referentiality. Not that intelligence requires self-referentiality.
I also remember having the sense that Hofstadter didn't really understand the "hard problem of consciousness". His discussions seemed to somehow confuse the sense of self with having subjective experience. But like I said, it's been a while since I read it.
by hax0ron3 - Yes -- I can't remember the exact words, but there's a striking bit in GEB where he directly addresses the question of "will a machine ever create art?" His answer is yes, but only after it has really lived life, experienced heartbreak, and so on. The machine would have to have a self, which he argued arose from self-referentiality and "strange loops".
Oh, I guess there's a question of whether you can separate intelligence from consciousness. His writing was mostly done in the long period where things that are very easy for humans were still intractably difficult for machines -- identifying objects in pictures, following simple written instructions, etc. So I think it's clear that he saw "intelligence" as being able to solve those simple-yet-intractable common-sense problems; the kind of thing LLMs now excel at.
by iainmerrick - A few quotes towards the end of GEB:
> I see no barrier imposed by Gödel’s Theorem to the implementation on computers (or their successors) of types of symbol manipulation that achieve roughly the same results as brains do.
And a distinction between intelligence and consciousness:
> It is entirely another question to try and duplicate in a program some particular human’s mind—but to produce an intelligent program at all is a more limited goal.
And then:
> Gödel’s Theorem doesn’t ban our reproducing our own level of intelligence via programs
That is pretty hard to square with "GEB argued that AI couldn’t achieve intelligence without first mastering Gödelian self-reference."
A bit more introspective:
> I think that the process of coming to understand Gödel’s proof, with its construction involving arbitrary codes, complex isomorphisms, high and low levels of interpretation, and the capacity for self-mirroring, may inject some rich undercurrents and flavors into one’s set of images about symbols and symbol processing, which may deepen one’s intuition for the relationship between mental structures on different levels.
by m-hodges - I don't think LLMs are properly self-referential. They reference a frozen training reality, which is not itself, but the old description of itself and the old world. Being aware would probably include continuously updating yourself (learning) from experience, including experience of oneself.by kzrdude
- They don't only reference a frozen training reality, they also reference the live rollout. Otherwise they wouldn't work at all.
They do continuously update from experience, but that only lasts for as long as the current rollout lasts.
by jstanley - ‘But the idea that you’d need explicit self-referentiality before you could get convincing and world-changing conversational intelligence?’ — has anyone ever formulated this actual idea or anything logically equivalent? This looks like a mighty straw man.by macleginn
- Two comments on this, trying to take a "which hypothesis fits the evidence" approach.
First, an LLM describing its own experience is not actually proof that it has any experience to be aware of, any more than an LLM confidently asserting any other fact means that it knows that fact is true. LLMs will describe music or tastes, in spite of the fact that it's never actually heard or tasted anything, based only on what it's read about them. In the same way, "non-aware spicy autocomplete" would produce an LLM that spoke about its own experience, based only on the input it has of people speaking about their own experience.
That said / secondly, from the little I understand of LLM architecture, I believe there are a large number of self-referential mechanisms built in. For one, nearly all transformers have a "residual layer", with various neural networks essentially reading and modifying it. This effectively forms a loop. Additionally, the "thinking" mechanism allows it to read what it's written and generate more things, which is again a loop.
So, maybe people didn't think, "Hey, we should build some loops, maybe that will make it conscious". But if "strange loop" is what defines consciousness, there are lots of loops in there onto which such a strange loop could conceivably form.
by gwd - > First, an LLM describing its own experience is not actually proof that it has any experience to be aware of,
Likewise, humans describing their own experience is not actually proof that they have any experience to be aware of
- Their lack of self reference is a core problem that undergirds a lot of faults that do occur during inference, but their breadth + the agent harness successfully covers it well, so it requires a bit of poking to witness. The “hallucination” phenomenon is exactly this. They don’t know the scope of their own knowledge, and they just say stuff, so if you go out of band, it has a higher probability emitting claims that aren’t true. RAG (I don’t mean embedding indices, but any information ingest such as an agent harness executing a search) are somewhat effective in covering for it, enough to make them very useful! But when it does go wrong, it’s generally the same reasons. It has a certain nature and sometimes you run afoul of it.
But I suppose it doesn’t harm its reasoning!
- Not sure self-reference solves the metacognition thing; an ant can pass the mirror test but probably lacks metacognition.
Though I haven't read GEB so I'm not sure how the strange loop thing ties in with either of those.
by andai - They symptoms sound a lot like humans, so I don't see how it stems from their lack of self reference. Most people you need to keep them in areas they understand or they go to pieces. The lack of self reference just means every time the context clears they reset. They are systems in a permanent state of extreme amnesia.by roenxi
- From your comment, I noticed a sort of pattern that is often seen when discussing LLMs. That they are these amazing things that can run so fast they trip themselves in their attempts at achieving a task. So we resort to refining the models, creating guardrails, orchestrating harness, so as to alleviate the 'hallucination' problem.
In contrast to human intelligence, there is an underlying mechanism that propels intelligent behaviour. A person is no less intelligent just because they lose sight, sound or inner voice.
by fyredge - Any signal that allows the model to see what's wrong helps.
Checking code is (relatively) easy, you can use static type checks, linters, and execute it to see if it's correct.
Fact checking is harder. A RAG can only check what's in the database, so you have to know what to know beforehand.
by k__