Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- With all the focus on coding and agentic use, I wonder if the rest of the world will notice or care? Most AI use is not for coding or harnesses contrary to what this website thinks. Maybe this is how American companies stay winning.
For example I prefer Kimi K2.6 1T parameter to Flash V4 0731 230B parameter, even if it is less intelligent.
by Gecko4072 - I agree that most AI use in terms of users may not be for coding or agentic use (everyday people are asking chatgpt for something or looking at google ai summary), but with respect to AI usage, I speculate that the vast majority of usage is coding and agentic because they're super token hungry.
In terms of the value proposition of AI replacing knowledge workers, all value is in coding agents (coding agents as general agents).
by ismael_rr - What is the rest of the world using LLMs for? Agentic stuff seems pretty universal. Obviously coding stuff is only interesting to people who want to code, but automating complex digital tasks seems useful in all kinds of contexts.by kennywinker
- You can look at the benchmark and the GPT-5 failures like answering "April 22, 2019" instead of the correct "Oct 23, 2018" for the question:
What day, month, and year was Carrie Underwood's album "Cry Pretty" certified Gold by the RIAA?
If your idea of the smartest person in the world is the guy who always wins tuesday night pub trivia, this blog post is for you. It also gets it's foundational factual claim wrong (as seen via epoch.ai). Very on brand.
https://epoch.ai/benchmarks/simple-qa-verified?view=graph&ta...
https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?lo...
by themgt - Not sure what you're trying to claim here tbh - Oct 23, 2018 is what the answer checker is looking for, it correctly judged "April 22, 2019." as incorrect: https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?lo... (row 10, I don't see a way to link directly) and the correct answer matches the Wikipedia article's claim: https://en.wikipedia.org/wiki/Cry_Pretty and the RIAA's site it uses as a citation: https://www.riaa.com/gold-platinum/?reload=1786913260239&tab...by Groxx
- It's a nice idea conceptually to decouple knowledge from reasoning but like everything in life I think it's something of a fantasy. When you ask a model to do something, its response is grounded in all the world knowledge it has from those facts. If I ask it to color a circle blue and then red, it knows what blue and red are, that they are colors, that they are different, and this is essential to complete the reasoning about that task. The idea of abliterating all the "knowledge" but not affecting the reasoning pretends there is a clean boundary between them that isn't there. Yes if you take it to the extreme - does it "know" Einstein's birthday - it is very stark. But I think the boundary to where general knowledge starts to interplay with broad reasoning is not nearly as far from those specific facts as people assume.by zmmmmm
- I think Opus 5 is the best example of this - "relentlessly proactive" agent, therefore reducing hallucinations, but at the expense of fast, factual answers and direct logical paths. Everything becomes a pamphlet compiled from first principles.by janpeuker
- > This mostly solves hallucination
Current AI is like the film company producing TV series or movies
Your question is like a story outline. You tell the film company that this is the movie you want. The AI film company then searches for existing similar stories. If similar stories do not exist or details are missing, screenwriters use imagination to fill in the gaps (remember hallucination? It's just a makeup.)
So you cannot solve hallucination of AI
by lilerjee - Indeed ! LLM are creative writers, not journalists. Relying on overfitting for factual accuracy in not tenable. I don't understand why grounded RAG with judges in not the norm.by wolfgangK
- This makes hallucination detection more important.
There's no reason that an LLM should have a vast number of obscure facts encoded. It can go out to a search engine for such facts. But the LLM has to be clear on what it doesn't know.
(Google's pricing for search from programs starts at $2.50 per 1,000 queries. If an LLM reaches out to Google, it has to pay.)
by Animats - The article really just glosses over this. Just because a model doesn’t kno a fact does not mean it will go and fetch it, the model just as well can invent the fact. For that to not happen the model needs to know that it’s missing the information and I don’t see the article making any explanation how this improves.
I wonder what’s tre latest in this field? Did we get a grip on this problem?
by DenisM - I think the idea is reasonable, however the SimpleQA Bench stopped measuring in Sept. 2025.
So newer data would be interesting.
(It seems a bit like an AI generated argument that uses old facts - something that happens to me quite often)
by kaufmann - There’s SimpleQA Verifiedby yismail
- Man, both that Blogpost and some comments here read like science-fiction.
Specifically, creative writing driven by nerds dreaming about a future, without proper grounding in reality, constraints and all that stuff.
Which is kinda ironic given the topic. And also important to do, because we should keep dreaming. We should just also be aware of when we are doing that and mark it as such.
by hypfer - But is Reasoning and Facts truly separate ?
To reason properly about the human condition (eg. World War) wouldn't you need to reason on some facts ? And then reason how some "facts" change the human behaviour ? How can you arrive via pure reasoning to predict how a collective of humans act ? We are not reasonable, humans are not logical deterministic machines confined to algebraic rules.
by pulkitsh1234 - You can also reason the same way on some complete nonsense. A lot of us do this all the time.by ruszki
- Great article, even if it will be interesting to see whether things continue to develop in such a direction or not.
> There's a version of this future where the model card stops listing a knowledge cutoff at all, because what's left in the weights goes stale on a scale of years instead of weeks.
Future?
Even just recently I’ve read of two approaches to this problem:
Cactus have come up with Needle [0][1], which is their tool-calling focused 14 MB model (still an LLM!) – no world knowledge engrained.
And instead of say, tool call structure, VibeThinker [2][3] focuses on reasoning over world knowledge.
Combine these two approaches with a reliable search tool/a safe way of accessing the internet for the model, and you’ve got a probably slightly slower model for factual questions, which on the upside however doesn’t hallucinate.
[0] https://cactuscompute.com/needle
[1] https://news.ycombinator.com/item?id=49246804
by msdz - That kind of setup is super dependent on a search engine, and search keeps getting worse.by kennywinker
- This AI generated post (100% on Pangram) is pretty out of date.
>On SimpleQA, a benchmark of factual recall with no tools allowed, the current leader is Gemini 2.5 Pro at 53%, so the best recall money can buy still misses half the questions.
SimpleQA hasn't been updated in a long time. Gemini 2.5 Pro is a sixteen-month-old model, not "the best recall money can buy".
>The part I find most promising is what this does to hallucination. When a fact lives in weights, a wrong fact is unfindable and unfixable.
This seems confused. LLM hallucinations don't come from the weights containing "wrong facts", they are artifacts that appear at runtime.
>When the fact lives outside the model, a wrong answer has an address. The model cites a document, so you can open the document. If the document is wrong, you edit the document
You can make any modern LLM explain its reasoning and find sources for its claims. None of this has anything to do with facts needing to exist in weights or in harnesses.
The internet is full of wrong information and I cannot magically edit it to make it all correct, so this doesn't help me.
>if a model is factually wrong a claim with a source is checkable and a claim from weights isn't.
Why? If a model's weights claim that Bart Simpson became President in 2020, why does this fact suddenly become uncheckable?
by COAGULOPATH - A true camper doesn't need to check Pangram, Jimbo. He goes by pure animal instinct!by andai
- >>You can make any modern LLM explain its reasoning and find sources for its claims.
>The internet is full of wrong information and I cannot magically edit it to make it all correct, so this doesn't help me.
My favorite RAG experience was asking Bart (or whatever they were calling Gemini back then) an answer to a question I knew.
It gave me the opposite of the truth (as was common with LLMs at the time).
But weirdly, it had cited sources for this "fact."
I checked the sources. Two of them, both AI SEO slop.
In this moment, andai was enlightened...
by andai - >>if a model is factually wrong a claim with a source is checkable and a claim from weights isn't.
>Why? If a model's weights claim that Bart Simpson became President in 2020, why does this fact suddenly become uncheckable?
Because in one case you have a source you can use to validate the fact, and in the other you don't. Though, as you explain earlier in your comment, the premise is misguided/hallucinated.
by Gander5739 - Yea the "When the fact lives outside the model, a wrong answer has an address" sentence seems aggressively AI written. Saw that and my senses went off.by claiir
- > This AI generated post (100% on Pangram) is pretty out of date.
Quite ironic given the topic. It seems that the author’s model indeed contained too much knowledge about old Gemini releases, and did not do enough tool calling.
by nojs - Seriously, anybody with a passing knowledge of LLMs knows thats not how they function. You can't encode logic in them because that's not how they work. It's a statistical model with useful emergent properties. It doesn't think, it doesn't reason, it isn't aware of facts or the rules of logic.by malfist
- I agree with everything you say except this:
> You can make any modern LLM explain its reasoning
You can make any modern LLM create a plausible, self-consistent explanation that looks like reasoning, but it's not "the reasoning it used to arrive at that answer".
- Ideally what I'd like to see is pluggable knowledge bases.
So if I'm e.g. coding a SwiftUI app for navigation, I'd take 9B of basic coding and reasoning, add 10B of swift/swiftUI, add 5B of GIS/geography knowledge and another 5B of frontend app design knowledge. My model doesn't need to know a single line of python.
Then when I want to research electronics components, I grab a 15B model of agentic research techniques, and add in 10B of electronics knowledge, etc.
I don't want general purpose models. They try to be everything to everyone. I want to click together a model that is laser-focused on what I am doing, and I want to run it locally
by kennywinker - This would defeat the AGI narrative/belief that so many building these models haveby edgyquant
- > I don't want general purpose models. They try to be everything to everyone.
I think the vast majority of people do want general purpose models. They want to be able to ask it any question, or ask it to perform any task, and for it to do a decent job at it.
I agree that it's really hard (maybe even impossible) to build something that's everything for everyone. But your average (or even above-average) LLM user doesn't want to choose from a catalog to stitch together a model that does just what they need.
I do think for certain domains this is useful and will make sense: the model backing a coding harness doesn't need to know about the politics of 400BCE Rome. But I'm skeptical that many software developers will want to do what you propose, picking knowledge bases that are tailored to their current task or project. And at any rate, for web-based chat interfaces, most users just want to type a query and get an answer.
by kelnos - If that is better (and possible) it will be baked in the tools. But is a model that doesn't know python better or worse at swift is what I wonder.by jvuygbbkuurx