Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- These models do pretty well in benchmarks and real world so I'm highly suspicious of this article. Further more, in the original report, the examples of bad answers are from Haiku - at least 7 out of 10. Anyone who knows anything about LLMs know that haiku shouldn't be used for anything pretty much.
There's no reproducible set either. I'm not gonna trust this report.
by simianwords - Single shot or with reasoning enabled? My experience is that reasoning dramatically reduces hallucinations and improves output quality. I don't trust models without it.by 01100011
- Each year I have Claude do my taxes (which are complex) and compare them to those of our tax consultant.
Each year it's exactly the same.
I guess I'm getting really lucky?
by qarl - Given most financial advisors tend to vend out suboptimal advice and steer customers in favour of products they receive a kickback for, I'm happy to be accepting of an unbiased LLM that's trained on bogleheads.org.by lukeify
- This will be the same story for every industry again and again. AI is not good in x=Finance because models were not RL trained heavily on x=Finance capabilities yet. This is only because Big Labs have finance benchmarks lower in their priority list. Their first priority was solving programming because that gives the best leverage at this stage. As a side-effect they were able to solve a Millennium Prize problem, since theorem proving was also code.
So finance advisors in the comments section of FT are falling for the classical pitfall. They assume there is something fundamentally wrong with “AI chatbots” that they can’t do finance ever. They mistake the current products in the market for the technology itself. In near future someone will release “Claude x=Finance” and their world will shatter.
by ozgung - So I downloaded that report which of course doesn't contain the most relevant information (the questions) but it contains some examples of wrong answers.
I fed the first question to Grok (which they claimed they tested as well) and it answered it correctly in detail.
I repeated it with another one - again correct answer. I then selected the question they said Grok specifically answered incorrectly and it again answered it correctly.
I am sticking with my first intuition: people are terrible at testing tools and probably wanted them to answer incorrectly/not fully (the questions are constructed in a way to make it difficult as well). They also have vested interest in the conclusion (they are financial advisory firm) so there is that to consider.
People reading ft will now think chat boxes are bad at answering financial questions while they are pretty good at it. Zero consequences for spreading fake news for Financial Times there but good for financial advisors I guess.
by bluecalm - Article is just a vague summary of https://www.saturnos.com/report/artificial-authority
Anecdotally, current models seem to be decent at general personal finance principles - certainly better than the majority of personal finance education that people get exposed to unless they seek it out and read a variety of books and sources. But I wouldn't trust them with direct decision making with actual money due to the training lag time on current tax policy, etc.
by ehe78qhe - by 2dvisio