Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- > Claude Telenovela
Nice, hadn't seen this one before.
by aesthesia - For someone like me who's never done any hands-on work in ML, the blog is really hard to understand. Whoosh, over my head.
But, I think the underlying problem is that we don't understand how this sh*t works. So, it's just an empirical, iterative mess.
Like physics in the the years shortly before relativity and quantum mechanics.
by physix - My read was: "The first 90% of the work takes the first 10% of the time, and the remaining 10% of the work takes the other 90% of the time." And that we are now squarely in the remaining 10% of the work.by kaycey2022
- - with all due respect, from a ux perspective, could you kindly add a page where i can see just the titles of all your blog posts
- https://ianbarber.blog/blogroll
- https://ianbarber.blog/archive
- https://ianbarber.blog/posts
- none of the above links work
- i really dont want to scroll 200 pages just to see what your blog articles are
by vivzkestrel - lol yeah I guess the best move right now is to fetch their /feed and iterate through <post>sby smasher164
- There is https://ianbarber.blog/feedby MarkSweep
- Why didn't this author compare Llama 3 with GLM 5.2 (released 1 week ago) which is a more standard attention based LLM? To compare 2 separate families of LLMs and then pointing out that they are different is not a surprising result and detracts from the point the author is trying to make.
https://sebastianraschka.com/llm-architecture-gallery/?compa...
If you look at it, the diagrams are very similar, but the main differences are that the feedforward is replaced with a MoE (router to multiple feedforwards) and the model has a different attention implementation.
by charcircuit - It’s written by AI.
- Yeah, not a great apples-to-apples comparison.
I think the point stands: MoE, a myriad of complex attention approaches, shared layers, you name it. And making it all work together well is a huge trial-and-error pain even for small models, never mind getting to efficient hardware utilization.
by alecco - > If you look at it, the diagrams are very similar,
The page links to the same site you do. No wonder it is similar -- the source is the same!
by lproven - > Why didn't this author compare Llama 3 with GLM 5.2 (released 1 week ago) which is a more standard attention based LLM? To compare 2 separate families of LLMs and then pointing out that they are different is not a surprising result and detracts from the point the author is trying to make.
The entire point of the comparison is that LLMs look vastly different today than before. Comparing more similar LLMs would detract from the point I thought the author was trying to make.
- The author is correct, the model architecture is now much more complicated. You can see this if you use llama.cpp and follow the project. The earlier models were always fully implemented. Yet with more contributors, as of today tons of latest models only have partial implementation. DeepSeekv3.2 isn't fully implemented, same with KimiK2.6, GLM5.2+, DeepSeekv4 has no implementation, MiniMaxM3 not supported yet, Hy3-preview no implementation. The latest models are just bare bones to run with lots of support missing for the advanced features.by segmondy
- It's the bitter-lesson to feature-engineering lifecycle.
When a technique or technology is new people are making massive gains by just applying it to some use case, or gathering more data for training, or giving it more resources.
As time goes on those "bitter lesson" gains start to hit the shallow part of the logistic curve and companies have to start investing more and more effort into engineering for each small, incremental gain.
by jordanb - Well put, thanks.by pezo1919
- I assume the choice of phrase "bitter lesson" is intentional irony (since the original concept is that you get better results by just scaling up and not trying to be clever with domain-specific knowledge)?by zahlman
- I got a very different message from this, actually much closer to the problem of incumbent advantage.
The known-good thing has been heavily optimized for performance, making it much harder for new technologies to prove that they are better. This is similar to the problem of gas vs electric engines - we had a century of optimization and ecosystem development around gas engines, which creates an uphill battle for electric motors even though they are (eventually) superior on every way /except/ having that massive ecosystem.
The problem isn't as bad here, because software is much more flexible than hardware, and scaling laws give a reasonable way to try things out at smaller scale before going whole hog.
by sdenton4