Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- I really like the Hugging Face guys, but...
> Modify one thing at a time
> Change only one variable per ablation while keeping everything else constant. If you change multiple things and performance improves, you won’t know what caused it. Test modifications individually, then combine successful ones and reassess.
This is an unintentional microcosm of what is flawed with the document.
- What's wrong with it? That's good advice in almost any optimization or troubleshooting context where variables may interact.by CamperBob2
- Where does "Smol" come from? It's supposed to mean "Small" right? If yes then what's the etymology and reason for popular usage?by forgingahead
- In the specific case of SmolLM, it originates from the meme in this dataset https://huggingface.co/datasets/bigcode/the-stack-smolby lewtun
- It's just internet speak from the days of tumbler. It usually has cutsie connotations.
Tumbler speak has a bunch of whacky things, notably "chimkin nuggers."
by potsandpans - Hugging Face is not just an AI information-sharing website; it’s also a great learning platform for all AI learners. This documentation is one of the most impressive hands-on resources I’ve ever read.by tsenturk
- What others would you recommend that are comparable in quality?by abossy
- Hi, Lewis here (one of the co-authors). Happy to answer any questions people have about the book :)by lewtun
- Finished. Great write up.
- Really impressive writeup. In your opinion, how long will this stay up to date? The field is constantly evolving, do you plan to keep updating this document?by matusp
- I'm a little ways through this and it's great so far, nice job.
One of the reasons people build one though is to learn. Most smart folks are quite aware that the reality of pre-training a real LLM is going to involve some head banging against the wall (ie, things don't go smoothly like "building an llm from scratch" book), and they want to go through the process.
- This was a good read. I was struck by the quantity of nuanced and applied knowhow it took to build SmolLM3. I am curious about the rough cost it took to engineer and train SmolLM3 - at ~400 GPUS for a least a month, and, based on the set of book co-authors, 12 engineers for at least three months. Is $3-5M a fair ballpark number? The complement is how much experience, on average, the team members had doing ML and LLM training at scale before SmolLM3. The book is "up" on recent research, so I am surmising a phd-centric team each with multiple systems built. This is not commodity skill. What the book suggests to me is that an LLM applications start up would best focus on understanding the scope and knowhow for starting from post-training.