Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- How much of this translates to Asahi?
Always been disappointed by the answer to that question in the past - fingers crossed this time
by oivaksef - Imagine 10 billion tokens per second even if we make no further improvement in frontier LLMs than what we have today that would be paradigm changing.by dyauspitr
- And the 5 years old AMD Ryzen 5 5600H is doing 7M?
Am I reading this right? Then I need to try this on Strix Halo
by throwa356262 - And it's only using AVX-2 and not AVX-512, AMX or ACE. Or built-in GPUs and NPUs (the M series doesn't emphasize matrix multiplication on the CPU side because it already has matrix multiplication units on the GPU, which is always attached).by rbanffy
- It could probably go quite a bit faster using Arm SME. The entire network could fit into the ZA register.by gok
- Model is 4K parameters - I don't know enough about that size of model to know if this impressive or not.by fwip
- Wait, does this fit in L1?by andai
- Small # of parameters means no memory bottleneck, which means blazing fast performance.by mcbuilder
- It's a trivial example. This won't be useful outside of a VERY specific domain without more parameters. Many people need to know about the bitter lesson.
https://en.wikipedia.org/wiki/Bitter_lesson
Over time, I'm sure we'll be able to filter information better and get parameter counts down, but I wouldn't count on that within the next 6 months.
by altcognito - Honestly not sure this is impressive. I ported microgpt to zig as a learning exercise, then moved scalar engines to NEON/metal just to see what happened. Besides metal being slower (I probably did something wrong, but it could be due to the fixed costs of memory transfer into the GPU not being worth it due to the small model).
Anyways, it was also stupid fast, particularly compared to the python version. But I was pretty sure that's irrelevant to real production architectures!
by pkilgore - Is your zig microgppt port available anywhere for perusal? I'm interested in both at the moment, have some zig workshops with a few local folks on the horizon and would love to use that opportunity to boot up the noggin on a few things.
>fast .. irrelevant ..
I ask myself this question all the time, but I'm not sure I'll like the answer: have we really attained peak performance in the engines, if we're "using python for everything" .. ?
by aa-jv - I often wonder if there aren't some low hanging fruit in the nature of the python ecosystem surrounding AI/ML these days. I'm sure there are a lot of eyeballs looking at the problem, but every time I fire up a chat and see the python loadout, it "feels" like there might be work yet still to be done to make things as fast as possible.
And then, while I'm waiting for a response on things, the mind wanders to thinking about what the rust camp are doing, with regards to AI/ML.
To my addled mind, it really seems like the first thing to be done when getting the models firing, is turn them on their own tools and optimize, optimize, optimize. I'd sure like to know what the frontier labs are doing to squeeze those ergs out of our substrates.
Does anyone have clues what the landscape looks like outside the python tooling, vis a vis higher-performance infrastructure? Being glib about it I admit, it just seems 'odd' that the C/C++/Rust camp are letting scripting languages drive the AI/ML ship.
by aa-jv - Mostly because many on that camp are averse to nice tooling, like REPLs, JITs, or incremental compilation.
They do exist, but again, not really loved enough.
https://clang.llvm.org/docs/ClangRepl.html
https://github.com/jupyter-xeus/xeus-cling
Additionally, while in the past there are been nice high level libraries for C++, which is where Java took its inspiration from in first place[0], current culture seems to hate having such frameworks around, and they are mostly gone nowadays.
[0] - It isn't writing Java in C++, rather how C++ inspired by Smalltalk used to look like, 1983 - 2000's, before Java got released in 1996.
by pjmlp - This is pretty cool. Implemented something similar myself (a really small language model with ~10M params) just to teach myself the ML behind the LLMs. Did not implement it in C obviously, just use PyTorch, but it's interesting to go through the c file and see how he has implemented stuff I took for granted in Python in C.
Anyway, I just tested this out myself on my AMD Ryzen 9 9800x3d. I got 7647173 tok/sec using karpathy's Shakespeare dataset https://raw.githubusercontent.com/karpathy/char-rnn/master/d.... Going to play around with it and see if I can get a CUDA kernal built to see what it could do on a 5090. Claude estimates with napkin math that we could get around 2B tok/s
by hasteg - This is not an LLM obviously , it's just for generating random names. But interesting to think of the possibilities of truly tiny language models if there were connected together.by ilaksh
- they'd just be too small to make sense of other models' outputsby vrighter
- Is token rate a function of parameter size?by dcow
- Isn't a MoE model basically a cascading tree of smaller models or some variation of that?by api
- Hasn't it been repeatedly shown that many small models perform worse than a large model of the same total parameters?by nomel
- LLMs are an imprecise, more of a marketing term, to define Transformer models based on the self-attention mechanism, trained with massives amounts of data.
And this implements a transformer. Actually it is a very cool didactic example.
by elzbardico - It's an slm (small language model) due to number of parameters and it uses the same architecture as an llm, but llms have billions of parametersby alightsoul
- > The most atomic way to train and inference a GPT in pure, dependency-free C.
What sense of the word "atomic" is meant here?
by Retr0id - No dependencies, self-containedby elromulous
- Check out this port of microgpt to C, posted 5 months ago. It got a 2500x speedup over the python version. https://github.com/moebiusV/cugptby MycroftJones
- Of course, an interpreted language implementation, versus a compiled language designed to port UNIX.
Even if counting that it is basically calling into C or C++ libraries, there is lots of boilerplate that the CPU has to consume, that just won't happen on the pure C implementation.
by pjmlp - In the README, the C version gets "452× faster than Python", the 2500x one is using OpenBLAS which uses lower level things than what's in C[1].
[1] see eg https://github.com/OpenMathLib/OpenBLAS/tree/develop/kernel/...
by fulafel