Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

github.com904 pointsby gitpusher42340 comments

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Hacker News
Is there a pipeline or approach to do this to any model? I'm particularly interested in Qwen 3.6 27B as it's the best for its size at the moment.
by docheinestages
This approach will only work for MoE models. There is a Qwen 35b-a3b. You just need to do GPU stop after router and read the requested experts to ram. And it is possible to build similar engine for this model (or feel free to adopt my engine) Not sure about generic approach for now, but coding with ai agents is relatively cheap now, you can try it
by gitpusher42
Okay this tidbit is interesting to me "5–6 tok/s M2 -> 31–35 tok/s on an M5 Pro". So where will be in just another gen or two?
my impression right now is that M5 gen is on the cusp of practicality for local inference.
If techniques like OPs here, start to make the RAM situation more amenable, by the time we get to M6 or M7 (or AMD's equiv next gen APUs on TSMC N2 nodes), local AI could be ready to go much more mainstream.
by tpurves
The speed of this perfectly correlates with the memory bandwidth of an M2 vs M5 Pro.
by trollbridge
I have an M5 128GB. Being on the cusp of practical is a good description. It will run, but prefill and token gen are still slow relative to my consumer GPU box.
It also gets very hot. If you’ve never heard the fans on Apple Silicon really spin up, it could surprise you. Makes the full GPU setup feel quiet by comparison.
I think after the hardware market calms down the ticket is going to be a light laptop with a second dedicated inference server on the network.
by Aurornis
I think it's M5 MacBook Pro, not M5 Pro, as he mentioned.
by hacket04
My assumption is that the difference is 90% from more memory. I'm making several assumptions because nothing here looks groundbreaking so I don't care to dig deeper, but the model + KV cache definitely cannot fit in memory on the 8GB machine, but probably can on the 24GB machine—or can at least get close. Assuming that this benchmark makes use of that, skipping SSD streaming will speed things up massively (I would have guessed much higher than the reported 6x speedup).
by hatthew
Without fundamental model architecture improvements the practicality largely depends on how Apple increases memory bandwidth.
Memory bandwidths (* = rumored):
```
  M1:       68 GB/s
  M2:       100 GB/s
  M2 pro:   200 GB/s
  M2 max:   400 GB/s
  M2 ultra: 800 GB/s
  M5:       153 GB/s
  M5 pro:   307 GB/s
  M5 max:   460 GB/s
  M6:       200 GB/s*
  M7:       240 GB/s*
  
  Nvidia 4090 1008 GB/s
  Nvidia H100 3.35 TB/s
```
Basically what we're looking at by the M7 generation is a tier shift, where the base M7 can do what the M2 pro did, and every tier moves up accordingly, with the M7 ultra becoming competitive with nvidia dedicated consumer hardware.
by Joeri
Exciting! Maybe techniques like these can enable systems with 30-60GB memory and very fast SSDs of the future run very large models hopefully.
by owaislone
How large?
With 64 GB of unified memory, you should be able to run a DeepSeek V4 Flash quantisation at 7–10 t/s, for example with: https://github.com/antirez/ds4 or https://github.com/steadfastgaze/MoEspresso (my engine).
The routed experts needed for the next tokens that are not already in memory need to be read from the SSD, so the speed becomes SSD reading bound and the larger the memory, the faster the inference.
by hacklas
Yeah! Check the Colibri and Flash-MoE projects. They’re already doing that.
https://github.com/danveloper/flash-moe https://github.com/JustVugg/colibri
by gitpusher42
> It currently generates 5–6 tok/s on an 8 GB M2 MacBook Air and 31–35 tok/s on an M5 MacBook Pro.
Where does this big a performance spread come from? I wouldn't naïvely expect SSD performance difference to be that big, and I would expect SSD performance to dominate...
by addaon
The M5 MBP has 24GB of RAM, more context in RAM perhaps?
by afzalive
Not only is it older, so Pro:Pro it would have much slower SSD, but doesn't the Air also have slower SSD than the Pro in the same generation? And maybe narrower memory bandwidth?
by Schiendelman
I am relying quite heavily on system caching and pread. And yeah, M5 is a way faster and I can guess Mac can cache something, even if process stays under 2gb.
It was 83ms read per token for M2 and 12ms on M5 pro. Total is 163ms/tok vs 30ms/tok for M5. So yeah, there is a faster read and faster gpu processing
by gitpusher42
My suspicion is that this is simply due to the M5 having more memory, and the OS already having most of the file cached. The M2 has more memory pressure and would cache fewer of the SSD reads
If that's true, inference speed would be even lower if you have only 2GB total, including OS caches
by wongarsu
The M5 SSD's performance uplift was fairly substantial, even when compared to the prior generation.
> In the Blackmagic Disk Speed Test, the SSD in the M5 MacBook Pro achieved read speeds of up to 6,323 MB/s, compared to just 2,031 MB/s on the M4 MacBook Pro. It's not like the M4 is "slow" in a vacuum, but the M5 SSD is over three times faster, which is a great generation uplift.
https://www.tomshardware.com/laptops/macbooks/m5-macbook-pro...
by GeekyBear
12 tok/s and almost instant response on M1 Max Mac Studio (with faster SSD than laptops) are impressive – gives hope that large models may run locally from SSDs instead of memory.
by nvch
Thanks for sharing! SSD read speed is the biggest limiting factor here, unfortunately
by gitpusher42
There are a lot of SSD streaming engines these days. But few to actually try some hard features.
There is one that could really improve the speed. Given almost all major models come with MTP head for speculative decoding. The same MTP head could also be used to speculative prefetch the expert weight residing on the SSD. If the expert weight can be preloaded before the GPU actually need them, the speed penalty from VRAM cache miss will be quite reduced.
If the technology demonstrates successful token rate improvement. future models could also come with pretraining heads to preload expert weights, and even make the training be aware of it.
by ycui1986
Worth mentioning why this is harder than it looks.
There is a different set of experts at every layer, and each layer has a small router that decides which ones to use.
The router needs to look at the state produced by the experts below it.
Drafted tokens from the MTP head can be used to predict which experts the first layer will want, but not beyond that. To know what layer 10 experts needs, you have to run layers 1-9 which means loading their experts.
So, yes, instead of a next-token drafter like MTP, you'd want something trained to predict the expert activation across all layers at once.
by hacklas
> The same MTP head could also be used to speculative prefetch the expert weight residing on the SSD. If the expert weight can be preloaded before the GPU actually need them, the speed penalty from VRAM cache miss will be quite reduced.
When using SSD streaming, the GPU is practically always waiting for the SSD to fetch the right expert, rather than the other way around. There is basically zero slack on the SSD side, so I'm not sure how "prefetching" is supposed to help. It would mostly hurt by fetching the wrong predicted experts, which already makes conventional MTP practically unhelpful for typical (not widely batched) SSD streamed inference.
by zozbot234
> The measured result is a reference point, not a performance ceiling.
Claude was here.
by woadwarrior01
In a way, the AI-text-policing is a golden age. Allasudden, folks actually give a shit about style??? Before AI was there _ever_ comments on HN "bro, the semicolons ... I just can't"?
Everyone's on high alert. Maybe writing will get better!
by spopejoy
Okay, I’m the first to get annoyed at LLM-ese. But language requires the ability to say that something is A and not B. There are certain phrasings of that that are painfully Claude-esque, but ffs the one you quoted is the type of thing an actual human being is just as likely to say.
by apothegm
Let my karma burn for saying this: Maybe it is time to let this go man. These comments are really the new incarnation of "grammar policing". (1)
They don't add anything of value, did the author use an LLM to fix his prose but no useless slop was added in the process: who cares ? Is the article useless slop: fine, downvote it to oblivion.
(1) For those not old enough to remember that wonderful practice please use your nearest LLM to find out or, you know, visit a library and do your own research.
by andrehacker
I'm sure it was a chatgptism first, but I wouldn't accuse a western company of distillation.
In all fairness, maybe it's just that they let some post-2022 recipe blogs get into the training runs around ~4.6-4.8 time
by jddj
"I ran more than 100 experiments. Most didn’t work. A few got me here."
And here.
by UmYeahNo
I grind my teeth when I see it. It's so pervasive that I worry I'll pick up the same ticks by reading so much Claudeslop.
by sebmellen
I am not native, my English is far away from perfect. I am using LLMs for checking my texts or grammar. I always trying to edit it properly, but sometimes I missing parts like that because I don't really have this "language feeling" as natives. Apologies for this
by gitpusher42