Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Sounds like a game changer if I see that kind of speed up on my hardware. So far I've prefered Qwen 3.6 because of its better tool handling, even though Gemma 4 is faster, but I saw they've updated the model template and that's supposed to be better now. Looking forward to trying this with llama.cpp.by regexorcist
- gemma4 has a specific problem with toolcalls that affects most runtimes. fixes for ollama and vllm are being worked on right nowby ch_sm
- Why is a separate MTP model even necessary?
An LLM forward inference doesn't just predict token vectors for the new last token:
In diagrams the forward pass is typically depicted as taking input tokens vectors <t1, t2, t3, ... t98, t99, t100> (here native context being 100 for didactic purposes) and generating output token vectors <t2, t2, t4, ..., t99, t100, t101>.
As far as I understand that is didactically only semi correct, it correctly depicts the locations of tokens in the input and output string, but actually the token vector at the t2 output position is NOT identical to the t2 vector from the input, but a token vector which after softmax gives P(t2 | t1).
And output token position t5 actually corresponds to P(t5 | t1,t2,t3,t4). I.e. the forward inference is modelling the statistical conditional N-gram function from inputs to outputs, from the bigram conditional probability P(t2 | t1) all the way up to P(t101 | t1, t2, t3, ..., t98, t99, t100).
Suppose you want to take bigger steps, nothing prevents one from calculating the forward function by sliding a fixed (committed output string) to the left not 1 position but say 10 positions, and then using the last 10 predictions as the new output prediction. That doesn't need a new MTP model. Perhaps it would take some careful modification to ensure the same original output distributions as if the tokens were generated one at a time, but this hints at the possibility.
One could also slide to the left 5 positions twice, not committing to all 10 new tokens at once but only commiting to the 5 oldest values of the 10 new values, and using the noncommited 5 last values as input vectors for the next invocation, so the model can push the new 5 vectors towards its final commited output vector value in 2 steps for better convergence...
Is there any reason multitoken prediction doesn't work this way, or is there some aspect of the conditional N-gram interpretation of LLM models that I am miscomprehending?
by DoctorOetker - Really excited to try this once it is merged into llama.cpp.
Gemma 4 26B-A4B is much quicker on my setup vs Qwen3.6-35B-A3B (by about 3x), so the thought of a 1.5 speedup is tantalizing.
Have tried draft models to limited success (the smaller 3B draft model in addition to a dense 14B Ministral model introduced too much overhead already)
by julianlam - On vllm with a 5090 I get 120-180TPS with the awq 4 bit quant + MTP speculative decoding
For gemma4 26B, same quantization, I get >200TPS.
Also note that qwen is extremely inefficient in reasoning; the reasoning chains are ~3x longer than gemma on average
by VHRanger - Has anyone managed to get this to work in LM Studio? They've got a option in the UI, but it never seems to allow me to enable it.by these
- I've gotten it to work with other models. They've got to be perfectly aligned usually, in terms of provider, quantization etc. Might be a bit before you can get a matched set.by svachalek
- Normally when LM Studio doesn't like it it's because of the presence of mmproj files in the folder. Sometimes removing them helps it show up.
They're somehow connected to vision & block speculative decode...don't ask me how/why though
For gemma specifically had more luck with speculative using the llama-server route than lm studio
by Havoc - Yes. Make sure you’re not using the Gemma sparse models since they don’t have a small model to use. Also I removed all the image models from the workspace.by AlphaSite
- It's not implemented in mlx[1] yet (or llama.cpp[2]), so it may take a while.by dvt
- In my testing the Gemma 4 31b model had the biggest speed boost in Ollama w/ the MLX runner for coding tasks (at about 2x). Unfortunately you'll need a pretty beefy Mac to run it because quantization really hurts the acceptance rate. The three other smaller models didn't perform as well because the validation time of the draft model ate up most of the performance gains. I'm still trying to tune things to see if I can get better performance.
You can try it out with Ollama 0.23.1 by running `ollama run gemma4:31b-coding-mtp-bf16`.
- I recently set up the 26B A4B model up on vLLM on an RTX3090 (4-bit) after a hiatus from local models. Just completely blown away by the speed and quality you can get now for sub-$1k investment.
I tried first with Qwen but it was unstable and had ridiculously long thinning traces!
by christina97 - The 31B is surprisingly fast too, for a dense model. Runs tg at least twice as fast as it ought to on my machine when compared to other 30B, probably due to the hybrid attention I guess. Ingestion is somewhat slower though.by moffkalast
- The A4B model is blazing fast and the model is super good at general inquiries. Notably worse than Qwen 3.6 for coding tasks but that says more about the Qwen model.by jszymborski
- Some of the early quants for qwen3.6 were broken. It's still finicky but with a little hand holding it's crazy.
Local models are the future it's awesome
- It even fits on a 3060 with turboquant / Q4 at decent speed (40T/s) for ~200$ (:by aimxhaisse
- I’m starting to think that googles strategy is a bit different then the other frontier providers.
Focusing more on performance to compute efficiency over pure performance. And maybe that’s why Gemini is (seemingly) lagging behind?
Other providers hitting capacity and hitting the limits subsidising their inference.
Google strategy seems to be about scaling and distributing these models to their existing billions of users.
by aleksiy123 - > Google strategy seems to be about scaling and distributing these models to their existing billions of users.
Yeah, part of that is installing a model in chrome to millions of users without consent.
by chakintosh - Isn't that where everyone's strategy is shifting?
- I like Google’s business model more than the other frontier model providers: sustainable. One thing I don’t like with Gemini Ultra is no visibility into token use or what the cost would be. I have been planning on letting my Ultra subscription expire and go with OpenCode with a fast inference provider to get this visibility, but this discussion thread gave me the idea of also trying the paid APIs with AntiGravity instead of a subscription. When I sit down to do a specific task I want accurate token usage and $$ data as I work.
- I don't view Gemini as falling behind. I actually view it as a somewhat distinct type of intelligence compared to the latest iterations of GPT5 and Claude. The latter are, increasingly, very focused on productivity and automation of work tasks. They're optimized for long, agentic, self-correcting reasoning loops. Gemini is very different: it feels to me like a much smarter baseline model, with much deeper intuition (especially its Deep Think mode), but it's not nearly as good at long-range self-corrective agentic loops. For months now my workflow has been to use Gemini for creative leaps and insights, while preferring Codex or Claude or GPT5.5 Pro for routine or precision work.by nilkn
- Watching the computer write text sort of reminds me of using a modem to call a BBS in the old days. This seems like going from 300 baud to 1200 - a significant improvement, but still pretty slow, and someday we will wonder how we put up with it.by skybrian
- There was a startup posted here which built custom hardware that let the AI respond instantly. Thousands of tokens per second.
- Check chatjimmy.aiby jeffhuys
- You're right about it being reminiscent of the dial-up area, but I don't believe it's 300 to 1200; it's more like 4800:
Modem vs Claude according to Claude:
300 @ 2368 characters - 1m 19s
1200 @ 2368 characters - 19.7s
2400 @ 2368 characters - 9.9s
14.4K @ 2368 characters - 1.6s
33.6K @ 2368 characters - 705 ms
56K @ 2368 characters - 447 ms
Claude @ 2368 characters - 7.9s
by garciasn - This is something I've been thinking about for a while...the current state of things really does feel kind of like the dialup era, wondering what the "broadband" era could look like. Watching tokens stream in is reminiscent of watching a jpeg load a few rows of pixels at a time, and the various different loading and connecting animations that applications implemented before things got fast enough to make them less relevant.
Some of the work in that direction like Cerebras or Taalas have been doing is an interesting glimpse of what might be possible. In the meantime it's a fun thought experiment to wonder about what might be possible if even current state of the art models were available at like, a million tokens per second at a very low cost.
by macNchz - Google is singlehandedly carrying western open source models. Gemma 4 31B is fantastic.
However, it is a little painful to try to fit the best possible version into 24GB vram with vision + this drafter soon. My build doesn't support any more GPUs and I believe I would want another 4090 (overpriced) for best performance or otherwise just replace it altogether.
by msp26 - Qwen is still better that Gemma though. Also you can tune it more for different tasks, which means that you can prioritize thinking and accuracy versus inference speed.by ActorNightly
- You could keep multimodal projector (understanding of audio, images & PDFs) in system RAM with `--no-mmproj-offload` in llama.cpp. Of course, then it is not accelerated with GPU, but you save its VRAM.by srigi
- MTP support is being addedto llama.cpp, at least for the Qwen models ( https://github.com/ggml-org/llama.cpp/pull/20533) and I'd imagine Gemma 4 will come soon.
The performance uplift on local/self-hosted models in both quality and speed has been amazing in the last few months.
by zdw