Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- In initial testing on Ryzen 395 / Strix Halo it's about 22 tokens/s generation and the output quality is impressive. Better and faster than 3.8 27B, and enough better to justify moving away from 3.6 35B even though 35B is still faster. The Unsloth weights don't come with vision support but you can add that back yourself. llama.cpp recipe in case anyone else wants to save some time on setup: https://pastebin.com/fcqsbDTvby hedgehog
- We should still expect significant performance improvements.
I believe Unsloth’s branch (and ggml’s) don’t support MTP or the ngram embedding yet.
by data-ottawa - > Better... than 3.8 27B
How can a "6b active per token" MoE be better than a just released 27b in the same family? Possible maybe, but quite interesting and requiring some explanation.
Edit: ok, on second thought, probably because the small "experts" are really rich in specialized training compared to the dense model. Still raising questions about the details, e.g. the reasoning abilities (or all meta-skills) of a "6b per token" network compared to the dense capabilities...
by mdp2021 - Adding to my homelab stack, hopefully it doesn't overthink like the little model. Actually, hoping it thinks a bit less. Wait actually I'm really praying it reasons a bit more directly. But wait, I'm really sure that it must be a bit better.by lnenad
- It will be interesting to see the token efficiency analysis. This is my first question now with Chinese models; I take raw benchmark performance for granted.by esafak
- Since you're running through the trouble of setting that up, if its 125B params, but only 6B is activated, does that mean you mainly need to allocate enough VRAM for that much of the model? Or do you still need enough VRAM for the whole thing (and buffer for context window)? Or maybe anyone can inform me, this is one area I'm uninformed in.
- You are absolutely right to push back on this. Let me think for a moment.by javier2
- My stack is basically deer-flow with Qwen3.5-122B-A10B; this hopefully will be a speed and intelligence improvement. Running deer-flow overnight on any research topic or verify clear scoped programming issue is really neat.
Also, heating my home during the winter is nice.
Oh, also, I use llamacpp with --reasoning-budget; very simple way to move on.
by cyanydeez - That's low reasoning for a model, but max for a HN comment.by grim_io
- Did you observe the model overthinking on practical tasks? While 3.8 does think a lot on xhigh I've found that it really depends on the task. On one-shot prompts that are usually the first to be posted during new releases it will tend to spend a lot more time thinking than doing. In other words the more open ended a problem space becomes, the more Qwen will tend to second-guess itself.
Conversely I've found that it can be as succinct as Muse Glimmer when it has a clear path forward. This can be either through well defined requirements or through unambiguous steps to take based on its own reasoning. While I do think it's fair to call out how much smaller model overthinks especially on one-shot prompts, in practice it hasn't led to an overall increase in time to task completion at least for what I've been using it for.
by xatnys - You might already know this, but a large part of test-time compute / 'overthinking' is just letting the model do more passes, and refine its activation residuals more.
For example, even if you make thinking tokens literally just '....' (absolutely meaningless; zero information), you still see significant performance improvements: https://arxiv.org/abs/2404.15758 and https://arxiv.org/abs/2607.22925 for some starters.
Treat thinking more like a "loading screen message" that's been RL'd to somewhat resemble its actual internal state; which happens in its activations, not tokens.
by dannyw - You’re absolutely right to be hopeful. Three honest possibilities, and I’ll be straight with you about each:
1. It overthinks — Just like the previous iteration. High confidence. 2. It doesn’t overthink — Improvement from the last model for your use case. Regression for others. 3. It sometimes overthinks — Best case all around. A feature, not an impairment.
One final thing worth mentioning: (I made myself irrationally angry writing this)
by redrix - I can't imagine the future any more. US companies playing it safe and control models releases. Chinese companies are just like open source everything.
It's like Chinese are incentivized to open source from day one (years ago). While most US companies are deciding in realtime.
It's crazy that we need both to survive and advance further in the future we have never imagined.
- It's in Unsloth Desktop already. Looks like it's 73GB, so 128GB Mac or Strix Halo etc will work. Exciting!by pram
- 73GB for the 1 bit model...by dist-epoch
- Download is available, but likely need to wait for an update, I get this which is understandable with the architectural change :
Original error: llama.cpp does not support this GGUF's model architecture ('qwen4exp')
Edit : Saw the pull request, should arrive soon enough https://github.com/ggml-org/llama.cpp/pull/27742
by cwizou - I only see a 1-bit quant posted on unsloth HF and it’s 72.5 GB. Is that what you mean? That’s much bigger than I expected. If you can’t run a 4 bit quant in on Strix Halo it becomes a lot less interesting. https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUFby andy99
- I'm really impressed. Gave QwenCloud $18, handed 3.8-flash a few big forks of a lot of code, it did some archeology and made a clean merge. Then it used the project's tools to bisect a regression and fix it.
Was not expecting it to just get that right without any fuss, and it barely used 10% of this weekly limit. Something like 90M cached in/400k out for $0.45 is wild
- Pelican: https://gist.github.com/SerJaimeLannister/8fdef9c00175da0ca6...
Aside from the pelican, I am sort of impressed by the fact that things are going the way in terms of really impressive small models.
Also I love how this uses N-gram embedding. I think that Longcat was the first one who used it (I submitted that submission on hackernews because I really just loved the idea of it that I understood), I am certainly more interested in local LLM models and its interesting how they are utilizing new architectures to do some really impressive optimizations!
(Do note that I created it using a free rate limited end-point that I found on the huggingface space section: https://victor-chat-with-qwen3-8-flash-next.hf.space)
- > I think that Longcat was the first one who used it
Wasn't it introduced by Gemma?
by stymaar - this is a new architecture (foreshadowing qwen 4)
> trained at just 1/9 the cost of Qwen3.7-Plus, while outperforming it across the board
by tosh - Can someone explain the intuition behind the en-gram idea? I know DeepSeek published a paper about it a few months ago and the Gemma models have a lightweight version of it; but it hasn’t clicked for me yetby schopra909
- Ngram is compressing several layers of multiplication to a lookup which negates the need to have the same model depth and reduces the model size that must be loaded.
- Quoting RGFusion from Reddit: LLMs run into an issue where the further you train a model, the more it overwrites facts with generalized concepts. You need the model to be able to do both. Intelligence arises from generalization, but without accurate information the model will hallucinate.
The engram table allows for a low-computational method of fact-recall. You can think of it like a better form of RAG, where the data doesn't take up any of your context window and it's injected deeper into the model's layers, freeing the lower layers to carry out abstraction. This results in better "focus" for the model, both in regards to its intelligence and context recall.
Basically, they've separated the specificity-critical portions of the models memory into a parameter space that doesn't need fast compute (you can run it on system RAM) and allows the model to be trained on higher volumes of data without ruining its knowledge-base.
https://www.reddit.com/r/LocalLLaMA/comments/1vy6smx/comment...
by a11r - Waiting for llama.cpp support to land, but this might be a big deal for Strix Halo users.
6B active params helps around the memory bandwidth constraints, but a 128GB box can probably run the Q3/Q4 quants fairly easily with a decent context size. This might actually be better for strix users than 27B, which was already very good.
by a_humean - In my early testing it's way better both quality and speed on Strix Halo (posted recipe in sibling comment).by hedgehog
- Using llama.cpp I one-shotted (2 hours) a reasonable asteroids clone on my strix halo/128 using the 1 bit quant, using my custom harness (which isn't anything exceptional).
It was tedious - a lot of second guessing itself, and quadruple checking things it fixed a couple of iterations back - but it got there and the result is a playable game.
Speed starts out strong, but definitely drops off as context grows. At the end (I think context about 70k) it was down to 12 output tps.
Mind a bit blown.
by GrinningFool - Didn't expect it to beat 3.8 27B so cleanly.
Opus 4.6 Max self-hosted at 30 tok/s on a 5k Macbook in Aug 2026. The LLM timelines are crazy.
by rohansood15 - >Opus 4.6 Max self-hosted at 30 tok/s on a 5k Macbook in Aug 2026. The LLM timelines are crazy.
How much memory does this translate to and what quantization (if any) were applied?
by gruez - My AMD strix halo box (haven’t benchmarked yet) should also run it reasonably well. It was $1400 at launch, and is $4K now.
Your mac is < $2K in Biden-era dollars. Presumably the economy will eventually recover; maybe in one Moore’s law doubling if the midterms go outrageously well. That’ll be two doublings since the halo launched. I’d expect this model to run on a sub $1K box by then. $2K ought to get you a 512b parameter model at that point. If we have to wait out the rest of the term, the cost cliff will be even more pronounced when it hits.
by hedora