Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • I do agree that Qwen 3.8 27B is excellent but slow and very token inefficient. My benchmark places it near opus 4.6 and codex 5.3 performance. 3.6 27B couldn't even complete the benchmark. Please see below for details:

    https://gist.github.com/nharziro/aed0c364ce2f295a493494c6f1b...

    by nharziro
  • Opus 4.6 performance with a local model that can be hosted on consumer hardware is an incredible result!!
    by matheusmoreira
  • According to the paper "Stealing reasoning traces from proprietary llms" [0] all frontier models overthink.

    Thinking is good.

    You just don't see it in proprietary harnesses because it's literally cryptographically hidden from you.

    [0] https://arxiv.org/pdf/2608.09867

    by johnnyApplePRNG
  • I run mine on an M5 Max with just 48GB of (V)RAM, and it fits nearly twice in Q4. Works perfectly. I'm kinda glad I didn't spend the extra $2400 to get 128. We don't really need more... and that's a good thing (tm). God knows I thought about it in store. But I thought... maybe this year will be the year of the local model? Maybe soon we won't need that much RAM? I was right.

    The fact that it runs at 15tk/s in power saving mode, and 30 in perf. mode blows my mind. I can run the model in the background, coding something for me in OpenCode, hosted in LMStudio, while doing something else. What a world we live in.

    Having something close to human intelligence (at least for reasoning and code), running on a laptop, is amazing.

    by dexterlagan
  • For day to day LLM experimentation (and even some business use cases), I'd say Apple Silicon would be first choice for me.
    by sgt
  • Have you looked at using oMLX?

    https://omlx.ai/

    by vorticalbox
  • The big problem with overthinking on a dense model is obviously the speed hit you take. Going from Qwen 35BA3B to 27B for me is about 7-8x slower (should be ~9x?). This makes me a lot less patient for useless thinking tokens.

    I’d want to compare this to the new Muse 30B model which is super terse and has a whole different way of thinking (no “Wait,”) and in my experiments was way more token efficient to the point that the absolute tok / s didn’t really matter.

    by andy99
  • Glimmer is stupider than 3.6 27B. You can't compare its speed to 3.8 and be done.
    by lostmsu
  • I love reading Glimmer's "thoughts". Why use many word when few do trick?
    by bogzz
  • Comparing with Muse Glimmer is a good idea. I ran the same exact HTML tool generating prompt against both Glimmer 30B and Qwen 3.8 27B. Results:

    Qwen: https://gist.github.com/simonw/121ad098860028b2fab603fa12da1... - 17,576 reasoning tokens, produced this HTML result: https://static.simonwillison.net/static/2026/qwen-over-think...

    Glimmer: https://gist.github.com/simonw/51e8ddb2ee597a5005fa63bd4927d... 1,021 reasoning tokens, this HTML: https://static.simonwillison.net/static/2026/glimmer-bbox.ht... - ugly but functional.

    In both cases paste in the URL https://static.simonwillison.net/static/2026/two-pelicans-on... to see them work.

    Both applications work correctly and fulfill the requirements. The Qwen one (which used the default xhigh reasoning setting) is massively over-engineered. The Glimmer one used whatever their default in LM Studio is and I would argue is a tiny bit under-engineered.

    Weirdly the Glimmer one doesn't work with images on other domains like https://static.inaturalist.org/photos/714731804/large.jpg - it fails with a CORS error, but you don't need CORS to load images and detect their width and height, and the Qwen one handles that URL just fine.

    That's because Glimmer added this unnecessary line:

      img.crossOrigin = 'anonymous';
    by simonw
  • This is true, but I think it understates the problem. I did a task I've done with a bunch of small models lately (https://github.com/swelljoe/flar/pull/17), and it did an excellent job, the best of any self-hostable model. But, it took eleven (11!) hours on my dual GPU setup. It really chewed on it, and spent a lot of time checking and re-checking. It is by far the slowest model I've used for the task. GPT 5.5 did a similar task in about 20 minutes. Most big models took about an hour or so, and most small models needed a couple of hours (but did a worse job).
    by SwellJoe
  • > But, it took eleven (11!) hours > ... > most small models needed a couple of hours (but did a worse job)

    I think a factor of 2 or 3 is a small price to pay for the best result.

    by scotty79
  • xhigh tells it to overthink and re check everything. Low tells it to only do the minimum thinking necessary. I would suggest to give qwen medium which doesn't inject any thinking directives into it and also to give as much context as you can, ideally around 500k tokens or even 1M if you can. Big complex tasks like these make the model hit the compaction trigger a lot and they end up re thinking the same thing several times in my experience.
    by fermuch
  • Was that with the default xhigh reasoning setting? I suggest trying again with reasoning set to low or turned off entirely.
    by simonw
  • It won't satisfy the people who just want to drop a model into their existing toolset and run, but I think there are a lot of ways to deal with this overthinking problem.

    For instance, it's a step backward, but I put {"reasoning_effort":"none"} and led it by the nose:

       User: We're going to make <silly demo>.  Please create a plan, but do not write code yet.

   Agent: <short and reasonable plan>

   User: Now please follow that plan and write the code.  No other chat.

   Agent: <reasonable code in reasonable time>
    Maybe this can be fixed with Jinja templates or something, or maybe it's a hack to your harness, but it shows you can get the model to reason reasonably.
    by xscott
  • I feel that local models are better for "processes" where you need a degree of predictability. Like summarising the daily weather for the family chat bot or analysing email inbox priority.

    SOTA cloud models are more for open-ended tasks where you need "creativity".

    by theshrike79
  • I'm doing much the same, avoid the long thinking loops and instead have more iterations on the plan with reviews from different angles.
    by regexorcist
  • To be fair a lot of models have quirks, I've never found a model swap that was transparent.
    by hedgehog
  • Yes, if you set reasoning to none you can force the granularity of the thinking.

    It will actually adhere to your request for e.g. 3 sentences max.

    Thinking mode will override any instructions in the prompt (at least for other models in my experience).

    Of course this will probably hurt performance, but works great for easy tasks that you know are trivial. Tons of pipeline, image recognition etc use cases where this works well.

    I'd be curious to see Qwen 3.8 27B low thinking benchmarks though.

    by adam_arthur
  • My impression is that when you allow the model to use internal thinking as opposed to asking it to output its thinking first, it's more likely to backtrack when I detects a flaw in its plan. Said otherwise: once producing user-facing output it seems to lock into an approach, for better or worse.
    by vanviegen
  • I feel like the current “reasoning” that LLMs are doing has got to be a dead end eventually. Every time I have to read another answer with “but wait” and “Actually,” as they “reason” their way to a (sometimes) better answer, I feel like there’s got to be a way to just shortcut to the actual correct answer instead of burning all these token going in circles mimicking actual thought
    by jedbrooke
  • Chain-of-thought output shouldn't be taken literally. The tokens are a substrate for computation, not necessarily evidence that the model is wasting time and electricity by gratuitously second-guessing itself over and over.

    You can see evidence of this phenomenon in models dating back to the OG Deepseek R1. It was common to see the model talk itself out of the correct solution in the <thinking> block, or fail to reach it at all, only to produce a correct answer in the response. And vice versa; it was also common to see it reason its way to the right answer and then fail to follow through in the response.

    by CamperBob2
  • I was pretty happy with Depseek Pro in Opencode util I discovered I can see the thinking trace by clicking on the "thinking..." communicate. All those seemingly unnecessary "but wait" messages are frustrating to read. But at least to some extent it's just model taking time thinking through the problem, and the trace produced doesn't have to be representative of what happens internally: https://arxiv.org/abs/2404.15758
    by neuroticnews25
  • So-called "caveman" thinking attempts to address this.

    The important part of "actually wait, I really need to XYZ" is just "XYZ".

    The model can attend to just "do XYZ" and produce almost the same vector modifications as full verbose "reasoning".

    by suprjami
  • I hated it at first too...

    Now though I'm considering all the hidden "thinking" in the models layers that happens for each token output. It is a wild amount of waste! We just can't see it.

    This kind of stupid excessive computation is fundamentally how these models are so good.

    One day hopefully not so soon someone smart or a foundation model will come up with a more efficient architecture. That's when things get really scary.

    by frabcus
  • It “thinks out loud” to populate its token space. Asking it to shortcut risks truncating that process.
    by russfink
  • I think you should avoid reading the thinking blocks unless you spot errors in the output.

    I am very confident the reason we get all these second guessing and "but wait" and "actually" is they train them on collapsed corrected sessions. i.e they take sessions that look like this:

      user: Do x.
  agent: the user wants me to do x. I think I need to do a and b first.
  agent: does a.
  agent: does b.
  user: No no no doing a was wrong you should do c before b.
  agent: undoes a. does c.
  agent: does x
    And they turn it to a session where the user correction shows up in the thinking. i.e

      user: do x.
  agent: the user wants me to do x. I think I need to do a and b first.
  agent: but wait maybe I should do c instead of a
  agent: does c
  agent: does b
  agent: does x
    by throwawayffffas
  • It needs to argue with itself to extract most of the knowledge embedded in the weights into the context. Asking it to synthesize ideas directly in a single go is simply unreasonable. And MoE models need to walk multiple experts to extract all the knowledge on top of that. So you need to give them the reasoning trace to first spill all the associations into.
    by mordae
  • One line of evolution seems to be toward some form of latent-space reasoning, as in [1]. Natural language seems like a relatively low-bandwidth channel for intermediate reasoning.

    [1] https://github.com/sapientinc/HRM-Text

    by nsingh2
  • I have this branch of llama.cpp that among other things (like patching the template to not break the kv cache, and saving conversations to disk so you can resume quickly days after) also accept the reasoning effort flag here https://github.com/alainnothere/llama.cpp/tree/disk-cache-ev...

    I did testing and the reasoning effort can be set per message, I was not aware of the option of none mentioned by @xscott, I tested but didn't see any change, I think there are just 3 values, xhigh, medium and low as per https://huggingface.co/Qwen/Qwen3.8-27B-FP8 , I did testing and the thing can do it's "I'll speak 10 million words to myself to ensure I'm not missing something" and then switch to a faster model, then switch... I did a test and the thing keep coherence and follow it's train of though-kens, you can see the result here... https://github.com/alainnothere/llama.cpp/blob/disk-cache-ev...

    by xlayn