Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- I recently found Gemma 4 e4b surprisingly effective for small "classification" style tasks for something I'm doing at work.
In this case, picking out "semantic" css classes on single dom nodes.
Was able to run it on my 4(?) year old M2 mbp with 16GB of ram and it runs in only 100ms or so per query. Probably it can run much faster, but haven't experimented with batching etc
With tight and targeted context control, you can use extremely small models for useful things. Ideally with problems where the harness can be mostly deterministic and you have known bounds on what you're trying to do
by adam_arthur - Beyond the models getting better; there are still huge gains available in the inference engine side with new tricks like Dflash, MRT, turboquant - for some usecases these can multiply the speeds. There are even some model specific optimized kernels like for DeepSeek 4 flash that seem wild.
Makes me feel we are nowhere near the optimum yet.
Examples: https://dasroot.net/posts/2026/05/gemma-4-speed-hacks-mtp-df...
by ThomasBb - what's MRT?by brrrrrm
- Recent models (Qwen 3.6 and Gemma) can really do coding locally. Feels like SOTA from maybe a year ago? But you would want about 32-40GB total memory. 24GB is just a bit short of that. A gaming PC with 16GB graphics card and 32GB RAM brings you very close to a usable coding system.by canpan
- How are you using that RAM with the GPU?by DrBenCarson
- That’s the exact ram/vram combo of my desktop - what model would you suggest for that gaming pc setup?by wktmeow
- > Feels like SOTA from maybe a year ago?
Agree but only for small projects. SOTA from a year ago still wins on larger projects
by solenoid0937 - I'd rather spend thousands dollars on a Mac than subscribing API. The local model allows me to do my work any time and anywhere, without worrying about privacy leak.by isaisabella
- me too. plus, I don't like the idea of needing massive datacenters, it's not good for anybodyby claysmithr
- I got qwen3.6:27B running on my 4090 (24GB) with ~128K context leveraging some of the recent turboquant/rotorquant memory optimizations for activations. Highly suggest going up to that. the q4_xl+rotorquant combo is pretty good.
Some reference code if you want to throw your agent at it. https://github.com/rapatel0/rq-models
by rapatel0 - Forgive my ignorance but aren't they already on huggingface?
I assumed turboquant optimizations are already everywhere - in llama-cpp, or the quantization machinery of unsloth and the likes.
by dmichulke - What is your exp on performance +40k tokens? I've not gone past that as I've heard reports that were problems start to arise. I'd be happy to know your experience in that regard.by altruios
- I have an M4 Macbook Air with 32Gb.
These are my current results for my models:
I seem to be doing ok with the Gemma model for file parsing / handling.┌──────────────────────┬───────────┬─────────────┐ │ Model │ Size │ Tokens/sec │ ├──────────────────────┼───────────┼─────────────┤ │ gemma-4-e4b-it-mlx │ ~4B (MLX) │ ~10.5 tok/s │ ├──────────────────────┼───────────┼─────────────┤ │ qwen3-8b-uncensor-v2 │ 8B │ ~6.3 tok/s │ ├──────────────────────┼───────────┼─────────────┤ │ qwen3-14b-uncensored │ 14B │ ~3.5 tok/s │ └──────────────────────┴───────────┴─────────────┘by ionwake - <=10 tok/sec is unusable. You are faster writing the code yourself.by ActorNightly
- I am running qwen 3.6 9b quantized model on my m4 pro 48gb and it is barely useful to do some basic pi.dev/cc driven development. I think 128gb desktops are the sweet setup to actually get meaningful work done. However, getting your hands on one of these machines is difficult at the moment.
As much fun as it is to run these things locally don’t forget that your time is not free. I am slowly migrating my use cases to openrouter and run the largest qwen model for < $2-3/day with serious use for personal projects.
by sourc3 - Thanks for saying this. There's so much nonsense out there online about local models being better than Opus 4.7 and the like. It's just not true for regular users.
I have a brand new M5 MacBook Pro - top end with all the specs and I've tried local models and they're barely functional.
by sjones671 - How does it (the openrouter version) compare to ChatGPT 5.5 or Claude Opus 4.6?by hparadiz
- I'm using the 30b MOE model on same spec with 65k tokens as a sub agent with tooling and it absolutely writes decent code. The dense 9b I agree wasn't great.by elij
- Was the choice of such a small model driven by a desire for high tok/sec? I ask because an m4 pro 48gb machine can run larger models (if model intelligence is the thing that would make it more useful).by carbocation
- Why not 35b-a3b? ...or gemma4:26b-a4b? Both will be more capable than 9b and run at roughly similar (perhaps faster) speedsby Casteil
- Critics are (rightly) pointing to the fact that these models are not on par with SOTA for complex coding tasks. But many seems to forget that a large part of white collar office work is Excel crushing, file moving, translating dry legal documents, e-mail drafting, PPT drudgery, etc. These are absolutely doable with 30-35b+ models with the added benefit of keeping company data private.by PAndreew
- Arguably excel and legal are much worse than code because catching the mistakes can be much harder.
Case in point, JPMorgan London Whale incident, $6 billion loss caused by an excel error...
by tjoff - I think the conclusion is flawed here? Sure qwen3.5 9b is nowhere near the sota models. It's 9b and was made a year ago? Everyone taking about local models is pumped about the models released in April this year. Qwen 3.6 27b and qwen 35b a3b if you have a sad GPU. Those are comparable to sota models, seriously.
- I think it's useful to be realistic about what you can do with a local model, especially something as small as the 9B the author is using. A 9B model is around the level of Sonnet 3.6 - it can do autocomplete and small functions but it loses track trying to understand large problems.
But the are interesting and fun to play with! I do a LOT of work on local agent harnesses etc, mostly for fun.
My current project is a zero install agent: https://gemma-agent-explainer.nicklothian.com/ - Python, SQL and React all run completely in browser. Gemma E4B is recommended for the best experience!
This is under heavy development, needs Chrome for both HTML5 Filesystem API support and LiteRT (although most Chromium based browsers can be made to work with it)
It's different to most agents because it is zero install: the model runs in the browser using LiteRT/LiteLLM (which gives better performance than Transformers.js), and Filesystem API gives it optional sandbox access to a directory to read from.
It is self documenting - you can ask questions like "How is the system prompt used" in the live help pane and it has access to its own source code.
There's quite a lot there: press "Tour" to see it all.
Will be open source next week.
by nl - Not to be nitpicky, but many of the 4-12b models are somewhere between GPT-3.5 and GPT-4o-mini. It's hard to find a good comparison though, because the benchmarks people score models against change so often. For reference, Sonnet 3.6 came out about a year after GPT 3.5by potatoman22
- But I was doing a lot more than autocomplete and small functions with Sonnet 3.5.
- > The longer you let it drive without constraints, the worse the wreckage gets. The velocity makes you think you're winning right up until the moment everything collapses simultaneously.
In my experience (so far), I can’t let the LLM write too much in one go.
I need to test the hell out of what it gives me, and I can’t ask for too much, at one time.
I tend to ask it to “flesh out” functions, where I have a signature, and a detailed headerdoc comment. I will provide a lot of guidance about the context, often attaching relevant files.
Even then, it often doesn’t give me what I need, first time, unless it’s a small function, with extremely limited scope.
That said, it’s been extremely helpful. It has accelerated my development greatly.
I have found that it gives me much better PHP, than Swift.
I suspect that may be because PHP is extremely mature, and there’s millions and millions of lines of high-quality code out there, in open-source repos, while Swift is probably mostly in closed repos, with open stuff not really provided by experienced developers (it’s a proprietary language used for shipping commercial software, so that may also apply to other languages).
What it gives me in Swift, most closely resembles stuff that enthusiastic newer folks would do, and want to show off.
- No, there are millions upon millions of mediocre lines of code out there.
And LLMs tend to converge on mediocrity. Which is totally fine.
by OtomotO - > In my experience (so far), I can’t let the LLM write too much in one go.
Second, but I've found a cheat code to make it much farther with minimal intervention.
Step 1: tell them your goal, have them generate a doc, include design principals, system invariants, and acceptance criteria.
No amount of CLAUDE.md or skills beats re-iterating the focus points directly in the prompt.
Step 2: tell them to summarize the doc (pay close attention here). Have them save it somewhere (I use docs/agents) once you're happy with it.
Step 3: tell them to build a detailed plan to meet the objectives of the doc.
Step 4: let them go wild.
Step 5: once they declare "done", feed their progress to another LLM (Gemini is quite decent for review, and free) -> mindlessly feed the feedback back to the implementing LLM.
Step 6: Say the magic words: https://github.com/cuzzo/clear/blob/master/docs/retrospectiv...
Again, I've found no amount of skills or CLAUDE.md beats slightly modifying a prompt to meet your exact goals specific to the design and what you know of the implementation so far.
Step 7: Have them rebuild a plan to address feedback.
Step 8: Let them go wild. Loop back to Step 5 until the LLMs tell you there's no major action items.
Step 9: Tell them to remove anything from the commit that's not strictly necessary, get rid of comment changes that aren't strictly necessary, etc.
Step 10: here and only here do you invest your time (worth 100x what you're paying them) to look at what they did. Here you can give them feedback to address anything you saw.
Step 11: Review.
Step 12: Profit $$$
I got a quite decent implementation of Finite State Machine and Thunk + Trampoline transformation of code in custom language I'm building in about 1 day, barely checking in while commuting to and from work on the train...
Occassionally, at step 11, you will find a gigantic turd and wonder how the LLMs converged on this. But, typically, it's at least good enough at that stage.
I don't even waste my time looking at anything they've done until they've converged on a good design and implementation with no holes, no feedback, no notes that does what a minimal, summarized doc clearly states and follows the design principles. Because they DEFINITELY haven't in a one-shot.
- > What it gives me in Swift, most closely resembles stuff that enthusiastic newer folks would do, and want to show off.
The same is true for rust-lang. Code that will immediately clone/re-allocate anything passed by reference and collect everything to the heap that is passed by `Iterator`/`IntoIterator`.
It is a massive performance anti-pattern and the hallmark of somebody "struggling" with the borrow checker. Naturally a lot of 1st & 2nd 'I just learned rust' projects lean on it. Which is totally fine for humans, you're learning. But with LLMs that pattern is now burned into their eigenvectors with the heat of a billion hours of H100 training time.
It has gotten to a point that all code I generate with Opus or Codex if there as iterator or reference in the argument, I start a fresh context, with a sort of `remove unnecessary clones, collections, and copies from the following code: {{code}}`
by hugmynutus