Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- I experiment a lot with local LLMs, particularly small ones like Qwen3.5 4B and 9B. I have build multiple experiments to make harnesses that use these models for code generation, planning, local search, etc.
These are really good models but the harness has to be built around them. I have a ton of generated system prompts for specific purposes. Even parts of a SolidJS stack, for example Route management, has its own prompt. These are experiments but the results are real. If we build harnesses around small models, we can build a locally running WYSIWYG editor which works on plain text prompts.
The performance, in simple tokens/second, is not the most important factor. For many private data points, like emails, I would rather have a local graph based search and LLM on top where the harness is specific to problems like calendar, contacts, finance, etc.
I run all experiments on an 16GB M4 Mac Mini but coding agents building the harness are a mix of Codex, Claude Code and opencode.
by brainless - > Running a large model locally comes down to one thing: how much RAM it actually needs in memory.
Not completely true. It's memory AND memory bandwidth. You can have 1tb of memory but if you have awful memory-bandwidth you'll also have slow tok/s. A3B helps with this, but so does MTP.
From my experience, you'd be better off running the dense 27b-mlx with MTP than the 3.6 version with A3B. You say your model is ~20GB of ram, but the 3.8:27b-mlx is 18GB and gets me very reasonable tok/s, and greater speed if you disable thinking when not required.
by c16 - The dense 27b Qwen on M4 Pro has a prompt processing speed of around 125tok/s which makes it ok to ask a quick question but impossible to use in an agent, as processing the first prompt of the agent with the tools and instruction can easily be 10 000 tokens
In this case the 35b a3b makes sense as it has a PP speed of around 800tok/s
by Kayou - Running it depends on RAM, which is what I wrote, bandwidth is important for speed. I chose my words carefully, but you are absolutely right.
- Maybe a silly question, but is there a reason/advantage to using mac minis over any other kind of small computer/laptop, ie running linux?
My understanding of using a mac mini for ai (ie running a claw bot or whatever) is to have it 'always on' and a better price/performance profile than a cheap vps.
Is there performance (silicon processor?) so unique? As I guess it's not their graphics units. I see tonnes of people using mac minis for AI, to the point it almost became a meme.
Edit: yes I know this article is about local models, my question is a bit more general.
by t1E9mE7JTRjf - No, it's just hype and people cargo culting local LLM guys buying maxed out Mac Studio for its massive and relatively fast GPU-assignable RAM.by numpad0
- The main difference total other laptops of non-Apple make is/was unified memory (graphic VRAM + RAM) architecture. No need for an extra dedicated graphics card to get 64+ GB VRAM.by wuschel
- Unified memory is the answer, it’s one of the cheapest ways to get >64GB of gpu addressable memory.by bbatha
- There are probably three main reasons: 1) unified memory (but you can also that with DGS Spark / Strix Halo), 2) access to your Apple account, so you can have a bot handle your iMessages, email, calendars 3) energy efficiency.
Apart from that, if it doesn't work out you still have a Mac Mini, which in itself is more desirable for many than a DGX Spark or Strix Halo if you have no AI use case.
by em500 - Quite a lot of "local doesn't work" in here - unfortunately, often with not much details about what the people actually want to use their models for. Which I'd be curious about.
I, personally, do use frontier models in the cloud for a lot of (meta-)cognitive analyses that are heavy enough to have me run against the limits of payed accounts regularly - so I'm neither a Luddite nor stingy with cash in this case.
However: I have pretty good experiences with local models as well. My solid but hardly extreme desktop (with one RX 9070 XT 16GB) mostly serves gemma4:12b and specialized models (embedding) to my local network. This is for general use like simple queries, simple code, reformatting and the like but also for two specific tasks that are permanently running:
a) It's connected to Home Assistant (as a second stage after very simple "turn light XY on" commands which get processed without LLM). So, I can mumble into my smartwatch "computer, how much gas do we have in the warp core and how much energy did the bussard collectors make from the cosmic dust today?" (or describe a more complex light scene or create an automation I want or whatever). The phone transcribes that - with a local model on device - and fires it to the desktop who has agentic access to HA, looks through the sensors and data, sees that I've tagged my solar panels and battery with nerd vocabulary. It makes the right conclusion, converts a few units and gives me back a nice overview. All hands-free while I'm sitting on the toilet.
b) It's the LLM backend for a personal radio station run by a fleet of nerdy/quirky AI DJs who's archetypes are represented more than well enough in the latent space of the "small" model to produce funny results. The DJs can produce consistent, individual segments and programs, run a playlist that works well for me (based on multi-layered audio analysis that also uses local LLMs), respond to song wishes and generally produce much better recommendations than Spotify ever could for me. And you can also put multiple of them in the "studio" to create hilarious crossovers that you would not get from a commercial entity because the IP owners would rather shoot each other in the face.
All of this doesn't even max the available resources, so I can shovel F5-TTS into the VRAM as well and have all my DJs have good, locally created voices (or voice clones of Captain Picard and Han Solo, if I wanted to) based on zero-shot voice cloning.
--> Far from "unusable". It just depends on the task. And I neither have to hand my keys to the Navidrome server nor to my Smart Home to any entity outside my local network.
- Tip: include pictures of your machine. Not sure if I’m alone in this, but I love to see other people’s setupsby sdevonoes
- Apple did great work convincing people their unified memory was good at AI. Even AI says Apple is the best of all time at marketing.
Meanwhile the stock market has Nvidia at the top... Until everyone gets cuda.
- Apple is working from the 'desktop' up to beefy servers with 64GB+ RAM. Nvidia is working from the 'datacenter' down to beefy racks with terabytes of RAM.
There isn't really an overlap yet.
Individual Nvidia cards exist on desktops but they're not really oriented for regular inference so individual developers are left with Macs or datacenter resources as their options.
by AdamN - > The main reason to run local: cloud APIs are rented land. They can change their pricing, hit your usage limits, or swap the model being served behind the scenes whenever they feel like it.
Yes but it's easy to replace them.
The main reason should be privacy.
by amelius - My biggest problem with running local LLMs on my M4 Max/128GB RAM is the prefill latency.
I've since acquired two DGX Sparks, and it feels so much snappier.
by jumploops - m5 max really fixed pp with the better matmul support, im sure the m5 ultra will be even crazier
the sparks have much slower memory bandwidth is the trade off
by c0rruptbytes - Would you mind sharing your local Mac setup and which models you currently use and whether it’s GGUF or MLX? I’ve the hardware same specs.by shell0x
- Recent performance data on my M1 Max 32GB MacBook using oMLX. I have been working on identifying suitable model and config for my use case and system. Using a refactor and suggest improvements prompt for a specific Django code block using VSCode Cline extension.
---
Qwen3.8-27B-4bit, Prompt Processing (PP) 66.3 tok/s, Token Generation (TG) 11.8 tok/s
Ornith-1.5-35B-A3B-MLX-4bit, PP 379.7, TG 45.8
Ornith-1.5-35B-A3B-MLX-4bit, PP 381.5, TG 46.4
Qwen3.6-35B-A3B-mxfp4, PP 389.6, TG 47.6
Qwen3.6-35B-A3B-OptiQ-4bit, PP 342.6, TG 44.4
---
Qwen3.8-27B-4bit generally runs out of output token before completing the task though excellent partial results.
Ornith-1.5-35B-A3B-MLX-4bit seems to get in the loop often specially with tool calls.
Qwen3.6-35B-A3B-mxfp4 seems to be optimal with speed and quality output.
I am going to test Qwen3.6-35B-A3B-4bit soon with same code block just to check my intuition that any derivatives don't seem to perform better than the originals.
by akg_67 - Interesting, what's your Context Window?by madduci
- > Prompt Processing (PP) 66.3 tok/s
I got 400 pp tps on a 10k token input. Your numbers seem suspiciously low, maybe the input was too short to measure properly? And this dense 27B is slow, the MoE A3B models get to 1000 tps.
by visarga - Nice setup; but, for simple tasks or questions, AI is currently free? And it will probably stay free, as I don't see Google starting to charge for using AI on its search engine? So costs can't be a motivation for running small models locally?
For more complex or important tasks, costs, autonomy and privacy matter, but then so does performance/quality.
So I'm not completely convinced it's really worth it; but it's tempting!
by bambax - At some point and for some tasks, predictability is important if not critical.
I’d rather use a tool where I know the limitations, over a tool where the limitations and strengths keep changing.
This way I know where in the process I ought to step in and pay attention.
- Getting undeserved safety filter refusals, watching LLM accidentally reading what you might not find comfortable showing, and hitting free tier/basic subscription quotas, are not fun. Going local removes somewhat irrational anxiety that come from those.by numpad0
- You are correct. In large part, the cost of something like Gemini on a very basic Google AI plan provides far more utility than local LLMs for coding assistance.
There are 2 main reasons for running local LLMS.
1. Process private data/work with uncensored models.
2. Use a large amount of inference that would quickly blow through rate limits and/or run up API costs.
The thing that is critical for 2 is that a) you have to have a sweetspot between a pretty good model, which means largest parameter counts, and fast enough token generation where you can run agentic loops. The latter is needed because you aren't going go get the "intelligence" of larger models to form shell commands and run tools to figure stuff out, so the only way around that is to have custom agentic loops to force the model into doing what you want, which results in more text processing.
From my testing, Gemma4:31b is basically the only local model that can be relied upon to produce accurate results. Qwen models chase benchmarks, which results in MoE models (thus the A3B in the model, i.e 3 billion parameters are only active during inference). In general, these are good for very specific tasks, but fail to be accurate in considering cross task data, whereas Gemma, being fully active does a much better job. If you only need to do a very specific deterministic task, those models are pretty good.
As an aside though, if your task involves pure text processing (for example take html data, make it into a markdown document), you can also additive train Gemma270M quite easily all on CPU, and on a decent CPU it gets like 50-100 tok/sec, no need for any extra hardware.
The thing with Macs is that while they can run those models and larger models no problem, the tok/sec is very slow. This limits effectively what you can do with the models. On the M4 that the poster mentioned, Gemma:31b will run about 20 tok/sec. That means that when you wants to write a whole code file or process large context, you have to wait for it to do things. Compared to workflow with larger models, where file generation often takes like <10 seconds, it takes a while to adapt.
The only benefit of using Macs is the price for Mini and cheaper studios. However, once you reach the total cost of about 2.5k (note that the M4 statedin the article us about 2k), building a gfx card rig is the way to go. You can get 100 tok/sec on a 3090, and it will feel a lot like the cloud models.
by ActorNightly - Only “Free as in free beer.”by Den_VR
- It's free like ads are free. Or certain kinds of advice.by ajb
- I figure most free AI is free as in free electricity in the coffee shop. You're welcome to use it for small reasonable loads, but try to build anything off of it and you'll soon find yourself barred from the establishment.
And that's probably good, otherwise the free ai would just be unavailable for everyone else
by RugnirViking - Traditional search is “free” too, but you see ads. If something looks free, then you are the product.by nzxt210
- I'm the author - hello! I talk about it in the blog post - knowing what's being run, knowing where it's being run, and not having anyone else control it.
- No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?by amanzi