Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • Nice writeup, thanks.

    I run something very similar except for directly using pi as the agentic harness I use little-coder that wraps pi with reasonable defaults for running local models. Even though my local setup is a bit slow, it is a thrill to do real work completely locally.

    by mark_l_watson
  • Here's a visual post for using LM Studio and VS Code (and Pi): https://blog.alexewerlof.com/p/local-llms-for-agentic-coding

    One way or another local AI is the future. I actually find weaker models more interesting because it keeps me sharp (at the cost of velocity of course).

    by hanifbbz
  • I poured a couple days into custom Burn inference for Qwen3-Coder-Next only to find it doesn't come with a speculative decoder, so on my M4 Max I can't push it much further than 120t/s. That's still kinda slow, though still faster than llama.cpp's 70.9t/s and MLX's 80.6t/s with the same model. Claude Fable 5 is recommending I use the Qwen3 MTP -- I worry that will compromise the quality somewhat, but might give it a try to see if I can get more usable speeds.
    by LoganDark
  • >64 GB

    Thats the rub. I have an M4 with 48G. I wonder if it is worth testing this out.

    My past attempts (with Ollama and various LLMs) were too slow to use.

    by reddit_clone
  • I've an M1 Pro with 32GB ram and it's running pretty well
    by jmkni
  • People are using 3090 (24GB) to run models, and it is the most cost effective way to run the. Yes, it is 2x faster, but memory wise you surely can spend 24gb on llm.

    Also there are smaller, still usefull models that can run on 8GB or less.

    by krzyk
  • i’m running m4 pro 48gb right now

    omlx + gemma 12b 6 bit + pi

    it’s feasible for sure

    MoEs for speed (qwen 35b, cohere 30b, gemma 26b)

    Dense for more methodical work (qwen 27b [reigning champ], gemma 31b, gemma 12b)

    MoE i recommend 5bit+

    Dense i think 4 bit is okay

    Play with your context size, you don’t really need that much, have lazy loading for tools and mcps

    my pi extensions for anyone looking for a skinny quick setup, i have use `--no-skills` right now too:

        "npm:pi-codex-goal",
    "npm:pi-simplify",
    "npm:pi-mcp-adapter",
    "git:github.com/elpapi42/pi-minimal-subagent",
    "npm:@wierdbytes/pi-statusline",
    "npm:@aliou/pi-guardrails",
    "npm:pi-lens",
    "npm:@juicesharp/rpiv-todo",
    "npm:pi-hashline-readmap",
    "npm:@mrclrchtr/supi-review",
    "npm:pi-cmux",
    "npm:@mrclrchtr/supi-context",
    "npm:pi-tool-search"
    think of local models as "zero sugar" models and that's where we're at right now. I think it's crazy how good these models are compared to last year's frontier models
    by c0rruptbytes
  • I'm running an M3 on an Air with just 16GB. I can still get useful results without an internet connection in "chat mode". It's a different experience than using Claude, for sure, but it's workable. I typically use the Qwen variants these days.
    by codazoda
  • M4 24GB here. You'll be fine, if you're anything like me minor latency is acceptable to obtain (a) privacy (b) reliability (c) CI/CD/guardrails (d) network independence (e) future-proofing vs. AIaaS. https://omlx.ai/ gives you intelligent local hardware based model download recommendations. That said it probably depends heavily on your workload, process and polish expectations. See also https://news.ycombinator.com/item?id=48089091
    by contingencies
  • Some of these models will be a bit of a squeeze at Q4_0 I suspect; almost certainly they will be using CPU. Probably the 31B Gemma will be too much. Maybe not the Gemma-4 26B QAT.

    But if you just want to play around rather than code, you really might find the Gemma 4 12B model worth mucking about with just so you've gone through the steps. Especially if you want to muck about with image analysis or audio transcription.

    If you're writing PHP I think you could even find it good enough. I've been modestly surprised. You can do that basic fiddling with the Edge AI Gallery app, which can enable thinking and has a customisable system prompt and some agent support.

    You could also try the 14B Deepseek R1.

    Honestly even if it is not good enough, if you are anything like me, I think you'll find that going through this process is really quite educational — it has made a lot of things more concrete for me in a way that I have found reassuring and valuable.

    by dofm
  • I have a M5 MAX with 128, local models are toys compared to hosted ones. I've spent a lot of time and money trying to make it work even 1/2 as well.
    by hkchad
  • For high Ram (unified), and relatively middling to lowish Tflops and bandwidth GB/s, usually MoEs are most hopeful. The current top-1 in the (iq, tok/s, @ context depth) ranks for me (M2 Max, 96gb) is DeepSeek-V4-Flash REAP25 <65gb gguf + ds4-server + pi agent. Not better than cloud API ofc, but useful enough to endure if I need to. E.g on a non-Internet 4h flight the battery (local llm draws 60w) held long enough. REAP supporting ds4 branch here

    https://github.com/ljubomirj/ds4/tree/reap-compact-support

    DS4F dropping to unusable <10 tok/s only at 784K context (!!) makes a big difference.

    by ljosifov
  • Useful stuff in here that I wish I'd seen a few days ago :-)

    I am not convinced that the MTP setup for the QAT model adds very much in terms of speed on my M1 Max, but it is definitely worth experimenting with.

    Fiddling about with local models has done so much for my conceptual understanding of what is going on.

    FWIW and YMMV but I also found the Gemma 4 MTP head was occasionally breaking markup in Opencode, causing the thinking to display untidily and ultimately in some cases missing the stop token. So I've stopped using MTP there for now.

    Recent Qwen 3.6 models have developer role support so it will occasionally surprise you with a structured multiple choice questionnaire.

    by dofm
  • when I started using QAT recently, I stopped trying to improve my configuration after that. I will try tuning my local environment again in a few months, but with QAT things are good enough for now.
    by mark_l_watson
  • I found a marginal downside to Qwen3.6-35B-A3B-MTP vs. the non-MTP equivalent on an M1 Max. I’ll maybe experiment with settings further though.
    by mft_
  • FYI you can open Claude code in the terminal, point it at this article and just tell it to "do it", if you're feeling extra lazy
    by jmkni
  • This is the way.

    I'm not Googling much of anything anymore. 9/10 times the information is awful, it's hard to parse out of whatever other spam it's surrounded by. Meanwhile, Claude will just do the thing one-shot or with a tiny bit of refinement.

    The gateway to knowledge and getting stuff done is the LLM.

    Google Search is a dinosaur.

    It feels like we're living a century into the future. Not even smartphones were this cool.

    by echelon
  • I have used omlx.ai with great success to both download multiple mlx models (including gemma and qwen) suited for my hardware AND to be able to automagically launch both open-source and close-source (claude code, codex) harnesses using these models. All from a web or desktop UI

    You would not need to follow a blog post with omlx IMHO

    by vladgur
  • It truly is the SOTA for local inference on mac. Even when there are regressions the dev(s) are insanely responsive. It is the most impressive opensource project I've seen in a awhile
    by fridder
  • FWIW I have not, on a 64GB M1 Max, seen any advantage from oMLX specifically or MLX generally over GGUF with llama.cpp.

    The Gemma 4 MLX builds I have found so far have been slower at the same quantisation and much slower with MTP.

    The built-in web UI for llama.cpp is really quite good once you have chosen your model. Otherwise I quite like LM Studio for tinkering.

    One thing I would say is that both Gemma-4 and Qwen 3.6 simply do not need a large chunk of the typical opencode system prompt. Better off without it.

    by dofm
  • In case anyone is looking for a sandbox to go with oMLX and Pi: https://github.com/Dotnaught/pi-sandbox
    by Dotnaught
  • I've been quite impressed with DeepSeek v4 Flash running via antirez's ds4[0].

    It feels like a GPT-4 class model in terms of "stored knowledge" but is better at long-horizon tool calling than any of the GPT-4 class models.

    Running on a 128GB MBP M4 Max, I'm getting ~24 t/s on generation and ~200 t/s on prefill. I was expecting it to feel slow, and it certainly does when e.g. generating code, but it's surprisingly useful as a "machine orchestrator" for simple tasks.

    For non-agentic usecases, it's a decent enough model to converse with, and has the benefit of being entirely self-contained/private.

    [0]https://github.com/antirez/ds4

    by jumploops
  • Not sure you really need huggingface-cli to download anything if you're just using llama.cpp. You can pass `-hf ...` and it will download the models for you. Set `LLAMA_CACHE` to change where the downloads go:

      LLAMA_CACHE="models" ./llama-server \
    -hf unsloth/gemma-4-31B-it-GGUF:UD-Q4_K_XL \
    ...
    by c-hendricks
  • Yes.

    -hfd for the draft model.

    by dofm
  • I wrote a similar post some time ago just used ollama and opencode https://blog.kulman.sk/running-local-llm-coding-server/
    by ig0r0
  • this is the way, given anyone could swap for oh my pi / pi / etc
    by takethebus
  • actually useful and the ollama gui could probably even simplify this more.
    by sleepybrett
  • Did you have any issue with tool calling inside opencode? I tried the same approach, but my models don't see any tool.
    by amrtn
  • Ollama is not a good choice - https://sleepingrobots.com/dreams/stop-using-ollama/

    As for oprncode, doesn't the system prompt eat too much of the context? Local models are really constraint in regards contex, and opencode AFAIR uses a 10k of it or some thing close.

    by krzyk
  • > The benchmark prompt was:

    > Write a compact Python function that parses a unified diff and returns the changed file paths. Then explain two edge cases.

    > Each benchmark generated about 128 tokens.

    Generating 128 tokens is probably not enough for good benchmark results. MTP speedup depends on how often the predicted tokens are accepted. In my experience, the very early output has a higher acceptance rate, so short testing can give false positive speedups.

    llama.cpp includes a tool specifically for benchmarking that will sweep the arguments for you so you don't have to restart the server and send it prompts:

    https://github.com/ggml-org/llama.cpp/blob/master/tools/llam...

    EDIT: Also the section about downloading the models should have mentioned that llama.cpp has a "-hf" argument that will download the models for you. I appreciate the author for sharing their experience, but for beginners this might not be the best guide to use.

    by Aurornis