Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- I'm running qwen3.6, gpt-oss and embeddinggemma with Ollama on Ryzen 7 8700G + 96 GB DDR5 with 12-14 tokens/sec. Consider this as a floor. On Strix Halo it will run 4-6 times faster depending on memory bandwidth. For my local task 14 tok/s is quite enough for the price I paid.by denn-gubsky
- With 128 GB of unified memory, large models can fit into memory, but memory bandwidth quickly becomes the limiting factor. With long contexts in particular, the KV cache can consume a lot of memory and bandwidth. Quantizing the KV cache to Q8 roughly halves its memory usage, making it one of the most effective optimizations. MoE models are also well suited to this hardware because only a portion of their parameters is active for each token. As a result, a 122B-A10B model can run better than a dense 70B model.
- Own a framework desktop 128gb.
GPU bandwidth limits the amount of tokens/sec that you get.
I 've mostly been running QWEN-3.6-35B-A3B at Q8 and QWEN-3.5-122B-A10B at Q4 with Q8 kv-cache on llama.cpp (use vulkan version). Also use MTP GGUF's you get ~ 50% speed up when predicting 2 or 3 tokens.
Haven't tested the new laguna S 2.1 yet.
Another thing I liked was that I had enough ram to also run an embedding open model for LLM Wiki apps.
Overall I think it's cheaper and better (intelligence wise) to get a $20 codex sub and always run gpt-5.6-luna than getting a desktop for local inference. Unless you have explicit needs for it (whether it is just experimentation or you have personal things you 'd rather keep on your computers or w/e). OpenCode go at $10 is also a great option.
by Iolaum