Comments
Hacker News
Own a framework desktop 128gb.
GPU bandwidth limits the amount of tokens/sec that you get.
I 've mostly been running QWEN-3.6-35B-A3B at Q8 and QWEN-3.5-122B-A10B at Q4 with Q8 kv-cache on llama.cpp (use vulkan version). Also use MTP GGUF's you get ~ 50% speed up when predicting 2 or 3 tokens.
Haven't tested the new laguna S 2.1 yet.
Another thing I liked was that I had enough ram to also run an embedding open model for LLM Wiki apps.
Overall I think it's cheaper and better (intelligence wise) to get a $20 codex sub and always run gpt-5.6-luna than getting a desktop for local inference. Unless you have explicit needs for it (whether it is just experimentation or you have personal things you 'd rather keep on your computers or w/e). OpenCode go at $10 is also a great option.
by Iolaum
I'm running qwen3.6, gpt-oss and embeddinggemma with Ollama on Ryzen 7 8700G + 96 GB DDR5 with 12-14 tokens/sec. Consider this as a floor. On Strix Halo it will run 4-6 times faster depending on memory bandwidth. For my local task 14 tok/s is quite enough for the price I paid.
by denn-gubsky
Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Own a framework desktop 128gb.
GPU bandwidth limits the amount of tokens/sec that you get.
I 've mostly been running QWEN-3.6-35B-A3B at Q8 and QWEN-3.5-122B-A10B at Q4 with Q8 kv-cache on llama.cpp (use vulkan version). Also use MTP GGUF's you get ~ 50% speed up when predicting 2 or 3 tokens.
Haven't tested the new laguna S 2.1 yet.
Another thing I liked was that I had enough ram to also run an embedding open model for LLM Wiki apps.
Overall I think it's cheaper and better (intelligence wise) to get a $20 codex sub and always run gpt-5.6-luna than getting a desktop for local inference. Unless you have explicit needs for it (whether it is just experimentation or you have personal things you 'd rather keep on your computers or w/e). OpenCode go at $10 is also a great option.
by Iolaum - I'm running qwen3.6, gpt-oss and embeddinggemma with Ollama on Ryzen 7 8700G + 96 GB DDR5 with 12-14 tokens/sec. Consider this as a floor. On Strix Halo it will run 4-6 times faster depending on memory bandwidth. For my local task 14 tok/s is quite enough for the price I paid.by denn-gubsky