Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Waiting for MLX version
- Is it possible to use a model that needs around 64 GB VRAM if you have four GPUs with 16 GB VRAM each?
- Yes, but unless they support NVLink (they don't), it's quite slow.by segbrk
- Yes, google "llama-server split model multiple GPU" for some concrete examples.by walrus01
- I am getting 14 t/s on my 16 GB card at full context with the UD-Q3_K_XL quant. Model link: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF.by sharmajai
- Yes, you can. Ideally though, you want to minimize the number of cards and maximize the amount of memory in each card.
Using multiple cards is one of the things that the models and software that Unsloth releases does really well in terms of ease of use and relatively good performance.
by strictnein - Of course. Models don't actually require VRAM. Nor do they require regular RAM. You could have 1 GB of RAM and swap the model to disk as you need different parts of it. And if you didn't have enough disks you could access weights via a network connection.by charcircuit
- I do use 2 amd gpus and I get high 40 for generation, 500 for pp and low 20/100 by the end of the context of 256k.
llama-server --host 0.0.0.0 --port 8089 -m Qwen3.8-27B-UD-Q8_u.gguf --spec-type draft-mtp,ngram-mod --spec-draft-n-max 3 --spec-draft-n-min 1
if you have an igpu and want to exclude or just use some gpus you can use
--device Vulkan3,Vulkan2,Vulkan1
in my case vulkan because of amd, you can see your devices with
llama-server2 --list-devices
Available devices: Vulkan0: AMD Radeon Graphics (RADV RAPHAEL_MENDOCINO) (33515 MiB, 29349 MiB free) Vulkan1: AMD Radeon RX 7900 XTX (RADV NAVI31) (24560 MiB, 4911 MiB free) Vulkan2: AMD Radeon RX 7900 XTX (RADV NAVI31) (24560 MiB, 7681 MiB free)
by xlayn - Yes, and if you have the PCIe lanes (say, an x16 lane - actually delivering 16 lanes! - to each GPU) it's also quite performant - it's called a tensor split in llama-server.
If your motherboard/cpu doesn't actually have those (few do outside some xeons, epycs and threadrippers) you can still do it - it's called a layer split and will work even with 1 lane per GPU. Each GPU will work at its maximum speed, but only 1 will be active at any given instant - imagine a relay race.
(Didn't mention which PCIe generation - obviously the higher the better. At v4 and up, even 8 lanes per GPU would be enough for a performant tensor 4-way split)
Edit: If you have more than 1 user at a time, the GPU can actually all be working all the time, if there are enough parallel requests to serve. But you need enough KV cache for all the sessions you're running in parallel.
by gruturo - These are very good!
I'm hoping for speed improvements because the only problem running the 27B model on my Macbook pro (M4 Max) is the speed: 20 tokens per second. I benchmarked and MTP actually makes things slower, so I disabled MTP altogether. I'm hoping there will be some breakthroughs or optimizations that will allow me to run this at 30-50 tokens per second, which would make a big difference.
by jwr - I have a 36gb M3 Max. I tested it across quite a few different options: llama.cpp, oLMX, ollama with different options.
So far ollama managed to be the most performant of them all. I will get 30 to 40 tokes/sec with it when using the -mlx version of Qwen3.8.
Whatever the sauce the ollama folks baked into the mlx + MTP mix is currently working the best out of the box.
by m1keil - The new IQ4XS has been working pretty well so far on 4090 16gb.by jadbox
- What size context are you able to squeeze in with less than 2gb of headroom? I have had some luck using a quantized kv cache but i fear that also decreases overall quality.by kamranjon
- No Dynamic 3.0 NVFP4 quants just yet from the look of it, as a heads up. Would love to see how those perform relative to others on the curve.by jjcm
- Might be off-topic but: is it possible to perform such a quantization on Apple devices? Something like Mac Studio Ultra M1 (even if it would take weeks/months)?
- Quantization is typically very cheap and fast. It can even be done on hardware that does not fit the model, by processing the weights layer by layer.
I use this project: https://github.com/vllm-project/llm-compressor
by verdverm - Just quantizing takes seconds-to-minutes, llama.cpp provides a nice tool[0]. Improving quality is then a matter of picking specific tensors to maintain at higher accuracy, checking on representative data, and repeating.
[0]: https://github.com/ggml-org/llama.cpp/blob/master/tools/quan...
by kristjansson - Unsloth use a property dataset they don't release, however you can indeed create quantisation locally on your machine and it's pretty easy, llama.cpp comes with everything you need.by smcleod
- Since it seems like this not only improved sizes but also performance I can't wait for some benchmarks and comparisons. If you don't have a separate GPU for inference, every single GB matters so a comparison between specific Q4 Quants is really interesting to me.
Currently I very much can't decide between going for a bit of a lower Q4 Quant to squeeze out a bit of buffer and ctx or wondering if a slightly higher (IQ4_XS vs Q4_K_M/XL) is worth it
This is crazy! But has anyone tried these lower quants on real projects?"We also made some smaller UD-1bit quants with UD-IQ1_S being 6.2GB (without MTP) which retain around 72% top-1% accuracy yet being 89% smaller"by throwa356262- Not 1-bit, but I’m getting pretty good results with some light coding using unsloth’s previous 2-bit quant of qwen3.8-27b. With these new quants i may be able to bump up to 3bit, tho it’s already running so slow (15tok/s average for the first 32k of context) that the speed hit might make it not worth the extra smartsby kennywinker
- I tried some 1-bit, 2-bit, and bonsai quants against closed eval sets. They were essentially useless for my case. The little errors accumulate and send the whole output off track quickly.
If you had some use case with very small output sequences they could be interesting to try. I think dropping down to a 9B-class model would produce better results for most cases.
by Aurornis - Are there benchmarks for the various Qwen3.8-27B quants that actually measure writing code, maybe even with multiple steps? Low KL divergence does not mean much when the model gets stuck in doom loops all the time.
I could of course download and test myself, but that would take days with my internet connection.
by johndough - Purely an anecdote, but I've found Qwen3.8-27b doesn't doom loop like previous Qwen models would. With that said, it absolutely thinks in circles- it'll prepare to do something, say it is now ready to do it, then follow that with three paragraphs that all start with Acutally... Oh wait, I should check first... Hmm, hmm... I should stop guessing and just do it. Okay, I'm ready to do the thing now... Actually, wait...
It takes forever, but it does actually get around to making things work, and it is more thorough and produces better code than previous qwen models. You just need to let it run quite awhile.
by zdragnar - We made something called Divergence-300 @32 (and later @512) which tests actual inference across 32 tokens on a held out test (Terminal Bench, DeepSWE, Math etc)
We do plan to do larger benchmark suites though!
- I tested Qwen 3.8 on the Blade CTF last night, it took 3 hours but got the correct answer.
I know that didn’t answer your question but I was looking for a test suite and couldn’t find anything.
After reading the logs, there is far less doom looping than with 3.6, but whether that’s a one off or not is up for debate.
Q4_K_P
- Hey Unsloth, your gguf are the first ones I look for when I want to download a gguf model. Today I was trying in fact to see, what's the smallest Qwen3.8-27B that I could run and get good results, say restricting it to 16GB of ram.. so I went, pick up the Qwen3.8-27B-UD-IQ2_XXS.gguf and them BAM, error on MTP... now I understand why after reading your announcement. Beyond the space saving, why removing the MTP? improves speed exactly for the group that could benefit from it.by xlayn
- Q2 quantization is basically giving a capable model a lobotomy. It will not accurately represent how smart or capable something like qwen 3.8 27B in Q8 will be.by walrus01
- you can still have it, no?
> We also removed the MTP module from smaller quants under UD-Q2_K_XL (8.37GB and lower) to converse around 500MB of disk space - you can use the Q4_0 MTP separate module if needed
- Hey we did not remove the MTP for sizes above 8GiB - but yes for small GGUFs under 8 ish GiB, we removed the MTP module (IQ2_XXS and lower), because it's 500MiB to 750MiB in size, and on small 8 GiB machines, even 500MiB is needed.
As someone in the comments said we made a separate Q4_0 MTP if that's helpful so you can use that.
But I would suggest using UD-IQ3_XXS for 10.9GB for 16GB machines or Q2_K_XL