Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Is this really worth it, though? Throughout the years my experience with quantized models has been that they feel like a lobotomized version of the original. Doesn't matter if it's an LLM, dedicated diffusion model or some other dedicated task. Sure, they get the job done. But a lot worse. The only ones that can somewhat hold up are the ones provided by the vendor directly. Gemma4 comes to mind. However I suspect they have some secret sauce other than just "let's quantize this" since they have the original model and its data at hand.
There should be more native 4bit, 1.25bit and likewise models. Those actually work great while making them smaller in comparison. But I guess there is some reason for them being pretty niche.
by numlock86 - it is not a flat quant but a dynamicby nicman23
- On a model of this size quantization has much less impact on quality of output. I'm running a 3bit version and find it comparable to sonnet, almost opus.by iaw
- I really don't think anyone is going to have a good time trying to run it on anything with 256GB of RAM no matter what the post says. 512 is the much more realistic minimum. I'm fortunate enough to have two 512GB RAM dual xeon workstations in my home office that I bought cheap before the price rise to mess around with things...by walrus01
- How is this model half the size of DeepSeek V4 Pro? Is it because DeepSeek did more aggressive cost cutting on the attention mechanism?by andai
- Can somebody help me understand the Quantization Analysis? It says "dynamic 4-bit UD-Q4_K_XL and dynamic 5-bit UD-Q5_K_XL are generally lossless" while showing a top-1% token agreement on the chart of 97.5%. Not what I would consider "generally lossless". Is this implying that some post-processing is going to account for the 2.5% loss? Beam search?by CGamesPlay
- Generally 97.5% token agreement is very positive. Like the article explains, the difference isn’t the model thinking the capital of France isn’t Paris, but rather maybe saying “The capital of France is Paris” instead of “Paris is the capital of France”.by dannyw
- I bet OpenAI and Anthropic hate the timing of glm 5.2.
Kinda shows they have a headstart rather than a magic moat
by Havoc - Nope, GLM 5.2 is only the latest and greatest in a long line of open-weights models. There are even fully open source models that are comparable to o1-mini (OLMo), or almost-fully-open ones that are comparable to o3 (Nemotron).
I'm super grateful to the open labs (who, importantly, do not have the word 'Open' in their name), all the more so to the likes of Ai2.
There is no magic moat indeed. It is math, engineering and of course copious amounts of data (and the political maneuvering required to secure it, e.g. how most everyone has trained on Anna's Archive by this point).
by achrono - I feel like the gap is closing to be able to run good enough models locally even for coding and I would assume it could make some companies a bit nervous. Am I wrong about that?by pheggs
- Honestly, Qwen3.6 is already what you need for the large majority of tasks.
(I only ask Opus every 5 to 10 requests, when my local Qwen fails or when I encounter a situation that is too world-knowledge specific to be worth asking, but that way you can live easily with Claude's cheapest plan without ever facing usage limit).
by stymaar - The hardware requirements to run this locally are still very high. Seems far enough off mainstream for those companies not to be too worried yet.
- locally on what hardware? something like the new dgx spark, ryzen halo, or mac studio will cost you ~ $4k plus whatever you pay for power. at the rate AI is currently progressing, i think you'd be optimistic to consider that as having a 2 year depreciation.
for $4k, you can get 20 months of claude max 200. i'd take claude over the hardware.
anthropic will have something to worry about when you can run a local model on your macbook that can code. but i think we're quite a ways off from that.
by notatoad - It's not economic to run them locally. It's amazing for privacy, and fun hobby. But you're either looking at super slow CPU builds with $10k in RAM, $90k worth of GPUs, or a really quantized model that doesn't compare in quality.
I might build one for fun, but it's not going to change the economics alone. Still exciting it's possible.
by scosman - The RAM requirements are still pretty painful.by fny
- You don't even need to run them locally for them to be a threat. Plenty of companies are looking at paying third party companies to host these models and they come in at fractions of the price of the frontier labs.by simplyluke
- I don't think so. I could easily see a company deciding to host and run these models for their own development. If you have a dev team of about 10 people, a one time $50k investment in an LLM server has to be pretty tempting. Unlimited tokens, decent performance, upgrade options, and potential product integrations.
For companies wanting LLMs in their products in general, I have to think going the local llm route is even more tempting. Somewhat dumb models are more than good enough for a lot of the things people are integrating LLMs into their products.
by cogman10 - If we didn't have a RAM/GPU shortage right now they would be more nervous than they are. But as it is very few people are going to be able to afford a rig that can run this model effectively. That's probably not going to change for several more years yet. I think if the Z.ai folks decide to come out with a flash version of GLM-5.2 specialized for coding that came in about about 80B params, then the US frontier labs would probably be more worried. Overall, the Chinese AI companies have been showing the way to do the same amount with less (sometimes much less) and as that trend continues it's going to make the frontier labs worried - but even the Chinese AI companies are going to want to protect their moat by not releasing capable models that are significantly smaller than their current flagship models. AliBaba Qwen seems to be there now - it's gotten mighty quiet from them lately - their latest 395B model is just too large for most folks to run at home and they don't seem to be making any noises about releasing smaller ones this time around.by UncleOxidant
- The most interesting part of this to me is not the benchmark table, but the packaging.
A model like GLM-5.2 being available as GGUF, usable through llama.cpp/Ollama/vLLM/SGLang/LM Studio, and wrapped for local agent workflows changes the category. It stops being an impressive open model exists and starts becoming this is something a small team can actually put into its development stack.
For instance, company buys an RX6000 setup for say $15k total. They could use this for handling data heavy sifting that would otherwise be a lot of Claude tokens.
It doesn't need to be as good as frontier-best. Just good enough.
I could see a business of people packaging this and handing it to companies who want Help Desk bots without any extra setup.
by draginol - > For instance, company buys an RX6000 setup for say $15k total. They could use this for handling data heavy sifting that would otherwise be a lot of Claude tokens.
Considering they might be spending thousands per month on API costs already, dropping 15K to save on one process might not be bad. On the other hand, also an opportunity to sell GLM 5.2 inference at near cost to other companies for less than whatever Claude costs. In theory it costs anywhere from $0.51 to less than $2 an hour to run it and use it 24/7 that's still wildly cheaper than calling Opus which doesn't bill per hour, but per million tokens, drastically higher. Hell, you could probably bill at $5 per GPU hour and still be cheaper. Whether you're looking to self-host or sell hosting for it, it looks way cheaper regardless. I think most decent open models will continue to fit in at least 32GB of VRAM so a 6000 Pro GPU is more than enough. alternatively, even on a 5090 you can get a reasonable amount of inference for way less than paying for Opus, Qwen would be your friend there though.
- There is a push from multiple directions at the same time:
- new AI desktops with GB10s. They are relatively cheap and you can cluster them and load 1TB of VRAM
- Nvidia, amd, intel, Cerebras etc pushing new hardware
- oss models getting crazy good, like glm 5.2
- flash models getting very good like deepseek V4 flash
- quantizations
- harnesses being able to use different models (big for difficult stuff, small for grunt work)
So hopefully soon for the ones who want to break free from APIs, we will be able to host at home a cluster of AI desktops at a reasonable price with Opus-level capabilities, can't wait!!
by Frannky - Hope you're right! Can't wait!
- My hope is on Intel Crescent Island with 480GB. I don't need 8x H200 performance (and cost), but I would like to run GLM 5.2 Q8.by Tepix
- I feel like "relatively" is doing a lot of work, there: at about $4k per GB10, that's $36k for a 1TB cluster. Cheap compared to equivalent H200's, but out of reach for home labs that aren't funded with OpenAI or Anthropic RSUs.by khafra
- "it can fit" on 256GB of RAM, but it will be heavily quantized and still run very slowly. The headline number is not token generation, its prompt processing. So if you get 10 tok/s and an API gives you 20-30 tok/s, it doesn't seem that bad on its face, but a mac studio or any other machine that's not loading all of it into GPU will do PP 20-50X slower than a purely GPU based setup, which is what actually makes this unusable without $50k in GPUs.
On top of that, you will still be heavily quantized.
- A nvidia spark thingie has 128GB unified RAM. They also have a dual port version of one of these things: https://www.nvidia.com/content/dam/en-zz/Solutions/networkin.... ie 2 x 100GB/s ports, they may even be 2 x 200GB/s. Once I've got my paws on one, I'll know more.
You can cluster these beasts too. Two and three (with two IP subnets) is fairly obvious. Four or more might need a switch depending on how much network latency affects things.
Apple seem to have forgotten about M series with gobs of RAM. I can't get the Apple shop to show more than 96GB of unified RAM and that costs a kidney.
by gerdesj - So close! My machine with 192GB RAM + RTX 3090 24GB can almost run this. It says it needs 24GB of VRAM and 256GB of RAM for MoE offloading.
https://unsloth.ai/docs/models/glm-5.2#usage-guide
In a prior thread, someone said it would take $500k in hardware:
by xrd - Funny I casually asked Gemini and it said 500k for unquantized with decent throughput.by uberex
- I have the RAM, but not the VRAM. What kind of speed/tps could you expect from a 3090 with 24GBs of RAM? I am somewhat tempted to pick a GPU with 24GBs of RAM.by cheema33
- With 2 wouldn’t have good results. Ideal range for coding is at least Q8.by mgambati