Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- "The layer-by-layer streaming is clever. Curious how the throughput compares to running a quantized model on the same GPU — seems like quantization might still win on speed?"by junsu22
- “Can run” and “is useful interactively” are different benchmarks. At this latency, I can still imagine batch or overnight jobs being interesting; for chat, time to first useful answer matters much more than whether the weights technically fit. A workload/latency/energy table would make projects like this easier to evaluate.by Alisaqqt
- We will see if this project has legs. This is the kind of efficiency we desperately need. Now if we can address efficiency with llm training.by book_mike
- What device do I need and how much will it cost to install one at home so that it works as quickly as the Claude Code answer (and it answers quite slowly)?by mrwaip
- If you want it to respond in under a minute, you need more hardware than this application is intended for. This thing's response is measured in seconds per token, not tokens per second.
To get Claude Code responsiveness from even a pretty small (but still usable) model, you need, maybe two 32GB GPUs? I run Gemma 4 31B and Qwen 3.6 27B on my dual 32GB GPU setup (cheap old Radeon Pro V620 GPUs) at about 20 t/s, which is not fast enough for comfortable interactive agentic use. A couple of new GPUs, like Radeon AI Pro 9700 at $1400 each, probably gets you fast enough for comfortable interactive use with small models like those. Those small models are not competitive with Claude models, however (maybe they beat Haiku sometimes). They can write a little Python or make a web page, they can't architect a real application.
To run a near-frontier model like Kimi K3 or GLM 5.2 at comfortable speeds, you need serious hardware with 768GB VRAM, minimum. I think Asus is releasing something like that for about $150k soon. You can run DeepSeek V4 Flash at almost comfortable speeds and in a decently capable quantization on two DGX Sparks or Asus GX10s (about $10,000).
Or, you could use DeepSeek V4 Flash from DeepSeek.com, at blistering speeds and with huge contexts, for something like a decade or two for that same $10,000.
by SwellJoe - You can run any frontier model on your PC if you just wait long enough...by xg15
- I guess the use case is something like: you have a slightly obsolete Mac or PC or a whole bunch of them, and just need to compose one or more convincing spam emails, but it's fine if it takes a full week to do it?by ilaksh
- And you don’t pay for the electricity.by speedgoose
- I’m still slightly confused on what this adds.
Let’s say I wanted to run a full size open weight model. I have a 128GB m3 max laptop.
Does this basically load layers in and out on demand? So I still have to download the full model to disk, but the RAM requirements go way down? The readme calls out that one still needs to connect HuggingFace, which leads me to believe that maybe you don’t even need to download the full model?
by cpfohl - If you point it at a huggingface model identifier, it will download it, I assume. No way around that.
It reads like it is keeping only the core and the active layer loaded at any one point, and streams layers from disk; there are several other solutions like this and if my understanding is right, this is probably better than an mmap implementation or just streaming experts in.
by dofm - I love it how the rampocalypse is pushing people to squash all the performance they can. I hope that this also leads to rethinking model architecture so that they can do the same with less.by seu
- I've been seeing a lot of projects that let one use large models on machines with small amounts of memory. They seem to all be doing significant quantization and/or expert streaming. What's the benefit of these projects over something like taking an unsloth quant and running llama.cpp with appropriate flags (-cmoe/-mmap) to manage VRAM vs RAM vs SSD?by hatthew
- Seeing a lot of these “run 1TB models with 1GB RAM” projects recently. Most seem vibe coded and probably won’t be maintained.
Hoping a winner emerges with some real momentum behind it.
by roger_ - I love how goal posts are shifting from "vibe coded apps dont really work" to "vide coded projects wont be maintained"by mkagenius
- You don't really need a maintainer when codex or claude code can set it up for you; thats how I got Trellis2 working on windows and tiny VRAM despite Microsoft recommending you have 24GB VRAM and Linux. Models are pretty disposable now.by hiramwen
- How a patchwork of Python modules constitutes an API(?) or whatever(?) and how to use it (does anyone?), is beyond me. There is nothing that I would call documentation, let alone concrete usage examples. I know more amusing ways to waste my time. When I want to play with models, I use llama.cpp.by classified
- No no, don't just say "vibe coded", say "Fable and $500 of credits"by akie
- For anyone wondering “how slow is this?”
IIUC, Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token
by imenani - I wonder what this measures in J/token.by xnorswap
- I have see a lot of "run <gigantic llm model name here> using cpu" or using 1gb of ram... and the short answer is sure, you can run it in 1mb of ram, or in a 286, it will just take a couple of years to produce the first token.
the canary in the coal that this has not change is nvidia share price.
by xlayn - At that point, how does this compare with simply running the model on the CPU?by xg15
- Ahaha thank you, I naively assumed the unlabeled graph in the readme was tps, not spt!by jackb4040
- Wow, you could do a lot at 292 tokens a sec—oh.
I have all praise for those taking this on and in my idiom would call it *the lord's work."
The image I reliably summon to mind is that compilation video showing the progress of Boston Dynamics bots. The curve between technically functional, to comically slow, to too slow for "real" work, on to, OMFG, may prove a (rough) curve.
It's work like this that moves things forward.
by aaroninsf - I do think these “run a bigger model than will fit in VRAM” projects are necessary steps, but are they functionally useful or helpful to anyone currently? For example, is anyone out there running a big Qwen for coding on a 16-32GB machine with these techniques?by bensyverson
- that's 0.003 tokens/second. To get an hour's work done that's normally 30 tokens/second (108k output tokens in an hour) will take 416 days at this rate. And if you're using 100 watts, during that time you will spend $124.61 in electricity, as well as not being able to use your device for something else, plus the noise and heat from your device.
For $124, on Moonshot's official Kimi K3 API rates ($0.30 per 1M cached input, $3 per 1M fresh input, $15 per 1M fresh output), you can purchase 42 million fresh-input tokens, or 8.3 million generated output tokens, in whatever mix you want.
So what you get is 80x more expensive and you wait 416 days to get it.
by logicallee - I hope I'm not the only one who misread it as 292 tokens/s and got excited momentarilyby malshe