Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- What device do I need and how much will it cost to install one at home so that it works as quickly as the Claude Code answer (and it answers quite slowly)?by mrwaip
- You can run any frontier model on your PC if you just wait long enough...by xg15
- I guess the use case is something like: you have a slightly obsolete Mac or PC or a whole bunch of them, and just need to compose one or more convincing spam emails, but it's fine if it takes a full week to do it?by ilaksh
- I’m still slightly confused on what this adds.
Let’s say I wanted to run a full size open weight model. I have a 128GB m3 max laptop.
Does this basically load layers in and out on demand? So I still have to download the full model to disk, but the RAM requirements go way down? The readme calls out that one still needs to connect HuggingFace, which leads me to believe that maybe you don’t even need to download the full model?
by cpfohl - I love it how the rampocalypse is pushing people to squash all the performance they can. I hope that this also leads to rethinking model architecture so that they can do the same with less.by seu
- I've been seeing a lot of projects that let one use large models on machines with small amounts of memory. They seem to all be doing significant quantization and/or expert streaming. What's the benefit of these projects over something like taking an unsloth quant and running llama.cpp with appropriate flags (-cmoe/-mmap) to manage VRAM vs RAM vs SSD?by hatthew
- Seeing a lot of these “run 1TB models with 1GB RAM” projects recently. Most seem vibe coded and probably won’t be maintained.
Hoping a winner emerges with some real momentum behind it.
by roger_ - For anyone wondering “how slow is this?”
IIUC, Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token
by imenani