Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- The main gotcha for local models is insane hardware requirements.
Even for $10K you get mediocre performance.
- Not anymore. Qwen 3.8 27B is so great that 5k investment gives you excellent speed and quality.by nicce
- I think this is worth revisiting once we get some solid 3rd party numbers from the new mac studio ultras, which admittedly are a bit over $10k with a 2tb ssd + 256gb ram. I think I'd be seriously considering it if I had a $200/month subscription of some kind.by Octoth0rpe
- This situation has improved quite a bit recently, Qwen Flash Next will run on a $4000 PC and can reliably implement small features on its own (feels comparable to Opus 4.5). It's a bit slow but pretty effective.by hedgehog
- Can a 27b model even do meaningful security tasks?
I thought the interesting cyber stuff is really at the edge of frontier
by Havoc - It depends what you mean by meaningful, but yes, qwen3.8 27b is pretty mind blowing to me. It can easily solve portswigger labs for example at q4_k_m.by prettyblocks
- I had hoped to get some new information out of this topic, but unfortunately found the same local "dead-ends" that I explored myself.
It unfortunately feels like we will be stuck waiting for a burst bubble before local hardware can be reasonably acquired for personal LLM usage.
by dell2024 - FWIW, I’m not dead ended yet.
Using frontier providers to manage prompt tuning experiments has been very exciting. Might be something there.
- From what I'm seeing elsewhere, context size up to 128k should be possible on this hardware. It really matters for agentic workloads to push that context size headroom up. Anthropic are spoiling us with models that do 500k context and beyond.by airspresso
- I've been using Claude Code Extension in VSCode (no phone-home configured), backed by DwarfStar on a LAN local MBPro 128GB M5. The context bloat is horrendous, leading to 5-10 minute prefills.
I've recently been exploring tools like headroom to help manage context, with some limited "success" (for some definition of success). What do others with similar setups do?
(I kind of hate to abandon Claude Code, as it seems to be the most capable coding assistant of the limited set of tools I've tried. But that horrendous context bloat is really painful!)
by fghorow - OMP. Opinionated but completely configurable. Probably the beat to have a lot of batteries and let you uninstall what you don’t want. Sadly Anthropic forbids its use on their subscriptions.by bitexploder
- Why are you using Ollama? Just use llama.cpp
- MLX?by bitexploder
- TLDR: Local models have a smaller context window, so your 35kB prompts that worked fine against a hosted 1 Million token window, crash out when you only have a 65K (!) token window locally.
I dislike being negative, but I was really hoping for more substance when reading this. It would have been an interesting topic.
by SyneRyder - The new DeepSeek models address this issue very cleanly. DeepSeek Flash V4.1 requires less than 1 GB memory for a full 1M context, down from about ~10 GB in DeepSeek Flash V4.0. This is a significant step towards making near-frontier models usable even on low-end consumer hardware, though of course with significant tradeoffs in overall performance.by zozbot234
- Thanks for the feedback. I wanted to get into more detail, but I spent the whole weekend working these problems and then constructing this post.
Dario’s behavior this weekend made me feel like this just needed to get out quick. In the future, I’ll be sharing more details about some other things in the process and some ways I found to use automation to accelerate splitting prompts for use on local inference.
- Funny times!
How many years after "public clouds" and non-local "disks" and "drives" we are ? :) And you still need to tell peoples that other have access to your private data :)
Wait, no... They even have access to a thingie you just about to think about! ;) That is a superpower, no less :>
And managers are firing peoples just to outsource "thinking" to some not owned by them cloud computer :>
by Woodi - > Everyone who begins learning exploitation hits a phase of exploitability grief about 3 month into dedicated, practiced study. They hack something they didn’t think they had the skill to break into and it terrifies them. They’re smart enough to know that, relatively speaking, they are an idiot, and if an idiot can do this then nothing is safe. That feeling is correct.by andai
- 20 years later… I don’t get joy from hacking things. But the 20 year wisdom is a lot of the time it doesn’t matter if it is safe. Just know when it does matter and worry about that :)by bitexploder
- Friends Don't Let Friends Use Ollama https://news.ycombinator.com/item?id=47788385by cube00
- just use unsloth or jan.ai or...by fl0id
- Things are different now on a Mac. Many small improvements make Ollama genuinely decent for many models now.by bitexploder
- and tell your friends about https://llama.app/ by Nvidiaby homarp
- The article doesn't really describe the problem: if your prompt is 35kb, your prompt is confusing, unfocused, and doesn't work right on any LLM, and is needlessly bloating your context.
At this point in time, due to how most people and companies run their inference engine, regardless of the model (yes, this includes the newest from OpenAI and Anthropic and the Chinese Tigers and Dragons), you run out of useful context that the model can accurately attend to around the 250k mark no matter how much they advertise their context size is.
You need to cut your prompt up. If you believe LLMs work, have the LLM help you shape the overall plan, and then have multiple sessions run each step in the plan without being bloated with the context of previous successful steps.
I don't see LLMs being production-ready until the context rot and sampling problem is fixed forever. This has not occurred, and the big inference providers aren't even bothering to integrate any of the research on that subject.
If anything, many of the bigger companies are actively making inference quality worse just to extend their runway a tiny bit farther before they go bankrupt.
The only thing the article gets right is this: if you're serious about LLMs, abandon Big AI and infer locally only. This is the only way you have control over the quality of the output.
by DiabloD3 - > you run out of useful context that the model can accurately attend to around the 250k mark no matter how much they advertise their context size is.
This was certainly true when I first tried the new models with a 1M context. After 200k things got weird pretty fast. I haven’t had that problem since Opus 4.8. I’m regularly bumping against 800k tokens in “lazy” adhoc sessions. “Lazy” in that I ought to do as you suggest, in the way that I ought to refactor this code, I ought to factor out the meat of this session, but in the moment it’s still producing useful output! Tool harness is a force multiplier too: tools that put all tool use in subagents are incredibly frugal with the main chat session.
- The "dumb zone" threshold is fuzzy but comes waay before 250k tokens. Like half that.by chrisweekly
- Attention is all you need.
And there's only so much attention to go around.
by fennecbutt - > "until the context rot and sampling problem is fixed forever"
I agree, prompt adherence seems to get worse when operating on large inputs. Does anyone have some notion of the SOTA with this? Can we expect big improvements by this time next year? (hopefully in open weights)
by hermitShell - The Claude Code system prompt was >50KB, though I think they trimmed it down heavily recently. (The newer models don't need as much hand-holding.)by andai
- > your prompt is confusing, unfocused, and doesn't work right on any LLM
You are assuming the entirety of the prompt is human prose, but it could be sets of data so the agent doesn't have to collect it every time, like program interfaces, commands, views, databases, tables, data models etc...
I could see this scale to multiple kiltobytes of metadata in the prompt easily.
by gchamonlive