Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Why does the screenshot on your pi terminal shows opus-4.6-medium from your claude subscription ? Instead of Qwen ?
- Ah, my bad! This image came from our backend, used for an unrelated article. I selected it by mistake rather than inserting the actual image that I'd uploaded. I'm updating it, thanks for the heads up!
For what it's worth, that image couldn't have been related. The other screenshots all showed thinking traces, and Claude doesn't share those.
by AdamConwayIE - I'd personally like to know more about what tools it used/wanted and the harness setup, because this sounds pretty cool. I have a dual Arc Pro B70 setup and currently get around 22 t/s which isn't great but isn't terrible either (it is at least less quantized.)
I've seen GPT 5.6 Sol happily invoke objdump and even write jobs to run headlessly which Ghidra when trying to disassemble a binary.
by jchw - I added a line to address this, sorry it wasn't there before! It was Pi and only used Bash-based tools.by AdamConwayIE
- My M5 Pro gets around 12-15 (6 bit MTP), although I haven’t worked on optimising it at all yet.
A nice thing about running locally is you can run an uncensored model and you don’t have to worry about TOS violations on your OpenAI account when you ask it to “reverse engineer this ancient router firmware and give me a licence key that will work on it”.
by trollbridge - I can't get Qwen 3.8 27B to do a simple code review on a fairly basic Python file. With thinking on it just ruminates forever and with thinking off it gives obviously bad borderline hallucinating advice.
Edit: I tried again with the 2.4T model and it still ruminates to death, but with thinking turned off, it generated genuinely useful advice.
Edit2: adding
to the llama.cpp executable parameters produces pretty good output.--reasoning-budget 8000 --reasoning-budget-message "Reasoning budget exhausted; give the final answer now." --reasoning-effort low"by jnwatson - how are you running the 2.4t model locally if you don't mind me askingby nialv7
- One of the big learnings from 3.8 27b is adding reasoning budget really hurts the model. you need to let it spin for as many thinking tokens as it wants to to get it out. Another big takeaway is reasoning effort set to low doesn't save you tokens: low is pretty uncertain about things so it ends up thinking more (you can find some tests from folks on youtube). The final question, as always, is what quant are you running it at? KLD matters _a lot_ when it comes to its performance and it especially manifests with MTP/DFlash acceptance rate which makes those long thinking traces take a long time.by Refefer
- I have this idea of using an obliterated version of this model for cyber work(or even this one, seeing that its guardrails aren't that strong) in a harness with the ability to spawn SOTA level subagents, faster and more capable.
The rationale is that the manager model sees the big picture and knows that the task is "unethical" while sota models are just given very isolated technical tasks that don't trigger any refusals.
Has anyone tried this? I would love to know about previous attempts of this approach.
by samuel - Yeah, the Chinese government used the same method last year to hack the US government using Claude Code.
Making each piece of work small enough to be plausible. Compartmentalization.
(Also saying "nah it's cool I have permission", heh)
by andai - In my benchmark Deepseek-v4-flash did much better than Qwen 3.8 27B at reverse engineering.by __alexander
- > This project evaluates local language models running on a single NVIDIA DGX Spark.
"Did much better" is a bit misleading w/o that context and 1 hour time limit -- your benchmark design heavily favors V4 Flash. From results on your page V4 Flash processed 1-1.5M tokens an hour, while Q3.8 27B was failed before even reaching 200K tokens.
By the way, how are you running V4 Flash on single Spark? Was it quantized?
by petu - As someone who was selling Windows desktop app for 10 years and made nice money out of it I have mixed feelings. On one hand it was always a losing fight against determined hackers on the other the tools weren't widely available so the problem wasn't as widespread. We lost quite a bit to piracy but could still make a decent business. With widely available LLMs I think that business model is truly dead though. Not only hacks/cracks but also any kind of smart idea you may have will quickly be reversed engineered from your binary.
If you never lost money to piracy you may think that "those people are not your potential customers anyway". This is not true because people will crack your software and then resell it - often pretending to be legit resellers operating under your brand. To add insult to injury they will send their customers to your support as well.
If I ever come out with something smart again there is no way I am shipping it as executable. SaaS it is for better or worse.
by bluecalm - Lately I genuinely believe that the future will be large frontier models generating and updating inputs/skills for "good enough" local models to solve our daily problems.
A lot of tasks which need a bit of intelligence don't really need that much compute. Just good enough documentation / skills, tool calling and a good enough local model.
Not sure what exactly this means for all those data centers that are getting built... But exciting times.
by saidinesh5 - What’s the fundamental difference between a frontier model and a local model anyway?
- Yes! My main use of very strong models is in writing my own coding harnesses for small local models, tailored for my needs. I also use very strong models to get much smaller skill files and also writing tools for my harnesses.
re: data centers: pump and dump. Wealthy investors will have made their money and walked away, and the corrupt democrat and republican politicians in Washington will, as usual, protect the interests of the ultra wealthy and leave the general public to pay for poor decisions. There will be a government bailout.
Anyway, on a positive note, I am all in for small local models that are augmented by strong hosted models for specific tasks. Use technology to help people, not make billionaires even more money.
- The data centers are for inference time serving, and as long as the GPUs go better, it shouldn't really matter. Token prices should go down 10-100x over the coming years, and then we will use more, because they have more capacity for the same hardwareby verdverm
- With AI being more useful with access to more of your data, I can't see myself using cloud AI models for purposes such as personal assistants.
Perhaps with differential privacy or confidential compute...
But ideally these models run locally.
by Tepix - i'm not good with paper work, in fact, i'm horrible with anything that's paperwork related. for the past few days, i ran this model on my rtx 4090 + rtx 3070 and told it to check all the bills, invoices, contracts for me and my small company. i used pi with llama and the pi-llama plugin. oh, boy - i hooked it to my email, told it to download all of the invoices and bills i had for both me and my company and organize them by company/date/ and then merge them with the ones i have locally. it did ocr, wrote scripts, organized everything neatly. i am now the most organized i've ever been in my life. Next: RAG on all the documents and bills i have. if you connect staan-search (there is a pi plugin for that) and ctx7 to this it almost does miracles. the downside is i have to sit next to my noisy threadripper as the magic happens and pay for the electricity, but that's about it, i'll gladly do that. and as i finished this paragraph, it also finished organizing all my personal documents on my san. i don't use the expression "game changer" easily, but it's hard to resist in this case. out of all the models i've used locally qwen3.8:27b blows everything out of the water.
my setup
# Logical CUDA0 = RTX 4090, logical CUDA1 = RTX 3070 export CUDA_VISIBLE_DEVICES=0,1
cd ~/projects/misc/llama.cpp/
exec ./build/bin/llama-server -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M --mmproj /xx/xx/xx/xx/xx/mmproj-Qwen3.8-27B-Q8_0.gguf --host 0.0.0.0 --port 8080 --jinja --parallel 1 --split-mode layer --tensor-split 6,1 --fit on -fa on -c 98304 -ctk q8_0 -ctv q8_0 --image-min-tokens 1024
i load more on the 4090 because it's faster.
usually the temp stays around 65 for both. utilization for 4090: 70-90% 3070: 30-50%. I get around 30-40 tk/s. if i offload more to the 4090 the tk/s goes up, but i stress the card too much and that thing now is worth its weight in gold.
note: the pi-llama plugin needs a patch for pi to send the model vision capabilities, seems it doesn't work out of the box.
by pi-victor - I hope you have backups.by Tepix
- Are you worried about the temps on the 4090 or just pegging the cores? I've found undervolting very effective at controlling temps with small performance loss. It was also easier than expected.
Oh and try MTP if you haven't already, massive performance boost
by apitman - Pro tip: usually you can download your emails to an offline collection. That way there's no risk. Hooking it up to your email sounds very very risky.by Neywiny
- Local models would be even better if they did not ship with all the refusal shenanigans built-in. You can safely bet organized crime has access to the best models without these hoops, which makes the case that the average user (=non-criminal) should have access too. As I understood from an ex-Anthropic employee, some orgs got access to Mythos based on their high enough spending level, not on other grounds.
Either we are in command over the software, or the corp is in command over us via the software. I can on a theoretical level understand the concerns, but either we ban all LLMs or we have a level playing field for everybody. Let's not forget: defense and offense are different sides of the same coin in software. I guess this wouldn't apply to bio weapons, but I am not in the know about that.
by exceptione - Ehh, it’s at least given as the excuse for gain-of-function bioweapon researchby binary132
- heretics and manual iterations get you very far to the point where i have ethical questions about whether this should be possibleby ramon156
- > with all the refusal shenanigans
Given the faults in simulated Intelligence that LLMs have, and a comparatively low level - which means, lower judgement abilities - to the best of us, there is a strident match having such employee judge the intentions of the employer.
Limiting the responses makes much more sense on cloud-based systems (you are using our infrastructure etc.).
by mdp2021 - Hugging face is filled with uncensored versions of your favorite local models, so in a way they are shipped without the refusal stuff, via the magic of fine tuning or however they get this stuff out of models.by seanmcdirmid
- Completely coincidentally, we're just about to launch a service that does exactly this (API access to uncensored open models)! We have a waitlist at the moment but will be live very soon!by TofuLover
- There are versions of Qwen3.8-27B that are unrestricted and available from hugging face.
"It will comply with harmful, unethical, offensive, or illegal requests that the original Qwen3.8-27B would refuse. It has no meaningful built-in guardrails."
by dantudor - I’d expect these shenanigans to get much worse over time for the average Joe.
Imagine a world where any random person can run a super-capable model on their own hardware with no limitations and no one to pull the plug.
Information has always been power and those who already have power won't just allow everyone else having the same tools as them
by ninahaberl