Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Somewhat off topic, but I've started wondering if we can actually make local LLMs feel smarter than the frontier closed models, by post-training it for your specific use case.
Say Company X has a software product which consists of a million lines of code, including a ticket for every bug and new feature for this piece of software. Then wouldn't it make sense to try using an open weight model but post-train it on that specific code base while using the tickets to teach the model about past bugs and features. Not sure exactly how, but it could involve doing reinforcement learning solving a past bug on that historic version of the code base, and rewarding the model if it comes up with the correct solution (as defined by the linked PR which fixed the bug).
So essentially post-training your local open-weight model using reinforcement Learning with Verifiable Rewards (RLVR) on your software products history of bug reports and their ultimate solution. And the same for new features.
by runeks - Instead, I'd build tooling for the model to be able to query the tickets, pr, commits, diff, etc
This is something I can reuse better.
by rovr138 - I wonder the same thing. Code is pretty open ended though so I wonder if it’s not the best example.
On the one hand it would definitely be useful for something like classifying support requests into priority. But would it be worth it to just train your own model? I guess one advantage is you could give it some well informed guidelines without training something on lots of data.
There is probably a better example between discrete labeling and code though.
by nonethewiser - You could do this for sure but its quite a lot of effort and a normal software stack is quite well known due to the massive amount of github projects and other opensource code.
You can also get a lot out of a harness in this case or your Agents.md or Claude.md file by just enhancing the context.
You might even question yourself if you are doing something wrong if a modern LLM really struggles with your code.
We do the finetuning only on small semantic data were it helps a lot.
I'm still wondering when we see smaller models (faster and cheaper) for more specific stacks like spring boot + java + angular + english only or so. Interstingly enough, i assumed LLMs are really good in any language but it seems that non english languages do reduce the ooutput quality of an LLM. At least last years GTC there was a talk about it.
There are companies though which ahve this exact problem with programming languages you normally don't see. ABAP for example is a very well known SAP language.
by Zylokloto - > And the comparisons in this post are not going to be running some 2.58-bit-gguf-in-ollama with a couple test prompts.
Genuine question : is there something fundamentally wrong with Ollama ?
I use Ollama because it is easy to set up and manage (and also because VLLM is not super Windows friendly).
I thought the main advantage of VLLM was better concurrency management (better batching).
But if the quality of the interference itself is an issue, then maybe I should reconsider my choice.
by JacobJack - In my limited testing of ollama (in a Docker container because I don't want Meta touching bare metal disk), I noticed that first of all the model is abandoned. Last updated in 2023. Secondly, it basically can't do anything. Anything I asked of it resulted in a lot of "I can't do that" so it's just stupid.
- Thank you for your responses. They've been very helpful. I'm going to test the llama.cpp binaries directly. That seems to me to be the closest to what Ollama is trying to achieve, after all.by JacobJack
- Maybe some were fixed, but:
1) Shipping with 2k default context window for the longest time, w/o any warning and being not easy to change (like any other setting). Totally made a lot of people think local LLMs are dumb as rocks. Just checked, still not fixed -- defaults to 4K if less than 24GB VRAM: https://docs.ollama.com/context-length
2) Registry mistrust due to Deepseek R1 naming. What model do you download/start with `ollama run deepseek-r1`? Not Deepseek R1, but this "for research purposes" thinking finetune of Llama 3 released alongside R1 paper: https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama... . For actual R1 you have to pull `deepseek-r1:671b`
3) Can't reuse existing ggufs you have. If you point it to gguf, it would create a copy of it in ollama folder.
4) Doesn't accept engine parameters from CLI args. Only env variables or modelfile. But some things are only in env variables (e.g. KV cache quantization). And even then not even close to what's available in llama.cpp.
5) Often meaningfully slower than llama.cpp
6) VC-funded wrapper for llama.cpp. For long time had questionable attribution to it. Now looks good.
> (and also because VLLM is not super Windows friendly)
llama.cpp is Windows friendly and fairly easy. Not GUI app with installer, but easy. Zip with binaries are on github, run
will pull the model you expect from hugging face (or -m for manually downloaded .gguf) and start web server with API and HTML chat app.llama-server.exe -hf unsloth/gemma-4-12B-it-qat-GGUF:UD-Q4_K_XLby petu - > is there something fundamentally wrong with Ollama ?by cube00
- I know nothing about "vLLM" -> anything better compared to "llama.cpp"?
I started with "Ollama" (precompiled version) and it worked and was good enough to understand the very basics.
Then I downloaded the sourcecode of "llama.cpp", compiled it with specific compilation options for my GPUs (CUDA/nVidia using proprietary module on Gentoo Linux) & CPU (AMD), and the same model ran twice as fast -> since then I stuck with "llama.cpp" (and "ik_llama.cpp" in very few cases).
I honestly don't know what made "Ollama" (precompiled) so much slower than "llama.cpp" (compiled locally) at that time and I'm too lazy to doublecheck now, in any case I now absolutely love all the knobs that "llama.cpp" has to tune your hardware setup & your workload, which is the reason why I recommend it.
by zepearl - From what I've heard, Ollama has a bad reputation because it's a thin wrapper around llama.cpp without attributing it properly, thereby stealing recognition from the maintainers doing most of the workby kangalioo
- People who use Ollama generally (not everyone obviously) don't always clearly understand what quantization they use when running models, so people end up saying "I tried running Qwen 3.8 27b locally and it was dumb" while Ollama would default to a Q4 version of the model, which has very different results from the BF16 weights, doesn't really speak to the model itself because it's been so quantized in that case.
Sure, makes things easier, but tons of people misunderstand what they're using, then base and share their experiences on that, without really specifying what exact weights they use too.
For a single local user, using llama.cpp directly shouldn't be a problem if you're already using Ollama's CLI, it works basically the same except you manage weights yourself, and if you put your favorite agent to make sense of the faux "registry + image layers" Ollama has prepared locally for you, you can reuse the files you've already downloaded with Ollama.
- > I will make you read the really long unpleasant version with math.
This is the version I want to read :)
I assume it is unpleasant in spite of the math, not because of it?
- I thought it was a link too because of the line under the with math but it isn’t. :/by a1o
- It is good that someone is having such a deep look. This is not exclusive to LLMs in the least. Every non-trivial program depends on hundreds of little details being correct.
That is also why I recommend including health checks in such programs. Some function that checks that all assumptions hold. That could be an endpoint, an automated test, a periodic diagnostic job, etc...
by ThouYS - At least I'd be in control of model quality vs. when Anthropic decides to randomly drop the quality of their offeringby nullpoint420
- Comments are mostly showing off M5s and 5090s without addressing the article.by utopiah
- I’m running Qwen3.8 aggressive uncensored Q4_K_P on a 4090 in a loop against the 2026 CrackMe CTF challenges.
Using oh-my-pi in a prebuilt environment that I let Qwen build too.
Codex wouldn’t even look at the files - literally, as soon as it read something with CTF it shut down. Didn’t even offer to fall back to a dumber model.
- This is similar to my use case as well. I’ll use a cloud model to identify security issues in a codebase and then use a report/spec output as input to the local model to build tests or verify the issue is real.by treesknees
- How's it performing on the challenges?by CamperBob2
- Much of this is why I stick to the rule of:
a) Don't quantize your KV cache
b) Don't run quantizations of the LLM that are worse than the best available Q8 (the largest possible file size unsloth GGUF for a given model like qwen 3.8 27B as an example). I would rather things go slowly but I have confidence that it's doing things more accurately.
by walrus01 - Even a 4-bit quant of Qwen3.8 27b is indistinguishable from Gemini 3.7 flash in our internal tests. With an RTX5090 card and ninfer, you can get ~800 TPS token generation (c=8) and ~140 Tokens per second single stream.by a11r
- I saw some guy streaming how he was deploying qwen3.8 37B on his local setup. Well, he was asking Claude to do it. It took him two hours of passing errors to Claude for the endpoint to start working, he then started testing it against DS4 Flash when Qwen had thinking disabled and Claude messed up sampling parameters, it was an absolute pain to watchby big-chungus4
- > It took him two hours of passing errors to Claude for the endpoint to start working
What? It's literally three actions and you're good: download llama.cpp, download the model on Huggingface, and run it with.
I have no idea how it's supposed to take two hours (unless you have a slow connection and the model download takes this much time, that is).
by stymaar - That's funny, because I just went through the opposite.
I got llama.cpp working with qwen3.6 and qwen3.8 by Googling and manually adjusting things according to reddit posts and Google not-really-helpful AI suggestions.
I tried settings up per-model stuff in settings.json, but again Google got in my way, and llama.cpp having 2 different settings.json (and Google lying about where 1 goes) made it far too difficulty to figure out. I spent hours on it.
Then I got fed up and asked Claude.
Immediately, it told me that the winget version of llama.cpp is for Vulkan, and I needed a different one and pointed at it. It doubled my speed.
Then it figured out what I was doing wrong with settings.json (wrong spot, global settings can't go in the per-model file, etc etc) and fixed all that, and got it working.
Then it tuned it somewhat.
Then I showed it the official settings pages for both models, and it undid the tuning and all the damage I had done with my tinkering, and got everything working.
In 30 minutes.
It was absolutely amazing.
Every time I see people recommending Qwen locally with llama.cpp, they just say "download it" and act like anyone that can't get it running is an idiot. But if there's a "using this settings.json" tutorial somewhere, I didn't find it, and neither did Google over a week of searching.
But Claude got it done for me.
Now, I admit, I haven't played with it much. Just before all this, I ran out of Claude on the $20 plan and bumped up to $100, and It's been so amazing that it's really hard to work on the local. Especially since it feels like Qwen3.8 35b a3b is probably around the corner, and why mess with 3.6 when 3.8 will probably release soon?
by wccrawford - Wow. I tried to get Qwen3.8 4B to parse song lyrics and analyse them. Getting ollama running was a minute or two.
However coming up with a prompt that didn't turn out total garbage was impossible. After wasting over an hour and I ended up getting Qwen side by side with Llama 3.2 3B, just to see if I was being stupid. Nope, it just looks like Llama is orders of magnitude better at this specific task for some reason).
If you think I'm doing it wrong, you're probably right, I don't know a ton about local LLMs. But I hand selected 50 songs, set up ollama with both LLMs, and for each iteration on the prompt text, ran both LLMs 10x times per song. Side-by-side comparisons showed that Qwen 3 4B was so bad that I actually downloaded Qwen again, thinking there must have been some mistake and I accidentally grabbed an old 1B model.
by raffraffraff - I just got qwen 3.8 27b mlx running on my Macbook Pro and honestly I’m pretty blown away by how not-dumb it is.by jonplackett
- How much RAM? And what do you use it for if I might ask?by hosteur
- We were trying running a local gpt-oss 80GB model on a H100, and honestly I was surprised how dumb it was.by rurban
- How many tok/s are you getting? What gen mbp?by alexchantavy