Birbla

Discussion summary

The discussion centers on the high cost of RTX PRO 6000 GPUs, with prices around £4000-£13,000 due to demand. Participants mention running models like GLM 5.2 and Qwen, and compare local SOTA capabilities to large-scale deployments by OpenAI.

What the discussion says

  • High GPU prices driven by demand and supply issues.
  • Running SOTA models locally is feasible with high-end hardware.
  • Large companies spend much more to serve millions of users.
  • Some models like GLM 5.2 are considered SOTA within their size category.
“Because if they were any lower we’d all be buying six each.”
— gizajob
“If you can run sota on a 40k setup, why do openai etc spend maybe 100x that?”
— dwroberts

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • You can get amazing local STT using parakeet which can use as little as 600mb of vram. Better or as good as whisper v3 large
    by zackify
  • Have we reached the capability of a local STT+LLM system being constantly listening for normal speech in a room and being able to understand when the human is addressing the system instead of talking to another human?
    by mrgaro
  • Is Whisper still considered SOTA for STT? Since it came out years ago, I'd have assumed there are better models by now.
    by chompychop
  • I don't have anything to compare against, since I have just started using it. But I was fairly happy with it on my personal recordings from my phone. Also, I ran it on my CPU (Core i7) and it was perfectly usable, as something to run when not using the machine for anything else.
    by venusenvy47
  • I'm a big fan of Parakeet v3 - I run it using the MacWhisper app, it's a 494MB model and the quality is excellent.
    by simonw
  • No, there are quite a few models which are smaller, more accurate, and faster. For example Parakeet TDT v3 is half the size, way faster, and lower WER. There's also Voxstral, which is much larger but also even more accurate.

    But the ecosystem isn't as mature, so Whisper is still a valid option, even now. For example Parakeet uses Nemotron framework (made by Nvdia), normally you need CUDA, so you need to use an ONNX version instead on AMD. Meanwhile Whisper has VLLM and desktop apps like Buzz.

    There aren't many benchmarks and they often don't have all the models, since STT doesn't get nearly enough attention as normal LLMs, but this is one of the more complete ones: https://artificialanalysis.ai/speech-to-text/non-streaming

    by randomblock1
  • I am somewhere in the middle, where I want something with more than 48GB/$2k of VRAM, but less than 384GB/$40k.

    I'm curious if GMKtec's EVO-X2, with ~96GB of usable VRAM, is still a good solution for something like this for $3,399.

    by beardsciences
  • It has lower memory bandwidth than most comparable Macs.
    by mft_
  • I've been happy with an OEM Spark (128G), enough so that I picked up a second one. Have 2x qwen and 1x gemma (both at 8bit and full context), plus embedding, Re-Ranker, and a 1.7B for little things. Running 6x models, probably going to add STT here soon, want to try talking more than typing.

    The caveat is that if you try to use multiple models on the same device at the same time, you thrash and destroy tok/s

    by verdverm
  • I picked up the 128gb version when it was $2,199 and it runs Qwen 3.6 reasonably well with a 128kb context. Not very useful for complex tasks but it can handle some web stuff.
    by sampullman
  • Related - what is the best isolation system available? Do I have to go full, fat VMs or can I get by with a Firecracker-like thing?

    Seemingly every available option has some subtle-gotchas about how easy it is to blow off your foot and effectively have no security at all. I use VMs because I actually trust that security is a foundational principle of the technology, not a well-if-you-use-these-20-flags-and-squint kind of deal.

    by 3eb7988a1663
  • Full fat VMs with GPU passthough I trust a lot less then CPU ones.
    by ZiiS
  • im not sure that there is a plug n play set up that will work for everyone, because as with any security boundary, each layer of hardening has a usability trade-off. i definitely feel you about the uncertainty of it all, how do you actually know everything is tight?

    personally, i think either a VM or microVM is the way to go. these things are actually designed as security boundaries, as opposed to containers. and as compared to bubblewrap, you can just give the agent a whole FS to work with and run it in yolo mode, whereas with bubblewrap you have to manually bootstrap the availability of each individual dev tool and make sure its config dirs and package caches and etc are mounted in a secure way and still will probably hit perm errors all the time. and there's just way less isolation.

    also, something that has limited support in harnesses but IMO would make a lot of sense is running the harness process in the host, but having all the tool calls and file system interactions delegated to the VM. that way you keep all your session data and auth keys on the main machine where it can never get into context. otoh it makes your harness part of the security boundary, so that's the trade-off.

    there's also all the usability questions around how to actually get data in/out of the VM. i have a script which can push local git repos into the VM and then pull from them as a remote, so the VM can't initiate any connection with the host doesn't need to hold git credentials. but ig for someone who wants their agent to push straight to GitHub that's a waste of effort.

    options i've tried or seen for the VM itself: - qemu + libvirt: takes some doing to wrangle it together, but very battle tested and configurable - crun-vm is a PoC of higher level integration layer between podman and qemu, which is a really cool way to go about it. seems maybe abandoned but i just think it's neat and very existing tools/standards oriented rather than starting a new project and brand so i mention - libkrun is a newer entrant, and several ppl have built wrappers around it: - microsandbox - smolvm (posted/discussed on here recently) - krunvm

    this is all Linux oriented, it's all i know.

    by igorbark
  • It depends - for what? If your security model is sandboxing an agent to ensure they don't nuke your PC, then there are a lot of options, you can use something like bubblewrap[1] or a microVM like libkrun[2] if your goal is light-weight, up to full Docker if you want the tooling that comes with that.

    [1] https://github.com/containers/bubblewrap

    [2] https://github.com/libkrun/libkrun

    by Catloafdev
  • On MacOS you have a seatbelt sandbox built-in. On Linux - docker with SELinux or similar utility over namespaces.

    You need to model attack vector first.

    `rm -rf` - restricted write

    `curl malware.sh | sh` restrict execution from writeable dirs (seatbelt/SELinux)

    Restricted write to sensitive directories would most likely neuter most malware.

    Credentials leak - cleanup environment, deny reading .ssh, .aws, other, and don't allow LLMs anywhere near production systems.

    I made a small utility for MacOS https://github.com/aka-rider/leash

    But it may be as well a bash script

    by aka-rider
  • For qwen3.6-27b you can also run the q4 variant with full ~250K context on one 3090. It's fast enough to not be frustrating so the speed gains with 2x 3090s wouldn't be worth it to me. Running a q6 on 2x 3090s at half the speed with a smaller context is an option, but you're really not going to compete with SOTA models there anyway so unless you already have 2x 3090s, I would say 1 is the best investment given current prices. It's good enough to do a lot, especially with a well-configured harness.
    by turova
  • That math (250k context, Q4 model, 24GB VRAM) only checks out at q4 quant for the K/V cache, which is probably not the best idea.
    by hypfer
  • Are you running qwen3.6-27b on one 3090 with your KV cache at q4? Ime there is significant long-context recall accuracy degradation at that precision. I prefer putting the KV cache at q8 and working with the 120k context
    by nabakin
  • Might as well add my own experience since I just set up a local llm this week. I went with a 32GB card made by Intel called Arc B70, which is cheaper than a 3090 and more has ram, at the cost of a slower memory bus. edited to remove something incorrect, thanks diablod3

    I went with this because a) the models I wanted to use are a little too big to fit comfortably in 24gb, plus I need room for a few additional small models for autocomplete and speech recognition, and b) I already had a cheap server to use and dual gpus would've required upgrading the mobo and power supply and probably the case as well.

    It was definitely a little tricky to set up. The Intel line requires a driver package called "level zero" to support something called SYCL (Intel's version of CUDA basically, AFAICT) that was tricky to get working. I am running llama.cpp in a docker container, which also required some fiddling to get the container to see the card. You also need a kernel from the last few months.

    Once I got it working though, the results are very impressive for a $1k investment. Qwen 3.6 35B at q4 quantization takes about 3/4 of the ram and delivers like 88 tokens/sec. So, if you want a decent-sized model for cheap, this is one way to go.

    by ineptech
  • That is incorrect.

    They both have GDDR6.

    The B70 has 256 bit it bus at a clock speed of 2375mhz (608 GB/s), the 3090 has a 384 bit bus at a clock speed of 2438mhz (936 GB/s).

    It isn't slower, it just has less channels, ie, it is less wide.

    by DiabloD3
  • Apple M series chips deserve a mention as another option, especially since you get a whole Mac laptop or desktop workstation too.

    They have unified memory and respectable inference performance, and for some variations can be cheaper than video cards, especially if you get an older-gen high-end M series with a lot of RAM used or refurbished.

    I've read that Apple has plans once the RAM bottleneck passes to offer more RAM in all their models, and that future M series GPUs and NPUs will be even better for local inference, so in the future I expect Apple to be a serious offering for local inference and AI research workstations.

    And what about AMD and Intel Arc GPUs? They don't get as much love but I've heard they can be compelling for certain shapes of a local LLM configuration.

    At this point though, I think we may be in a "renters market" for LLM compute. If you want privacy it might be better to rent GPU time in raw form or use spot pricing at various providers. It probably only makes sense to build if you have extreme privacy/security needs or just want to do it cause it's cool.

    by api
  • MLX is super underrated right now, tons of performance unlocked as of recent. Love to see it!
    by maxxxml
  • > once the RAM bottleneck passes

    Do we have evidence that this will actually happen? Maybe the belief that it won't pass is what requires evidence, but I think there's a widespread feeling right now that things are just getting permanently worse and this is one example.

    by mwcampbell
  • There also exists an in-between possibility, that is, if you get 128GB of vram (there are now multiple options in the market to get that amount with a unified memory architecture) you can run DeepSeek V4 flash at good speed via DwarfStar. I'm not going to spend money on this, but my gut feeling is that this would be the right compromise for a lot of people.
    by GTP
  • I just started using it on an m4 max 128 and it's the first time since buying the machine a year ago that it feels like local llm "just works" for reasonably decent coding.

    Use pi though; claude code has way too much bootstrap context; slows everything way down.

    by jonaustin
  • "A great way to go is 2x RTX 3090s for a total of 48GB VRAM total. You can then run Qwen3.6-27B, which is an awesome model."

    Just want to note that for $3k you can get an M5 macbook pro with 48gb of shared memory, and it will not be a giant box. Also, consider committing to spending that money on a cloud hosting provider, which will be at least somewhat cheaper if not significantly cheaper. It is awesome being able to run models locally though.

    by datadrivenangel
  • To summarize a video I saw recently [0] rebutting your arguments, MacBooks can get crazy slow when running local models or even just Claude Code and Codex due to their poor implementation, to the point that the laptop itself becomes unusable.

    There are other arguments for running an ssh-able box in a closet somewhere too as with KVMs you can give an agent remote control over the machine itself such that it has vastly more capabilities than if it were controlling its own machine it's running on, as well as not needing to keep the MacBook open all the time just to have the agent finish running.

    [0] https://youtu.be/9tGrhrVKCrE

    by satvikpendem
  • The standalone mini/studio is better if you dont want to have a constantly hot laptop

    Get a regular laptop and use the network to access the LLM

    by boredatoms
  • You can also buy a Jetson Orin with 64GB of unified memory.
    by amelius
  • That's a reasonable option, just be aware that you get about 1/3 as much memory bandwidth with the M5 Pro, or 2/3 with the M5 Max [now you're at $4100 for the lowest-end]. So both your prefill (flops-bound, M5 has a lot less) and decode (bw-bound) will be slower.
    by jbellis