MacBook vs. Dedicated GPU for LLM

Join the discussion

• Hacker News
• With a dedicated GPU, the lag is in transferring data to the GPU. You don't have that lag in ARM.

  But it really depends on what it is you want to do. An MLX optimised recent model will run fine and at decent speeds. Granite4.1 (a few months old) for example takes up 2GB of memory, insanely fast and results are good vs much bigger models like gpt-oss-120b (a year old). It even runs on an M1 mac with good speeds.

  The models are only getting better.

  by EagnaIonat
• From personal experience - it works, but you won't get a comfortable time to first token (latency is high). The reason is that prefill on Macs is bad. You need to have a lot more cores to do it quick. It's close to instant for small models on NVidia GPU's but on Macs it takes a few seconds to get the answer for a simple prompt. And the time grows proportionally with your context size.
  by zihotki
• MacBooks have lots of RAM and no PCIe bottleneck, but ~10x fewer FLOP/s than a much cheaper Nvidia GPU. Test LLMs on rented GPUs on vast.ai or other similar services (beware storage etc). Don't spend thousands before trying and knowing exactly what you get.

  Also beware local models tend to be slow. Also, the main optimization trick for LLM inference is running large batches (concurrent users) and you won't take advantage of this (batch=1).

  IMHO using Macs for LLMs is a fad. An expensive fad.

  by alecco
• I wouldn't buy any Mac products for LLM usage. It's cheaper to just buy a subscription. You enjoy Mac for your personal/professional use. I have been running LLM on Mac, honestly for such an expensive machine I wouldn't want to use it for long runs just for a quick 30 minute usage. It heats up so much you can make an omelette on it. I would NOT want my dedicated MacBook run hot for long these things last forever. Fairly confident if I run LLM's all day long their life will be short lived. In my family my kids are still using their laptops which are papas handovers. I have an iMac from 2014 that my kids still use.
  by lifesucks1
• Both are going to be super super slow and low payback.

  You gotta really want it right now.

  It's still early!

  by epsteingpt
• It depends. I have a M5 128 so i can play around with large models and even keep several of them loaded at once and use something like llama swap to access them all via bifrost or litellm. You won't do this without some serious local GPU's with big memory. The downside is the speed, it's not fast, but fast enough to tinker and develop with worrying about ongoing cloud cost. When done developing and you need to really scale up this is when you can swap to cloud computing and get the job done faster. My $5k macbook can do more than a $50k nvidia/intel/amd setup, just not as fast.

  So you need to decide whats important to you if you want to work locally, large/many models or speed. It's the pick 2 problem speed, size, cost, pick 2 or go with cloud and accept your development time is also spent $$ on each iteration.

  I was hoping for the M5 ultra by now, but looks like that's not coming until much later this year for a much higher price now.

  by hkchad
• Around February or march I started looking into hardware options to help me start learning about training models and working with them. My budget was limited and an apple refurbished 32 gb Mac mini was far and away the best option for my budget. I wish it was faster but I can let it run 24/7 with no noise and minimal power draw. I just arrange long running tasks for when am asleep or at work. Then as a huge plus I have an awesome daily driver machine for whatever else I want to do
  by g-technology
• With a M5 16c 48GB and Qwen 3.6 35B Q4 I get up to 1900 PP/s and 80 TG/s. With an Nvidia 5090 I get 7800 PP/s and 280 TG/s.

  Together with pi mono I wouldn't want to go back to Claude & Co. Speed, quality of the answers, short answer times at any time of day - once you have eaten from the fruit your definition of SOTA will change...

  For reference, I do software development since 30 years, I am not vibe coding the umpteenth todo list.

  by dust42
• My opinion is that you should wait for 6-12 months before making a purchase either way.

  Open weight models are getting good. With GLM 5.2 now chasing Opus, I'm very excited to see a smaller model's distillation.

  Plus, the OLED MacBook Pro should be released by then.

  by nichch
• The OLED touchscreen MacBook is rumoured to be called MacBook Ultra now, and it has been delayed quite a few times. I will probably cost the same than a decent bike.
  by speedgoose
• This is my opinion too. Even if you buy hardware like a cluster of 8xGB10s or 4 A100s, they'll still be slow and a little dumber than what you're used to. We need to wait a little for better hardware. Lots of companies are pushing the frontier, so hopefully it'll come very soon.

  Competition and innovation will hopefully make the bubble pop, and we'll get reasonably priced local hardware to run very intelligent models. Something like Talaas with GLM 5.2 would be pretty cool. Or Apple printing the latest model onto hardware—it would give a new reason to buy a new Mac every year (a new ai model with every new version).

  by Frannky
• MacBooks with their unified memory behave like a slow GPU with enormous amount of video RAM. So you can run large smart models slowly.

  Dedicated GPUs have less video RAM so can run smaller less smart models quickly.

  by JSR_FDED
• Unified memory is also a kicker for crashing a Mac.
  by emmettbt
• how much memory is actually useable by gpu in macbook? as it is shared?
  by mzubairtahir
• > MacBooks with their unified memory behave like a slow GPU with enormous amount of video RAM. So you can run large smart models slowly.

  With the model using MLX the speed increase is night and day. Even non-MLX is good.

  You also don't have the transfer costs related to moving CPU data into the GPU.

  by EagnaIonat
• Macbook M5 64GB - can run gemma-4-26b-a4b-it-4bit and Qwen3.6-35B-A3B-4bit at about 1500 tps prefix and 45 tps decode on contexts up to 100K tokens using MLX. It's faster than Claude. I was really surprised, chat quality is also similar to Claude for gemma4. Agentic works but does not compare to cloud models, you can still make agents where top level is code.
  by visarga
• Do Mac Pros provide more headroom? noob here, noob questions
  by exabrial
• If you want a massive MacBook anyway then it's great. They are decent for local LLMs, awesome for local image models and it's a MacBook so AppleCare+ has your back. IMO it's a no brainer if you wanted a MacBook anyway but it's a poor choice if your reason to buy it is to run LLMs.
  by jpgvm
• are you saying because of speed or it just cant run them?
  by mzubairtahir
• I agree. To run an acceptable model (e.g. Qwen/Qwen3.6-27B or google/gemma-4-31B) with a good quantization (minimum Q5) with a good context size (min 64k) you could buy 2 or even 3 GTX 5060 16GiB VRAM for ~550$ each. Fyi the much faster MoE models were useless for my usecases - e.g not able to correctly identify me/I/you, endless thinking loops, etc.

  I'm currently running those models using an RTX 5070 12GiB + RTX 5060 16GiB + RTX 3060 12GiB with a 96k context size with MTP/speculative decoding and I'm quite happy (the 5070 is about 4x faster than the 3060, the 5060 is inbetween them so about 2x faster than a 3060).

  by zepearl
• I asked a few of my friends that are ML engineers this question and all of them said to run the LLMs in the cloud with their infrastructure because it was going to be way faster. If you just want to tinker around I would look at @JSR_FDD's comment.
  by cylentwolf
• There are more factors than speed though, like: privacy
  by derwiki
• Next you're going to tell me that car salesman recommend just leasing new cars, doctors recommend just following the standard of care, construction workers just recommend subbing it out, and your tech friends say just use AWS.
  by throwawaytea
• My dad avoided buying a home PC for a long time, because he felt the systems he used at work in the 80s were so much more powerful than anything for the home at the time, that he didn’t see a point.

  I can’t help but draw parallels here.

  by al_borland