Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- I really hope with Ternus taking the helm Apple starts to remember that it has products outside of iPhone.by paxys
- So for people who don't understand, there are two markets for Apple hardware in this space:
1. Running an agent like OpenClaude. The $599 Mac Mini was an insanely good deal for this. I happened to buy a M5 Pro Mac Mini for $999 last year for other reasons. The equivalent is now almost $2000; and
2. Hardware for running inference on local models. This to me is the far more interesting market because Apple has a real opportunity to disrupt NVidia's stranglehold on the market.
With current architecture, the largest model you can reasonbly run is the amount of memory on the GPU and is a function of the quantization (eg int4, int8, fp8, fp16, etc) available and the number of parameters. NVidia aggressively segments the market. The most VRAM on a "consumer" card is 32GB on the 5090, which allows you to run ~31B parameter models.
In comparison, the RTX 6000 Pro has only slightly more CUDA units than a 5090 but has 80GB of VRAM. A few months ago they were $10-11k. Now they're ~$16k.
Macs use a shared memory architecture. Apple has previously sold Mac Studios with up to 512GB of RAM. Almost all of that memory can be used to hold much larger models without taking a penalty for interconnections between different GPUs or machines. Plus Apple interconnects between computers are actually relatively good by chaining TB5. It's still slow but it's about the best non-enterprise option available.
But the previous Mac Studios just didn't have the raw FLOPS and memory bandwidth. The M5 Ultras are up to 1.2TB/s of memory bandwidth. M3 Ultra had ~900GB/s. RTX 5090s and RTX 6000 Pros are 1.8TB/s. The current best HBM3 NVidia DC GPUs are at 3.2TB/s IIRC. But the M5 Ultra has a claimed ~4.5x the FLOPS of the M3 Ultra.
We don't have our hands on these yet but it probably means they are going to be much closer to a 5090. I expect ~50% of a 5090's inference speed. That may sound bad but it's actually really good because a 256/512GB Mac Studio can probably locally run the best Flash models. With NVidia hardware you'll need to spend many tens of thousands for that.
We'll see what the inference speed is but I expect it to be usable. DeepSeek v4 Flash, for example, will be entirely runnable. We're not at DeepSeek v4 Pro local yet.
by jmyeet - Mac Mini's were really nice HTPC candidates, too, before the AI boom. Like all things genuinely useful and affordable, they were snatched from the hands of normal consumers by a bunch of schmucks chasing the latest gold rush.
- It's fun to see that even an extremely large company can find unexpected product market fit [0]. Per this article, "The company reportedly did not possess an engineering team dedicated to business customers or staff focused on developer relations, and lacked an enterprise AI strategy." That sounds insane in retrospect, but I think there's just inherent uncertainty in what people actually need and will use things for.
[0]https://pmarchive.com/guide_to_startups_part4.html: "In a great market—a market with lots of real potential customers—the market pulls product out of the startup... The product doesn’t need to be great; it just has to basically work."
by setgree - There is a lot of "AI demand" that isn't just running inference on an LLM whose weights you downloaded.
I'm training a model using reinforcement learning with self-play. I can and do use vast.ai when scaling but for experiments it's far faster, and cheaper, to run it locally until the bugs are all figured out. Just provisioning a new instance and copying the relevant checkpoints and things can take 25 minutes. It's zero locally.
by alasdair_ - Apple was also allegedly "caught off guard" by the Macbook Neo demand.
I don't really see how they couldn't see the Local AI demand or demand for a cheaper Macbooks. This just reads like marketing imo.
by HDBaseT - I’m curious to know if these local AI setups are legitimately useful compared to cloud. I’ve struggled a lot to get something useful out of the hardware I have.
I realize I’m somewhat limited (16GB RX 9070), but still, it seems really far off from the kind of experience even a basic $20/month subscription gets me.
Any tips anyone might have are appreciated! I’d love to be local first and would be willing to buy hardware to get there.
by Grombobulous - I'm convinced that this is just guerilla marketing from Apple. When this started spreading a day or two ago, it was all from no-name spam media sites that are paid to publish articles. They all claimed "a source" is where they got the intel, without specifying the source. It was spreading like wildfire on socials.
The same thing happened with Mac Mini's and OpenClaw. Nobody cared about or was using Mac Mini's for OpenClaw, but there were all of these very suspicious posts from accounts that were clearly Apple marketing bots (you could tell by looking at their post history, where they would drop "Mac Mini" into every conversation they could across all different subreddits and unrelated topics). Then it became fairly common.
So Apple's marketing department is seemingly using the same strategy again. Because still, it's impossible to find a reputable source for this claim.
by nullbio