Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- not sure ive ever seen a #1 post on HN with only 5 stars
- I think this is cool not for kimi but for sth like glm flashby alex7o
- I wonder if faster SSDs would help?
In particular you can still get used Optane SSDs on eBay, although they’re fairly pricey. (not the bogus m.2 ones that are slower than a halfway decent consumer NVMe)
by pjdesno - Probably not. Each read here is a whole 17.5 MB expert file, so the time per read is set by the drive's throughput, not its access latency — 17.5 MB at 7 GB/s is ~2.5 ms, which is what we measure at queue depth 1 on the SN8100s.
What actually moves the barrier is how fast the slowest of 16 concurrent whole-file reads completes: more drives on direct ports, stable tail behaviour under load, and scheduling. Our ladder shows even that with diminishing returns (one drive ≈52% of four, three ≈90%).
by Argonautlabs - Most desktops and mini-PCs can read simultaneously from only 2 internal M.2 SSDs, 1 of which is PCIe 5.0 and the other is PCIe 4.0.
They may have more M.2 sockets but usually except for the first all the others share the same PCIe 4.0 path.
Those 2 M.2 SSDs have a theoretical maximum aggregated throughput of 24 GByte/s, but what is achievable in practice is typically only slightly above 20 GB/s.
If you have 1 or 2 USB 4 / Thunderbolt 4 ports, the aggregated throughput can be increased with external SSDs, like in TFA.
With 2 internal SSDs and 2 fast external SSDs, a higher throughput than in TFA is possible.
by adrian_b - LLMs tend to care for sequential bandwidth more than random access. From that perspective, Optane probably doesn't offer much of an advantage vs other NVMe drives even from the same era while losing out greatly to newer drives with faster busses to fill. There could be something special to this particular instance but https://github.com/argonautlabsai/deltafin/blob/main/k3-publ... and the overall tok/s numbers seem to suggest it's just the sequential that's playing in.by zamadatix
- Now imagine the token/s rate decline after context fill at 200,000+ context.by walrus01
- Fair, and we didn't measure it.
Decode was flat from 128 to 512 generated tokens (0.926 → 0.923 tok/s drafter-off), but that's a 6-token prompt plus the output — total context under a thousand.
The current configuration admits about 4.4k tokens of context at all, and at anything like 200k the killer wouldn't be decode, it would be prefill: today it reads each layer's experts once per 64-row pass,
so 200k tokens of prompt would be measured in days, not minutes, until the scheduling fix.
by Argonautlabs - That's next level masochism.by willmadden
- And macOSismby netc
- Should LLMs be designed to be modular, so that instead of needing access to the whole model, for a given prompt, only a small subset of the model would be used? If knolwedge was sufficiently modularized, most of it could be ignored.
Maybe a hyopthetical model of 5T of indexable weights could be used with only 50 GB of GPU ram, efficiently, because it stays resident in the GPU.
by robrenaud - Isn't that the definition of an MoE model?by gsora
- There's a tendency to cite only decode speeds, but in practice, an LLM generates far fewer tokens than it has to read (unless you ask general knowledge questions). So the effective performance is much slower than the decode rate suggests, because a 512-token prompt already takes 6 minutes to loadby kgeist
- Generally infill is significantly faster than inference due to batching. Is that not the case here for some reason?by vlovich123
- I missed the explanation for how the SSDs are connected.
Maybe a dumb question.
by bluechair - SSDs are connected via Thunderbolt 5 enclosures. I have one Gen4 and three Gen5 ssds inside enclosures. You can see specs here
https://github.com/argonautlabsai/deltafin/tree/main/k3-publ...
by Argonautlabs - The SSDs are necessary because Apple's architecture doesn't allow RAM upgrades. Reminds me of someone who said "640KB ought to be enough for anybody".by amelius
- Gates never said that, for what it's worth.by kulahan
- Apple uses SoC's which means the RAM is simply not upgradable because it is a SYSTEM on a CHIPby pmarreck
- You would have the same issue with a GPU. I have 8 slots on my CPU but it cant handle 2.8TB of ram. I cant even buy 2TB for it because even though the mobo is rated for it, nobody ever made those chips (someone will sell me 8x256gb 8Rx4 for $15k but there’s no way the memory controller can handle that). 1TB is $10k and forced it to run slower.
- To be fair, nobody has upgradable memory in any system that has enough memory bandwidth and compute power to run LLMs with decent performance. It might be interesting to compare against some decade-old x86 server or workstation stuffed full of LRDIMMs to reach 1.5–2TB of RAM, but the bandwidth would be only slightly faster than a desktop today with high-end DDR5: nowhere close to GPU bandwidth. So performance would still suck.
Designing for extreme expandability comes with pretty steep tradeoffs.
by wtallis - You currently can't run a 2.8T locally; there's just no way. So, it's a good start.by mandeepj
- But my local is a 8xH100, you insensitive clod!
- A medium prompt in only 11 days.by dusted
- 3600 words in one hour.by meerita
- A medium prompt = 1 million tokens?by jgalt212
- when it finishes answering you already figured out the questionby pvab3
- Reminds me of Deep Thought from Hitchhiker's Guide to the Galaxyby lukeduff
- It's kind of insane how having this tech at this speed 5 years ago would have probably still been seen as insanely useful and revolutionary. If LLMs were more capable but dramatically slower, I wonder how it would impact how we use it? Dramatically more thought being put into prompts, much more preparation probablyby schmorptron