Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • Does anyone else feel like the writing is on the wall for a future of local models? Spamming data centres everywhere, powering them, having to commit insane capital to hardware, all the effort to serve inference over a network reliably - when here we are with a frontier model nearly running on a laptop.

    Local AI on your device seems like a much more likely future to me than datacenters in space. For inference at least, training is another story.

    by mindwok
  • 0.01 tk/s on an M1 Max is not "nearly". This is completely unusable, and in no way cost effective.

    0.01 tokens per second means 1 million tokens ($3 worth of API usage [1]) takes 3.2 YEARS.

    [1] https://www.kimi.com/resources/kimi-k3-pricing

    by nomel
  • Super cool, and I appreciate the upfront speed disclaimer
    by jrhizor
  • Local AI is going to win. Not because it's cheaper btw.
    by knighthacker
  • > ~60–76 s/token

    I don't know if I'd call this "running"

    by tjwebbnorfolk
  • I commented similarly below, but as a terrible programmer, I probably perform about 1 minute per token too (at Kimi 3 level). It puts into context how I think about intelligence
    by tito
  • Had the same feeling when I first saw min/km units in some (human) running context.

    UPD: I know it's not the same at all, just the reversal of units that gets me

    by sermah
  • Exactly my machine 64GB M1 Max So happy about this! ♡

    idk how people access (soldout) and even afford 512GB RAM MacStudio's. Isn't it $40k or so?

    by ALLTaken
  • You're lucky, mine is the M1 Pro 16GB
    by hmokiguess
  • I just checked eBay. There's an insane price difference between used and new. $5k vs $40k.
    by teaearlgraycold
  • Would be interesting to see how fast it would be on 4x mac studio 512gb machines.
    by mips_avatar
  • Going to try this on my M1 Ultra 128gb.

    The point of these engineering tricks is to see the envelope of what's possible. You can use these tricks to both run a bigger model on smaller hardware or run a smaller model on smaller hardware.

    by mannyv
  • Anyone who knows the state of NVMe hardware more than me know if this would obliterate the lifespan of your drive? Seems like the biggest limitation to me (some people are probably fine with letting their Macs churn over the weekend).
    by nlessard
  • Reads are not generally life-limiting for flash. (Well, no more so than power-on time in general. You still have aging mechanisms like electromigration, but these are orders of magnitude slower than write-induced damage.)
    by addaon
  • No problem at all to read data over and over. In fact, LLM weights are a great candidate for low-quality flash that can't handle a lot of write cycles, and you want a large amount of storage cheaply...
    by trollbridge
  • The title should probably be edited to specify "M1 Max" instead of "M1 Mac". You aren't running K3 on a base M1 anytime soon. Either way, still a very impressive project.
    by acmnrs
  • Done, Mac -> Max
    by tito
  • 0.01 tk/s is unusable for anything, you would wait a whole day for just 1000 token of output, what is the point of projects like this?
    by Azantys
  • You can't improve what you can't measure.

    Consider this like if it were the first test

    by Francute
  • 16 tokens / s is not nothing.
    by winstonp
  • 'Large Language models? They can barely produce gibberish sentences, what would this tech ever be useful for?'

    - bunch of people only ~4 years ago

    by anigbrowl
  • So you subscribe to the belief we won't in future find mentalism in other galaxies or solar systems which operate on mechanisms we don't understand and think v e r y s l o w w w w w w w l y ?

    (note. I am not a believer in AGI)

    "useful" is highly contextual. The clock of the long "now" is not useful in the sense you mean, to synchronise your wristwatch. I'm still glad it exists.

    by ggm
  • I can only guess some post purchase remorse.

    Need to justify buying an expensive rig that doesn't do what you expected.

    Specifically thinking the people they could do something AI with cpu, and realizing it isn't feasible. Happened at my fortune 20 company. They had to get approvals and ofc it was useless. Plenty people tried to explain, but they were the principle engineer, and out ranked everyone.

    "It's not going to work", the topic changed, and we never spoke about it again.

    by whatsThisBtn4
  • I like seeing the latest and greatest model crammed into new systems to see how it fares. To deal with the speed, one person on reddit suggested using it in an email interface rather than a chat interface.
    by tito
  • It is fun.

    Also its answering the question of what gonna happen if you wake up tomorrow and datacenters are gone. Or internets are gone.

    Some people on our globe live in countries with no internet whatsoever. Of course most of them dont have Macbook with 64GB RAM either, but it's much much easier to get than internet connection or rack of GB200.

    SOTA LLMs are efficiently compression of all the knowkedge humanity has built. Having ability to run it at home to extract said knowledge is important no matter the speed.

    by SXX
  • SSD streaming on an M5 Max 128GB: https://x.com/antirez/status/2082136334160818528

    Soon decent speed across two Mac Studios with 512GB of RAM.

    by antirez
  • Cool stuff. Do you have a the hardware and a way to bridge the compute? Or just hopeful?
    by xfour
  • Thanks for sharing antirez! (huge fan btw)

    We've made some updates to deltafin since this was shared 5 days ago. I wonder if you'd be able to get better speeds now? https://github.com/gavamedia/deltafin

    by chrisringrose
  • 0.3tx per second is decent speed?

    And it gets worse with every token.

    by whatsThisBtn4
  • wow cool! I like watching the new models come out and how they end up crammed in to run on local machines. I learned what mxfp4 is thanks to this latest Kimi release - although it sounds like it means that there's less room for compression in the model compared to others.
    by tito