Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • this is cool but like, are we just vibe coding NAND burners at this point? these decode times don't really tell the whole story, because prefill becomes the bottleneck.

    half an hour to process 10k tokens on an M5 seems... not great

    by brrrrrm
  • agree, prefill is the weak spot right now. it goes through the same per-token path as decode, which is dumb for long prompts. The fix is on the list: during prefill we can batch the expert reads for the whole prompt per layer instead of per token, that amortizes the IO a lot. until that lands, honest answer is this is good for chat-length stuff, not for feeding it a 10k token document.
    by leonickson
  • SSD NAND reads are nearly infinite. Still makes me uncomfortable, but writing is what kills. There's a reason SSDs are rated by TBW, not TBR.
    by piyh
  • Not great for coding, or realtime agent interactions. But for background processing tasks overnight? Seems like it’d work pretty well
    by kennywinker
  • This is how progress happens, someone gets to 3t/s, the next person gets to6/s and eventually we get to 100t/s.

    People like this person are laying the foundations.

    by fsuts
  • See also: https://github.com/Helldez/BigMoeOnEdge
    by lern_too_spel
  • How can I do this with, say, Gemma?
    by lenerdenator
  • Yeah, original TurboFieldfare supports Gemma https://github.com/drumih/turbo-fieldfare
    by gitpusher42
  • Is there something that already runs like this on android / linux ( / windows)? (ollama or something?)

    Or could this be ported to work on other such platforms?

    edit: AI mentions a "BigMoeonEdge" project

    by erelong
  • "As far as we know, that is the first time a model of this class has run natively on a phone.".

    I feel like I've seen a similar statement on a lot of these streaming weight projects.

    400b model on an iPhone: https://x.com/anemll/status/2035901335984611412

    by sallymander
  • > One expectation to set honestly

    Hello Claude!

    by throwawayffffas
  • Let me just make sure first instead of guessing
    by sbsbdbdbfndj
  • The web and connecting to other services is very important for almost all of my use cases. While I believe we are going to get better and faster models, the web index is certainly not downloadable and maintainable for 99.99% of the folks who are able to use local models. Any good solutions exist?
    by gizmodo59
  • If you're really into self-hosting I've been experimenting with SearXNG and early signs are promising
    by morgoo
  • There are many search APIs available, I like Kagi's.

    Microsoft and Amazon both provide web snapshot services that purport to give you a kind of agent-first internet archive. You can approximate something like that using common crawl, but it's a huge amount of data. Downloading the internet is impossible or a bad idea for almost everyone.

    by pbronez
  • I know relatively little about the workings of LLMs, but I keep seeing projects like this that run massive MoE models using very modest amounts of RAM, perhaps excessively so. I wonder, is there a way to make the RAM usage tunable? I have a Macbook with 32 GB of RAM, and it'd be great if I could run the same model but take advantage of the additional RAM to make it run faster.
    by CyLith
  • It's tunable, --cache-gb N on the CLI. In my sweep the speed barely moved between a 1GB and 6GB cache (43% vs 70% hit rate, same tok/s) because right now the bottleneck is GPU dispatch, not the SSD. so more RAM doesnt buy much yet. once the kernel work lands it should start to matter, so on 32GB I would just set 8 and let it age well. Also the hit rates themselves answer the "can you even know which experts stay hot" question, reuse across tokens is very real.
    by leonickson
  • I guess you have to know which experts to keep “hot” in ram, which you can’t know beforehand, so there wouldn’t be much gain.
    by ianmurrays
  • That's about the turning point for just using typical quants for me. Larger still and you can just do the full model. Smaller to this degree and you need all sorts of extra tricks to get anything.
    by zamadatix
  • > I wonder, is there a way to make the RAM usage tunable?

    In LM Studio I can tune it by selecting different quantation of the model, by selecting how many layers of the neural net to be loaded to GPU (rest stays in main mem, evaluated by the CPU), and by adjusting context window.

    by fodkodrasz
  • This looks useful, you can increase the RAM cache so if you have a Mac with 24-32GB it should speed up a lot and still run models that wouldn’t normally fit. I’m going to run some tests…
    by adrianco
  • Thank you for using TurboFieldfare as a starting point for this project and thank you for mentioning it at the README. I am glad it inspired more people to explore area of on-device AI further!
    by gitpusher42
  • I read a comment on here a few months back I wanna restate. Basically, there is a good chance that Apple is betting that the LLMs in the future will be so efficient that those that consumers will use everyday will be easily computed by the iPhone or even bigger ones on Macs. Honestly makes the most sense that we are heading that way in a few years latest.
    by AHASIC
  • What hardware advances would we need to see for that to happen? It feels like everything in that arena has kind of plateaued.
    by Mistletoe
  • The innovation curve doesn’t always reward the innovators. It pays to let others test the waters before you.
    by greggsy
  • I know everyone wants to crap all over these setups that are impractical, but this is how progress happens.

    People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc.

    Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.

    by dghlsakjg
  • If 1t param model would run from SSDs those will stop costing only $200.
    by jarek83
  • There are certain physical limits. Calculations need to be done. Either less calculations are necessary for the intelligence, or u accept less intelligence. But there is a limit in what u can do with specific hardware.
    by hagen8
  • >when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.

    I am 100% sure this won't happen in 10 years time. At least not on a $200 SSD. But I wouldn't be surprised if it ran on $1K to $2K HBF SSD. It will still be better than a $50K Rack.

    by ksec
  • > wearing the hard drive

    Do reads wear out nvme? Iirc only writes do

    by Perz1val