Comments
Hacker News
Pretty incredible performance for the footprint - really interested to see what could be done on slightly more powerful SBCs like some that have been mentioned in this thread.
by kamranjon
i don't care about microcontroller, what is the decent option to have local llm in my raspi4 that does not take 30 seconds to answer?
by anonymous344
While running LLM on tiny device is awesome, I'm more impressed by whatever training has produced the weights
by NooneAtAll3
>esp32-s3
This microcontroller is a beast, currently using it to do dev work on a pi4.
Having two usb ports with one otg lets you do some neat things that would cost $100+ otherwise
by spacedoutman
Why can't this scale to run much larger models on CPU backed by flash with good access patterns?
by chrishynes
Voice-to-Text and Text-to-Voice models are approaching that size. I wonder how close we are to getting small devices that can chat with us.
Imagine a world where your toothbrush could give you tips about dental hygiene - or advertise toothpaste. What a time to be alive!
This is a really neat use of the per-layer embedding trick. It's also worth noting that there viable TTS models that are ~20-30M param, so it might mean you can have a ESP32 with no network access read stuff out to you in near real time!
by rao-v
It's crazy what $5 can buy you in a microcontroller these days. Have a look at these Milk-V boards:
The duo has up to 256MB of memory, and a 1TOPS@INT8 TPU. They run Linux and are $5. I bought 5!
by titzer
Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Pretty incredible performance for the footprint - really interested to see what could be done on slightly more powerful SBCs like some that have been mentioned in this thread.by kamranjon
- i don't care about microcontroller, what is the decent option to have local llm in my raspi4 that does not take 30 seconds to answer?by anonymous344
- While running LLM on tiny device is awesome, I'm more impressed by whatever training has produced the weightsby NooneAtAll3
- >esp32-s3
This microcontroller is a beast, currently using it to do dev work on a pi4.
Having two usb ports with one otg lets you do some neat things that would cost $100+ otherwise
by spacedoutman - Why can't this scale to run much larger models on CPU backed by flash with good access patterns?by chrishynes
- Voice-to-Text and Text-to-Voice models are approaching that size. I wonder how close we are to getting small devices that can chat with us.
Imagine a world where your toothbrush could give you tips about dental hygiene - or advertise toothpaste. What a time to be alive!
- This is a really neat use of the per-layer embedding trick. It's also worth noting that there viable TTS models that are ~20-30M param, so it might mean you can have a ESP32 with no network access read stuff out to you in near real time!by rao-v
- It's crazy what $5 can buy you in a microcontroller these days. Have a look at these Milk-V boards:
The duo has up to 256MB of memory, and a 1TOPS@INT8 TPU. They run Linux and are $5. I bought 5!
by titzer