Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- 9.7 tokens/sec actually seems like a lot! That’s fun!by cr125rider
- Wont this wear out the flash memory quickly? I wonder hiw many read cycles can it surviveby rvba
- https://docs.espressif.com/projects/esp-idf/en/stable/esp32/...
It's write and/or erase cycles that cause wear.
by logifail - Reads are effectively infinite. I'm not aware of any upper limit shorter than "end of the universe" timescales.
Flash wears out from writing and the answer is in the tens to hundreds of thousands of writes per cell.
by vitally3643 - This is a really cool project. Thanks for sharing!by althea_tx
- Pretty incredible performance for the footprint - really interested to see what could be done on slightly more powerful SBCs like some that have been mentioned in this thread.by kamranjon
- i don't care about microcontroller, what is the decent option to have local llm in my raspi4 that does not take 30 seconds to answer?by anonymous344
- Run a smaller LLM. It won't be as smart but the one shown here isn't either.by Rohansi
- While running LLM on tiny device is awesome, I'm more impressed by whatever training has produced the weightsby NooneAtAll3
- It's quite sad people collectively behave as if leaderboards have served their time.
In the small parameter regime there is no room for benchmaxxing, so instead of leaderboards becoming useless, their utility was merely reduced to establishing ever smaller models with similar performance on the benchmarks, forcing compression or redundancy to be recognized and eliminated at the modeling level.
by DoctorOetker - >esp32-s3
This microcontroller is a beast, currently using it to do dev work on a pi4.
Having two usb ports with one otg lets you do some neat things that would cost $100+ otherwise
by spacedoutman - If you want to do this at the $1 price point, you can on RP2350, albeit with some limitations. In particular, it maxes out at full speed (12Mbps). The trick is to use the on-chip USB peripheral for one, and connect the other to GPIO pins backed by PIO.
This works today with tinyusb and pico-pio-usb, but I'm also playing with a Rust port which I'm hoping will have higher performance.
by raphlinus - Why can't this scale to run much larger models on CPU backed by flash with good access patterns?by chrishynes
- My guess is because the ESP32's flash is only ~1/4 the bandwidth of the internal SRAM. If you do this on a more powerful system not only is the gap much wider but you also have much more compute you need to keep fed with bandwidth to be efficient.by Rohansi
- Someone did this exact thing recently, but running GLM-5.2 with something like 16GB of DRAM, a standard desktop CPU and nVME SSD.
I think they got something like 10 _seconds per token_ (not tokens per second).
EDIT: it was 25GB of ram and up to 20 seconds per token! https://github.com/JustVugg/colibri
by AussieWog93 - Voice-to-Text and Text-to-Voice models are approaching that size. I wonder how close we are to getting small devices that can chat with us.
Imagine a world where your toothbrush could give you tips about dental hygiene - or advertise toothpaste. What a time to be alive!
- PLEASE DRINK A VERIFICATION CANby walrus01
- And this is why I am extremely bullish on Apple.by EtienneK
- > tips about dental hygiene
They already can. Philips' higher end models have Bluetooth connections and provide feedback through the app.
> advertise toothpaste
God please no. I don't want to have to look for an adblocker for a toothbrush AI.
by mschild - This is a really neat use of the per-layer embedding trick. It's also worth noting that there viable TTS models that are ~20-30M param, so it might mean you can have a ESP32 with no network access read stuff out to you in near real time!by rao-v
- One of the things I have been wanting to try for a while now is something like this with a layer per MCU. I have some crazy ideas with RP2350's talking to each other with dedicated lines fed by PIO going through a combination of interpolators and dual multiply instructions.
PSRAM, Flash, and even SD cards may not have the best bandwidth individually, but they can reach quite impressive rates when you have a shitton of them running all at the same time.
The large scale dedicated hardware systems will still have the edge for performance per watt, but the low entry level and slow incline does make these things quite appealing.
by Lerc - It's crazy what $5 can buy you in a microcontroller these days. Have a look at these Milk-V boards:
The duo has up to 256MB of memory, and a 1TOPS@INT8 TPU. They run Linux and are $5. I bought 5!
by titzer - It really puts into perspective how much of a leech ARM has been on the entire industry. What being a monopoly does to a mfer.by moffkalast
- Which is why folks should stop thinking that coding in Assembly, or C is the only way, as if microcontrollers were stuck in the 80s hardware.
Heck, that one would have no issue running Xerox PARC OSes.
Naturally there are still use cases were a PIC with 4KB would be the best option, but lets not behave as if there isn't anything better in most cases.
by pjmlp - I bought 5!
What are you going to do with 120 of them?
by onion2k - You can buy sub-$0.50 microcontrollers. But even at $5, I don't know why you'd want to run models on them, it's an environment constrained to the point of being useless for this task.
And I hope it stays that way, I don't want MCU shortages...
by skippyfish - Don't forget the 128 bit vector ISA with 32 registers, supporting up to 64 bit int and FP, and with LMUL=8 you can process 1024 bits with a single instruction (at 3 cycles per 128 bits for most operations). Fully supported by GCC and CLANG (xTHeadVector) and compatible with RVV 1.0 with just a command line switch if you use the C intrinsic functions. (a lot of code working on 8 bit elements is binary compatible with RVV 1.0 too e.g. typical memcpy(), memset(), memcmp(), strlen(), strcpy(), strcmp())
When I bought my 64 MB Duo they were $3!
Then for a long time they were $5 for the 64 MB, $7 for the 256 MB, and $10 for the 512 MB.
Sadly, like everything else, they've gone up considerably this year.
by brucehoult