Discussion summary
Small AI models are gaining popularity for use in areas with unreliable networks and limited hardware, with developers exploring local and offline solutions. Discussions include the benefits of small models, hardware options, and potential applications like neuro-symbolic AI.
What the discussion says
- Small models improve offline and low-bandwidth AI applications.
- Using rented GPU instances is a cost-effective way to train models.
- Neuro-symbolic AI can enhance small models' capabilities.
“I've been working on small local models for years with txtai.”
“Most of the model work is just spinning a spinner, not actual computation.”
Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- I looked into this a bit but unfortunately because of starlink most of this won’t be needed
- Fascinating to wonder whether the bigger model finds fewer or more counterfeits than the on-device one.by enoint
- What about small (offline) AI Models in places with weak hardware?
- 99% of the model "work" (meaning the connection to your computer) is just spinning a spinner - something that makes me want to wrap it with a mosh shell so I can just keep moving from network to network.by bombcar
- I think neuro-symbolic AI has a lot of potential here, since small models can handle a lot of conversational inputs, while relying on wired-in solvers for more complex symbolic math/computation needs. https://jjd.io/posts/swollm-bbh-leaderboard.htmlby jdonaldson
- 100% agree on this.
I've been working on small local models for years with txtai (https://github.com/neuml/txtai). I've published close to 100 models that can run local for RAG, Agents, Vector Search and more (https://huggingface.co/NeuML/collections).
by dmezzetti - Where is a good place to start with training SLM these days if you don't have the compute locally?by monkeydust
- I rent cheap GPU based instances by the hour and run on those. Nothing fancy, but $20 can get you a decent amount of compute on a A6000 or H100.by calgoo
- > The RxScanner is a handheld spectrometer that scans a pill with infrared light, then sends the item’s molecular profile to an AI model equipped with a pharmaceutical database. In seconds, the AI identifies the medication from its molecular profile—or reports that it’s phony.
Is every tech, including database search "AI" now?
by prmph - Can't wait to be killed by my toaster because some sexy mossad agent seduced it.by egormakarov
- This made me laugh out loud :)by sscaryterry
- Have you ever tried to indulge an all-consuming urge to kill when you don't have opposable thumbs? Or hands? Or anything other than a bread slot?by a96
- Has anyone used the Rx Scanner mentioned in the opening?by bix6
- Is anyone making LLM-in-a-box for emergency supply kits yet?
I feel that would be handy in all sorts of situations when networks are down.
by tim-fan - I don't if anyone is doing this yet, but I think a small LLM w/ data sets for RAG reachable via APRS and LoRa would be very useful, not just as an individual but for the community around you.by tristor
- Smallest local model able to work with offline wikipedia dump would be one step above just having an offline wikipedia dump.by rasz
- > Is anyone making LLM-in-a-box for emergency supply kits yet?
Maybe someone should be making this, but for rebuilding society in the event of a disaster - a solar-powered black box with most of humanity's knowledge within. Even something running one of the Qwen models would be useful.
"So, we had a nuclear war and need to start from scratch. How do I turn this rock into a computer chip?"
by bluerooibos - I've been mulling over a good use of a large philanthropy spend in the next decade, and I would love to build a bunch of hardware "oracles" that include an LLM. Ideally solid state, visual/audio, solar + usb-c, so, good in a lot of doomsday scenarios as well as just out hiking. It's a fun thought experiment. I imagine making like 1 million of them, they could be sold and genuinely useful, but also given away; once owned, you could use them, or store and put in an emergency box, bury next to the 10k year clock.. a lot of possibilities.by vessenes
- Project NOMAD: https://andrew.ooo/posts/project-nomad-offline-survival-comp...
I think it was on HN at some point too. Basically it's a bunch of offline data dumps (wikipedia, maps etc) with a local AI model on top that you can use to query them with natural language.
by theshrike79 - https://developers.google.com/edge/gallery
Put that on a spare phone
- This is couched in prepper nonsense, but it's got LLM, WikiPedia, maps, etc. A bunch of genuinely useful stuff to keep on a USB stick or whatever: https://www.projectnomad.us/
But, the current model you really want for an emergency kit is Gemma 4 12B QAT 4-bit. At ~7GB on disk, it's small enough to run on a tablet or any modern computer, slowly if you don't have a GPU or modern Apple silicon, but exceedingly smart for its size, excellent vision capabilities, good tool user, surprisingly good reasoning.
by SwellJoe - > LLM-in-a-box for emergency
For most actual emergency scenarios, a device that focuses on storage of large amounts of prepared normal reference material [0] will be wayyyyy cheaper, more durable, portable, and able to run on batteries or being constantly plugged into a somehow-still-normal electrical grid. (Think an e-ink tablet that can run off a 5V battery pack buffering a literal handcrank.)
In contrast, imagine spending the money to build a beefy LLM-running computer with good GPU/RAM, and somehow mothballing it (to depreciate, unused) in a "safe" location for the big earthquake/flood/etc... Then when the disaster strikes and you dig it out, how will you power it when you need it, and for long enough to do anything useful?
Even if wall-current civilization is 20 miles away on the other side of the mountain, are you going to carry it on your back, or are you going to carry food and water to live? If you do drag it there, are they going to let you run it when it cuts into light for surgery or heat to sterilize drinking water?
by Terr_ - I strongly believe this premise in the article is correct - we will see a lot of tiny, hyper specialized models for individual tasks, and perhaps that will converge with an orchestration layer for a generalized intelligence that controls these specialized tiny models, that will be quite capable.
I don't foresee AGI arising out training bigger LLMs (Though investors won't realise that for a while yet).
It's actually how organic brains work - specialized tasks are offloaded to local cortical columns. The overall coordination between these sub-brains creates emergent skills/abilities.
by N_Lens - What about recent models providing correct proofs to open math problems?by looofooo0
- I think the harness and local context should supply that missing piece between general model and bespoke application. Each application has its own context and action quirks that don't generalize well. Maybe it's just 5% but that is genuinely specific. So its rightful place is in context engineering.
I have a long-ass post about how this could be implemented. https://old.reddit.com/r/VisargaPersonal/comments/1um9uyv/st...
by visarga - No this will never work. Domain specific models will never be a thing because intelligence carries over and compounds.
Why didn’t OpenAI release a math specific model? Why not a literature specific one? Why do they instead have generic models of different sizes? And how did all labs converge on this?
Why does Fable just not train on non cybersec and non biology data but instead have clearly costly and annoying classifiers?
by simianwords - > It's actually how organic brains work - specialized tasks are offloaded to local cortical columns.
How are small isolated language models more similar to that than MoE in LLMs?
- General purpose models are always more robust and generally better than smaller narrower models. My bet is that compute will catch up and any “small” model will still be generally capable, just smaller than sota, rather than intentionally narrow. The exception would be for very well defined tasks where the data distribution never varies, but these are rare and don’t really need “AI” anyway when they do exist.by andy99
- I think future is probably more similar to speculative execution (inference/decoding). A small LLM is used to speculate and a large LLM is used to confirm if needed. If the small LLM is accurate enough on N tokens it’s cheap for the large LLM to say looks good and keep moving along.