Small AI Models Gain Traction In places with unreliable networks

Join the discussion

• Hacker News
• 100% agree on this.

  I've been working on small local models for years with txtai (https://github.com/neuml/txtai). I've published close to 100 models that can run local for RAG, Agents, Vector Search and more (https://huggingface.co/NeuML/collections).

  by dmezzetti
• SLMs for the rescue!
  by fpauser
• I looked into this a bit but unfortunately because of starlink most of this won’t be needed
  by mountainriver
• Fascinating to wonder whether the bigger model finds fewer or more counterfeits than the on-device one.
  by enoint
• 99% of the model "work" (meaning the connection to your computer) is just spinning a spinner - something that makes me want to wrap it with a mosh shell so I can just keep moving from network to network.
  by bombcar
• What about small (offline) AI Models in places with weak hardware?
  by HelloUsername
• I think neuro-symbolic AI has a lot of potential here, since small models can handle a lot of conversational inputs, while relying on wired-in solvers for more complex symbolic math/computation needs. https://jjd.io/posts/swollm-bbh-leaderboard.html
  by jdonaldson
• Where is a good place to start with training SLM these days if you don't have the compute locally?
  by monkeydust
• I rent cheap GPU based instances by the hour and run on those. Nothing fancy, but $20 can get you a decent amount of compute on a A6000 or H100.
  by calgoo
• Has anyone used the Rx Scanner mentioned in the opening?

  https://rxall.net/rxscanner/

  by bix6
• > The RxScanner is a handheld spectrometer that scans a pill with infrared light, then sends the item’s molecular profile to an AI model equipped with a pharmaceutical database. In seconds, the AI identifies the medication from its molecular profile—or reports that it’s phony.

  Is every tech, including database search "AI" now?

  by prmph
• Can't wait to be killed by my toaster because some sexy mossad agent seduced it.
  by egormakarov
• This made me laugh out loud :)
  by sscaryterry
• Have you ever tried to indulge an all-consuming urge to kill when you don't have opposable thumbs? Or hands? Or anything other than a bread slot?
  by a96
• Is anyone making LLM-in-a-box for emergency supply kits yet?

  I feel that would be handy in all sorts of situations when networks are down.

  by tim-fan
• This is HN, not Reddit.
  by burgerone
• I don't if anyone is doing this yet, but I think a small LLM w/ data sets for RAG reachable via APRS and LoRa would be very useful, not just as an individual but for the community around you.
  by tristor
• Can you expand what you mean?
  by cdnsteve
• Smallest local model able to work with offline wikipedia dump would be one step above just having an offline wikipedia dump.
  by rasz
• electricity outage and battery running out is the end game for any real prolonged external emergency. Internet connection is just the soft edge.
  by sumitkumar
• Oh my... I can think of a 101 things more useful in actual emergencies than an LLM-in-a-box. Unless you have a weird definition of "emergency" (if so: please define).
  by RetroTechie
• I've been mulling over a good use of a large philanthropy spend in the next decade, and I would love to build a bunch of hardware "oracles" that include an LLM. Ideally solid state, visual/audio, solar + usb-c, so, good in a lot of doomsday scenarios as well as just out hiking. It's a fun thought experiment. I imagine making like 1 million of them, they could be sold and genuinely useful, but also given away; once owned, you could use them, or store and put in an emergency box, bury next to the 10k year clock.. a lot of possibilities.
  by vessenes
• > Is anyone making LLM-in-a-box for emergency supply kits yet?

  Maybe someone should be making this, but for rebuilding society in the event of a disaster - a solar-powered black box with most of humanity's knowledge within. Even something running one of the Qwen models would be useful.

  "So, we had a nuclear war and need to start from scratch. How do I turn this rock into a computer chip?"

  by bluerooibos
• https://developers.google.com/edge/gallery

  Put that on a spare phone

  by robotswantdata
• This is couched in prepper nonsense, but it's got LLM, WikiPedia, maps, etc. A bunch of genuinely useful stuff to keep on a USB stick or whatever: https://www.projectnomad.us/

  But, the current model you really want for an emergency kit is Gemma 4 12B QAT 4-bit. At ~7GB on disk, it's small enough to run on a tablet or any modern computer, slowly if you don't have a GPU or modern Apple silicon, but exceedingly smart for its size, excellent vision capabilities, good tool user, surprisingly good reasoning.

  by SwellJoe
• > LLM-in-a-box for emergency

  For most actual emergency scenarios, a device that focuses on storage of large amounts of prepared normal reference material [0] will be wayyyyy cheaper, more durable, portable, and able to run on batteries or being constantly plugged into a somehow-still-normal electrical grid. (Think an e-ink tablet that can run off a 5V battery pack buffering a literal handcrank.)

  In contrast, imagine spending the money to build a beefy LLM-running computer with good GPU/RAM, and somehow mothballing it (to depreciate, unused) in a "safe" location for the big earthquake/flood/etc... Then when the disaster strikes and you dig it out, how will you power it when you need it, and for long enough to do anything useful?

  Even if wall-current civilization is 20 miles away on the other side of the mountain, are you going to carry it on your back, or are you going to carry food and water to live? If you do drag it there, are they going to let you run it when it cuts into light for surgery or heat to sterilize drinking water?

  by Terr_
• I strongly believe this premise in the article is correct - we will see a lot of tiny, hyper specialized models for individual tasks, and perhaps that will converge with an orchestration layer for a generalized intelligence that controls these specialized tiny models, that will be quite capable.

  I don't foresee AGI arising out training bigger LLMs (Though investors won't realise that for a while yet).

  It's actually how organic brains work - specialized tasks are offloaded to local cortical columns. The overall coordination between these sub-brains creates emergent skills/abilities.

  by N_Lens
• No this will never work. Domain specific models will never be a thing because intelligence carries over and compounds.

  Why didn’t OpenAI release a math specific model? Why not a literature specific one? Why do they instead have generic models of different sizes? And how did all labs converge on this?

  Why does Fable just not train on non cybersec and non biology data but instead have clearly costly and annoying classifiers?

  by simianwords
• What about recent models providing correct proofs to open math problems?
  by looofooo0
• I think the harness and local context should supply that missing piece between general model and bespoke application. Each application has its own context and action quirks that don't generalize well. Maybe it's just 5% but that is genuinely specific. So its rightful place is in context engineering.

  I have a long-ass post about how this could be implemented. https://old.reddit.com/r/VisargaPersonal/comments/1um9uyv/st...

  by visarga