Ternlight – 7 MB embedding model that runs in browser (WASM)

Join the discussion

• Hacker News
• Why do these things download into the browser automatically? This could be used to distribute malware and also or hog excessive browser memory.
  by rvz
• That's... how the web works? You download things on demand.

  There are JS files larger than 7MB in the wild. They run on JIT engines that displayed severe CVEs over the years. PDFs, video running directly on special hardware encoders. That's the web now.

  A WASM model is not that offensive.

  by gaigalas
• This doesn't add any malware risks beyond what a JavaScript-enabled browser already allows.

  Re excessive browser memory use: Yes, it adds non-negligible weight, but again, you could already achieve excessive browser memory usage before this. For comparison, a true color 1080p image, uncompressed (which is needed for actual display on screen) is only slightly smaller at 6.22Mb.

  by akoboldfrying
• cool stuff
  by ljcoco
• Very cool! I'd love to point it at my own corpus to index/embed. Would be cool if you could give it a link to a markdown file or even a website to crawl.
  by newspaper1
• love the idea! Will think of a way to host it probably on huggingface
  by soycaporal
• That's really impressive, congratulations. It's nice to see novel applications of browser models.
  by gaigalas
• thank you! hopefully it can unlock some novel applications, that would be cool
  by soycaporal
• Interesting project. Happy to see someone who shares an interest in tiny vector embeddings models. I've worked on tiny (1MB - 4MB, 250K - 950K parameters) embeddings models called BERT Hash https://huggingface.co/blog/NeuML/bert-hash-embeddings

  Keep up the great work!

  by dmezzetti
• What we need is a W3C LLM API like the one Chrome already offers: https://developer.chrome.com/docs/ai/built-in
  by esafak
• I think standardizing the runtime is pretty effective, it then open up portability
  by soycaporal
• If it was like Math (Math.round, Math.PI, etc.) it could be Language, as in:

      Language.complete('the quick brown fox jumped over the lazy') 
  

  and maybe even static methods on Image

      Image.generate('a spaceship flying toward a planet')

  by yesidoagree
• Prime example of wasm supremacy over JavaScript. Stack machines for the win hehe
  by iberator
• FWIW -- Granite r2 small is a 30M model, still small enough to run on CPU, and a good baseline for fine tunes.
  by jbellis
• awesome, noted, looking for capable teacher models to distill other architectures
  by soycaporal
• Great, now my websites are gonna push entire LLMs onto my browser in order to use my CPU to make inferences about my shopping habits or whatever.
  by CobrastanJorji
• Disabling WASM is the new disable JavaScript
  by antonvs
• Ha, I was literally thinking this but from the other side.

  "Hmm, 7MB would barely make a dent in the size of the app and allow us to do some of our basic ML without calling the backend"

  Probably a lot more practical to use this though: https://developer.apple.com/apple-intelligence/

  by paytonjjones
• If you think about it, running a crypto miner without being asked is probably less annoying than downloading an entire LLM, but only the first will get you in jail.
  by iammrpayments
• Demo works quite strangely. For example "how to use typescript with createContext" show only typescript entries on top. Similarity search failed.
  by bvrmn
• Nice work.

  It’s advertised 7MB, but also comes with a 5MB mini version.

  Looks like mini saves space by using 256 element vectors internally instead of 384, but then projects it up to 384 at the end for compatibility.

  It’s a third smaller, but the loss is not linear, looks like you give up less than 1/3 of information with the smaller data path.

  by WhitneyLand
• so this is really cool and I think could be the missing piece for something I wanted to build, I found this awhile back and using https://github.com/npiesco/absurder-sql you could keep the entire raw corpus in browser (persisted via IndexedDB/SQLite)...then you could generate + cache embeddings on demand with Ternlight (instead of pre-indexing everything i.e., https://weaviate.io/blog/chunking-strategies-for-rag). then this opens up the door for Reciprocal Rank Fusion (RRF) aka hybrid retrieval where you combine FTS5/BM25 from the native SQLite plues the semantic search using from TernLight!
  by scritty-dev
• Cool project! I tried something similar a while ago [1] - I wanted to load up an embedding model and semantically order texts, all in the browser.

  So I pull ONNX weights from HuggingFace (MPNet, MiniLM), use Transformers.js to embed, and use a clusterer from scikit-learn (running on pyiodide - it was a surprise to me that this worked flawlessly) on the page - all client-side.

  [1] http://sol.quipu-strands.com/

  by abhgh
• Thank you for this! Local models will bring privacy at some point, and I already know an excellent use case for such a small embedding model (cheap and fast search in a product base). Relying on the CPU is also a plus in my case.
  by chris-hartwig
• that's great! let me know if there is anyway I can support, or any specific use case a roadmap could address!
  by soycaporal
• I added an offline search engine to app.wazzup.im/search (no login or payment required).

  First search downloads the model from the internet and subsequent runs are from the cache.

  The model is very small so it's not the best for everything but it's good for basic math and coding.

  Give it a try.

  by wazzup_im
• Here's a video https://youtu.be/X6M7T0lLqTo
  by wazzup_im
• In Safari, stuck on:

  Loading model... + Loading search results...

  Or sometimes "Service Worker API is available and in use." + "Loading search results...".

  by Barbing
• Can the 30 second embedding time be done beforehand and sent to the browser?

  Inference is nice and quick after that.

  by aetherspawn
• yes, you could run a 1 time indexing run on the server side, and just ship the embeddings to frontend
  by soycaporal