Discussion summary
A 7MB in-browser embedding model called Ternlight was discussed, highlighting its potential uses and concerns about browser resource usage.
What the discussion says
- Some users expressed concerns about malware and memory hogging.
- Others emphasized that downloading resources is standard web behavior.
- Several users praised the technical achievement and potential applications.
“This could be used to distribute malware and also or hog excessive browser memory.”
“That's... how the web works? You download things on demand.”
Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Nice work.
It’s advertised 7MB, but also comes with a 5MB mini version.
Looks like mini saves space by using 256 element vectors internally instead of 384, but then projects it up to 384 at the end for compatibility.
It’s a third smaller, but the loss is not linear, looks like you give up less than 1/3 of information with the smaller data path.
by WhitneyLand - so this is really cool and I think could be the missing piece for something I wanted to build, I found this awhile back and using https://github.com/npiesco/absurder-sql you could keep the entire raw corpus in browser (persisted via IndexedDB/SQLite)...then you could generate + cache embeddings on demand with Ternlight (instead of pre-indexing everything i.e., https://weaviate.io/blog/chunking-strategies-for-rag). then this opens up the door for Reciprocal Rank Fusion (RRF) aka hybrid retrieval where you combine FTS5/BM25 from the native SQLite plues the semantic search using from TernLight!by scritty-dev
- Interesting project. Happy to see someone who shares an interest in tiny vector embeddings models. I've worked on tiny (1MB - 4MB, 250K - 950K parameters) embeddings models called BERT Hash https://huggingface.co/blog/NeuML/bert-hash-embeddings
Keep up the great work!
by dmezzetti - Cool project! I tried something similar a while ago [1] - I wanted to load up an embedding model and semantically order texts, all in the browser.
So I pull ONNX weights from HuggingFace (MPNet, MiniLM), use Transformers.js to embed, and use a clusterer from scikit-learn (running on pyiodide - it was a surprise to me that this worked flawlessly) on the page - all client-side.
by abhgh - amazing.. glad to know this integration path worked fro you!by soycaporal
- Thank you for this! Local models will bring privacy at some point, and I already know an excellent use case for such a small embedding model (cheap and fast search in a product base). Relying on the CPU is also a plus in my case.
- that's great! let me know if there is anyway I can support, or any specific use case a roadmap could address!by soycaporal
- What we need is a W3C LLM API like the one Chrome already offers: https://developer.chrome.com/docs/ai/built-inby esafak
- I think standardizing the runtime is pretty effective, it then open up portabilityby soycaporal
- If it was like Math (Math.round, Math.PI, etc.) it could be Language, as in:
and maybe even static methods on ImageLanguage.complete('the quick brown fox jumped over the lazy')Image.generate('a spaceship flying toward a planet')by yesidoagree - I added an offline search engine to app.wazzup.im/search (no login or payment required).
First search downloads the model from the internet and subsequent runs are from the cache.
The model is very small so it's not the best for everything but it's good for basic math and coding.
Give it a try.
by wazzup_im - Here's a video https://youtu.be/X6M7T0lLqToby wazzup_im
- In Safari, stuck on:
Loading model... + Loading search results...
Or sometimes "Service Worker API is available and in use." + "Loading search results...".
by Barbing - Can the 30 second embedding time be done beforehand and sent to the browser?
Inference is nice and quick after that.
by aetherspawn - yes, you could run a 1 time indexing run on the server side, and just ship the embeddings to frontendby soycaporal
- This would be nice as an Astro (or generic meta-framework plugin) that automatically parses all generated html files and generates a small db of embeddings.
This way on the frontend you can lazily load this. Maybe you could even store the HNSW in chunks and just load the pieces you need for your specific search query.
i.e. like https://pagefind.app/ but to get fully static vector search.
by aziis98 - We really wanted to use sqlite-vec for this for our SSG but last we checked it hadn’t implemented HNSW/had good support for running vector search in-browser yet (I think it was still doing full-table scans?). I was pretty disappointed because after so many months/years, to not have that suggested to me that they weren’t up to task of delivering on their project, and I had recommended them as a worthy project for a grant I had also applied for, that they won and I didn’t.
If anybody knows of a good solution in this space, or if I’m wrong about SQLite-vec, please let me know. For our own SSG we’ve basically decided that we’ll give it a couple months while we work on other infra we want, then if they’re still not done we’ll just do it ourselves.
by weitendorf - This would be a pretty cool addition to the duckdb HNSW search project I found on here some time ago: https://github.com/jasonjmcghee/portable-hnsw
What I think is really cool is that the search happens using http range queries across statically hosted parquet files.
I think things like this could bloom into a relatively open and distributed search ecosystem that isn’t controlled by major corporations.
by kamranjon - very cool, I'll look into this. Thanks for sharing.by soycaporal
- Cool idea. I love range requests and other static-hosted client-navigable formats!by TheTaytay
- Similar idea here that may be of interest: SQLite DB on static host via HTTP range + WASM.by lexh
- This is cool!
but also maybe you could put a button on the landing page to trigger the demo because it's a bit startling to hear my fans go crazy when opening a webpage.
by dirteater_ - CPU cycle maxxing, who said GPUs were special?by soycaporal
- Same here, when the fans started up I got startled. However my bread toaster often scares me tooby jonnonz
- Agree. But this also reminds me fondly of the days where the sounds of my computer so intimately indicated what’s going on.by Waterluvian
- Hobby project, I wanted to "ship a useful model in a web browser". so I distilled a small sentence encoder from MiniLM with ternary quantization-aware training. Also wrote the inference engine from scratch and shipped in Rust → WASM SIMD.
It's an embeddings model, not an LLM: text goes in, a 384-dim vector comes out, and cosine similarity between two vectors tells you how related the texts are — regardless of shared words ("reset my password" ↔ "I forgot my password" → 0.88). Used for semantic search, FAQ/intent matching, and clustering. Running it on-device means search-as-you-type semantic search is performant with no API dependencies.
Demo (2k React docs, fully on-device): https://ternlight-demo.vercel.app
Two tiers on npm: - @ternlight/base (7 MB, ~5 ms/embed, more capable embedings) - @ternlight/mini (5 MB wire, ~2.5 ms/embed).
Bundled for Node and browsers.
Repo - see technical details (MIT, training pipeline included): https://github.com/soycaporal/ternlight
Curious if this is something useful, what are the use cases for on-device embeddings.
by soycaporal - What is the process for adding different text? What are the limitations on that process? The demo is very cool by the way.by abrookewood
- Thank you for this tool!
We've just used it to embed the entire django doc + our private knowledge base, allowing us to search in the 2 sources instantly!
by sodimel - Awesome! Besides size, how does this compare to gte-small?
- Huge kudos for sharing everything including your training code. Awesome project!by dannyw