Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- That README hits all my “this is authored by an LLM” instincts. I presume the codebase is also written by an LLM?by pja
- Standard llama.cpp can mmap the gguf, so it'll stay on disk if it doesn't fit on memory, and the kernel page cache will ensure the hot parts ("resident trunk") stay resident.
What's the benefit of a custom implementation at all?
by mappu - I couldnt find anything explaining the name of this company on their website but is it okay that they’re riding on the name of an open source tool?
SQLite code itself is public domain but I’m not sure about the name.
by righthand - Be advised that the firm behind it ("sqliteai") had a nasty history of using non-open source licenses, e.g. Elastic License. I advise against using anything by them for this reason even if this project currently has an open license.by OutOfHere
- Approximate calculation is putting the cost at ~$5 per million tokens (assuming 42W sustained, 20¢/kWh), and that's excluding hardware and other costs.by bgirard
- Really cool!!
For all of the other commenters - this project isn't about practicality today. Obviously this isn't gonna be as good as using a cloud provider. But the tool draws a line of what is possible. Combination of making the models more efficient, and making local machines more capable can one day get us to a world where very high quality local models are economically feasible.
by ruler88 - They say it's a waste that you pay for the tokens and then the inference provider pays for the electricity. Isn't that how everything works? I pay cucumbers and the farmers have to pay for the water and the fertilizer...
I hope that reasoning is an after-the-fact justification by the LLM that wrote this.
It's a ver interesting idea and I wouldn't mind trying it out, but with a smaller model. At 0.5t/s and reading many gigabytes from the SDD every second... I wonder if this wouldn't be extremely practical if targeting a 500gib or 250gib model, something that is still outside most consumers' laptop.
by ikurei - Idk about you guys but I'd find 0.5t/s useless. Even for long tasks.
I'd rather just shell out the money to offload as much as possible to say 2x 4060ti 16gb with tensor parallelisation. Anything but that low token rate.
This is the sort of thing I'd expect in 20 years for some cyberpunk esque "turtlebot" that thinks at 0.5t/s, is solar powered and performs some menial civic maintenance background task like cutting grass, or scrubbing pavements. Or the "slowbot" that sits in the garden slowly pruning a bonsai, only just keeping up with the growth of the young plant.
by fennecbutt