Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Claude might as well be .5 tok/s. I end up waiting several minutes and what it tells me could usually be summarized in under 100 words.
So I could potentially live with this if it was concise.
by bilsbie - In my experience Kimi k3 is even more verbose.
- I have one setup that gets about 1.5 tok/s of a very large on prem LLM, on a system that lives under my desk. It's used for overnight project review runs and code review that it is fed at the end of each work day. When I look at it the next morning it has done quite a lot of useful work. Dealing with a big slow LLM as an effective tool is really about planning the workflow to feed it.by walrus01
- This is not how thinking works. Claude uses tens of thousands of thinking tokens to get to 100 words. Kimi is no different.by bpodgursky
- Thanks guys for all the comments, I am going to rewrite the README (without using an LLM)by marcobambini
- Pretty soon we'll have 3T param models down to 1 bit. They'll be able to tell you whether they're off or on.by hddambo
- Interesting project. The headline number (29 GB of RAM) is for 4k context.
From what I've read elsewhere, Kimi K3 is quite verbose in its thinking. At the quoted rate, it would generate only a total of 1.8k tokens in 1 hour. Is that enough for it to get any thinking done and produce output on more complicated prompts?
by logicallee - I saw someone’s excellent idea that if you have a slow system like this, you should communicate by email. It is no longer meant for realtime iteration, but more pointed questions for which there is more effort and time expected on both parties.by 0cf8612b2e1e
- That README hits all my “this is authored by an LLM” instincts. I presume the codebase is also written by an LLM?by pja
- I prefer to have such detailed readme's created by a LLM while iterating over no iteration documentation at all and usually the later is the standard.by cdud3
- Yes, but very slowly.by pkulak
- it is definitely the claudiest. it's weird, it's like there's a spot developing in my head next to all the other spots where i park mental models for how people write, but instead of being for a person, it's for the terse-staccato-prosodic-diarrhea that claude generates in readmes by default.by a-dub
- >Contributors
>...
>claude
You don't need to presume. If someone is so lazy that they tell claude to commit their code (ie. they're too lazy to run git commit themselves), the chances they reviewed the code is slim.
by gruez - Yeah I'm begging these authors to at least *read* the LLM generated README's. They're so, so incomprehensible because the LLM has a super limited theory of mind for readers. They always assume that external readers have access to the full context and history of decisions in the project development. These decisions and instructions from the user are extremely important for the model and almost completely irrelevant for an outside reader looking at a "finished" product. So, we get sentences like this:
"Where the levers were is not where they are. Overlapping the expert reads with the arithmetic was worth ~1.6x and shipped; the two that looked bigger — reading fewer bytes per token, and keeping more of them in RAM — were both measured and both refused, one because this family's router has no tail to demote and one because a cache the machine will not leave resident cannot be bought at any price."
What the fuck does that mean? Obviously some internal development decision, using the absolutely inscrutable internal terminology that Claude loves. If people would just read what they publish, I'm sure this would stick out immediately.
I'm not an LLM hater, I use them a ton and they work very well for writing complex code, it's undeniable. But they generate absolute dogshit first draft writing.
by bglazer - I wrote tons of software, even a programming language by hand https://github.com/marcobambini/gravity.
I'm using my skills to orchestrate LLMs and agents, and I can write better code much faster. As developers, we can choose to adapt to new technologies or become extinct.
by marcobambini - Yup, I hate to engage in anything that looks like a "shallow dismissal" but the project documentation seems to outright contradict itself wrt. whether it's running the model at genuinely native precision (though the claimed 3-bit quant is potentially interesting) and the headline claim of achieving 2 secs/token in a mere 29GB RAM footprint looks outright nonsensical given what we know about K3 itself (~115GB in dense parameters alone at native precision, plus ~25GB active sparse experts per token and some comparatively minor footprint for the KV cache). This is just not very helpful.by zozbot234
- Standard llama.cpp can mmap the gguf, so it'll stay on disk if it doesn't fit on memory, and the kernel page cache will ensure the hot parts ("resident trunk") stay resident.
What's the benefit of a custom implementation at all?
by mappu - Letting the kernel use SSD based swap space for something this big would be a good way to destroy its cumulative write endurance over a period of just a couple months. I would be very interested in seeing SMART self reported drive cumulative write and wear out stats if this was done for more than a short test.
In my experience llama-server is better run with --no-mmap on things that will fit entirely into RAM. Though obviously you need a 2TB server for full Kimi k3 and 1M context.
by walrus01 - There was another project posted a few days ago that was quite similar and someone asked the same thing to the author. They said they tried that first and they got a 10x speed boost by doing it manually. It is the same reason why database engines write their own backing cache logic: the kernel's paging logic is generic and on-demand, while someone that's aware of the actual usage patterns can prefetch and pipeline the right data before it is used.by jnwatson
- I couldnt find anything explaining the name of this company on their website but is it okay that they’re riding on the name of an open source tool?
SQLite code itself is public domain but I’m not sure about the name.
by righthand - Be advised that the firm behind it ("sqliteai") had a nasty history of using non-open source licenses, e.g. Elastic License. I advise against using anything by them for this reason even if this project currently has an open license.by OutOfHere
- Are these the sqlite developers..?by airstrafer
- Approximate calculation is putting the cost at ~$5 per million tokens (assuming 42W sustained, 20¢/kWh), and that's excluding hardware and other costs.by bgirard
- And what if I have PV?by darkwater
- I got to the same conclusion another way. There's ~2.6million seconds a month and this is getting 0.5tok/s which is 1.3million tokens a month. Give some room for overhead and a reasonable rule of thumb;
The cost to run the machine per month is the cost per million tokens.
- Really cool!!
For all of the other commenters - this project isn't about practicality today. Obviously this isn't gonna be as good as using a cloud provider. But the tool draws a line of what is possible. Combination of making the models more efficient, and making local machines more capable can one day get us to a world where very high quality local models are economically feasible.
by ruler88 - They say it's a waste that you pay for the tokens and then the inference provider pays for the electricity. Isn't that how everything works? I pay cucumbers and the farmers have to pay for the water and the fertilizer...
I hope that reasoning is an after-the-fact justification by the LLM that wrote this.
It's a ver interesting idea and I wouldn't mind trying it out, but with a smaller model. At 0.5t/s and reading many gigabytes from the SDD every second... I wonder if this wouldn't be extremely practical if targeting a 500gib or 250gib model, something that is still outside most consumers' laptop.
by ikurei - If you grow your own tomatoes, you'll have free tomatoes!
(Doesn't really get you a BLT but hey... at least you'll have saved the world a bit because they're not from the supermarket)
/s
by Garlef - It is worth thinking about, when you're buying something, how much of what you're paying is the supplier's margin. And whether they have efficiency advantages over you doing it yourself. But for large LLMs it does seem like there's a pretty big efficiency advantage to the rack-scale hardware in datacenters compared to hacks like this.by rcxdude
- Idk about you guys but I'd find 0.5t/s useless. Even for long tasks.
I'd rather just shell out the money to offload as much as possible to say 2x 4060ti 16gb with tensor parallelisation. Anything but that low token rate.
This is the sort of thing I'd expect in 20 years for some cyberpunk esque "turtlebot" that thinks at 0.5t/s, is solar powered and performs some menial civic maintenance background task like cutting grass, or scrubbing pavements. Or the "slowbot" that sits in the garden slowly pruning a bonsai, only just keeping up with the growth of the young plant.
by fennecbutt