Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- i like your detailed breakdown. Thanks. <3by souravsspace
- They may have all this amazing architecture but Kimi has been super dumb recently. I reckon they’re under compute pressure and quantising to stay afloat. I was a heavy k2.5/2.6 user earlier in the year and built serious features with it, but even k3 now does stupid shit like fail tool calls and get stuck in endless thought trains. K2.6 was spinning its wheels on a problem for over 10 minutes today and then deepseek v4 fixed it in under a minute.
Something is wrong at moonshot.
by dools - Deepseek is also struggling with simple tasks for the last two days. Claude is perfect, with the additional credits.by rurban
- Yeah something is up. I have the same problem with K3 as I had with earlier kimis. I ask it to write <bash>code</bash> every turn, that does not seem very difficult, but kimi gets this wrong a large percentage of the time. Yet, the code it writes is pretty good.
- I know some of these words, but I'd like to understand all of it better. Would anyone be so kind as to recommend a good learning resource for LLM design and the math involved?by ngcazz
- So, unlike what leaders of western labs labs would like you to believe (that Kimi is just the result of distillation attacks), they are introducing new and novel approaches.by constantlm
- "attack"by hlynurd
- "Kimi is largely a byproduct of distillation" and "Kimi is introducing new and novel approaches" are not mutually exclusive, and I'm not sure it's clear from the paper how much of the improvement comes from the new approaches. So I wouldn't take the new approaches to be much evidence about whether the distillation attacks occurred.by FinchNova12
- Moreover, Distillation is a misused term here. Distillation means training a smaller student model using a larger teacher model to completely mimic its behavior. As in you take a base model and create its smaller “turbo” version. Like distillation in Chemistry it means it’s %100 purified version of its teacher.by ozgung
- No! It's just not fair!! What about our financial bubble!by boomskats
- Even if they are distilling, I don't particularly care. Anthropic and others have been distilling copyrighted material by to build these models, largely without permission.by nostromo
- Just tried K3 out for the first time today and it's a legitimate threat.
Temporarily (maybe permanently) using it as my daily driver but it's wild how comparable it is to Opus 4.7/4.8 (what's been my go to for a bit now—wrote a quick post on what I found today [1]).
by rglover - threat?
that's a weird way of describing a near frontier open weights un-crippled useful coding buddy
do you work for OpenAI or Anthropic per chance?
by igravious - Agreed. Dropped my $200/mo Claude I was saturating for the $100/mo moonshot sub and have no complaints
Plus I can actually see the thinking traces and I’m not getting downgraded/denied by anthropic constsntly…
by ATMLOTTOBEER - Better than Opus 4.8 on complex tasks but tends to overthink. It found a bunch of bugs and architecture issues that only 5.6 Sol Max and Fable on my C++ projects.by kroaton
- Great breakdown. After using Kimi extensively, it's fascinating to see how architectural choices like KDA and NoPE translate into such strong real-world performance. Really impressive engineering.
- Anybody getting the result that Kimi 3 is more expensive than Opus 5 or Sol on Cursor? Pretty sure Kimi 3 sucked up a good chunk of my ultimate plan in a few prompts. Anyone have any tools or ways to understand per model usage towards cursor subscriptions? I know there are alternatives to cursor just haven’t made the move yet. (Edit spelling)by gboss
- Oh no! First time dealing with an American corporation?by cadamsdotcom
- The assertions doing the rounds that Kimi K3 and GLM 5.2 are way cheaper than Claude/GPT are not true -- DeepSeek V4 Pro is a lot cheaper but K3 and 5.2 ain't. Turns out that you actually have to fork out some cash for frontier-esque models, be they Chinese or American. Hope that helps.
Source: my bank balance
by igravious - I feel like the Kimi team is amongst the best in the industry to pick and choose what is meaningful from the other models. For example, avoiding the expensive and empirically uncertain mHC in favor of simpler residuals. Latent MoE.
My only doubts are around Linear Attention instead of DSA as this is inherently lossy. You are kind of banking on that your query is inherently in the embedding space of the model already and can be lossy.
by augment_me - They need to work on token cost for unrestrained problem solving.
- I wonder if Kimis KDA architecture is a way closer to online models(models that learn during interference).As KDA updates its state matrix: St =St−1 +βt (vt −St−1 kt )ktT
The expression (vt −St−1 kt )ktT is literally the gradient step of a linear regression model updating its weights (S) on a single sample using Mean Squared Error loss.
Interesting what other labs comes up with from this.
by Giho - "Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead."
It just baffles me that this even works at all. Doesn't it just become a token soup? Is attention that precise that a second token can tell its the second token just because it learns to accumulate something in the embedding space without any sort of inductive bias?
by Ilaurens - At least the SSM layers should have relative PE built in via recurrence, in a hybrid attention model like this.by natrys
- When you have recurrent blocks in your model, you implicitly have a timestep T(amount of recurrent steps). Similar to Diffusion Transformers, it then becomes valuable to encode the knowledge of where you are in this chain somehow. NoPE is more flexible than RoPE for this.by augment_me
- Linear layers use decays (like IIR filters) that naturally provide relative positions. Full attention layers can then be free to develop concepts that attend to each other regardless of distance.by fspeech
- Causal masking allow model to learn implicit positional embeddings. The meme that a transformer block is permutation invariant is not true.by itkovian_
- As a sibling comment points out you don't strictly need positional embeddings for decoder-only causal transformers. You definitely need it for non-causal ones (e.g. the encoder of the original transformer paper!).
And yes accumulation is a good intuition for what's going on. You could imagine a part of the attention head that just kept writing to the same part of the residual stream causing that to keep accumulating (simply via attention summation) as more input tokens come in thereby functioning as a kind of index without the need for any positional encoding.
by dwohnitmok - The primary author of RoPE, Jianlin Su, is currently working at Moonshot (KIMI)by yhocd
- Sabastian Raschka is one of the great LLM researchers/authors. I highly recommend his substackby thatsgcasey