Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- At first I felt bad about not having come up with this solution. But then I realized I have problems with writing binary search by myself in JS and immediately felt better.
Now way I could have come up with Kimi Delta Attention.
by piterrro - Lots of linear algebra codes are actually “easy to write” in a way. It isn’t like conventional CS where you are always going a bunch of recursive nonsense going on. There should be mathematical relationships between all of the variables, there are well implemented libraries for the common mathematical concepts, and it is rare to need to go more than a couple loops deep (anything more complex than that should get shunted off into a library anyway).by bee_rider
- I've only read 1-4 without prior knowledge of qkv attention and I like the explanation. The only thing I didn't get from the text was why we divide by the square root of d_k, but ChatGPT explained it was to counter variance scaling with the dimension of the matrix. I probably couldn't have come up with this myself, but I feel that it actually makes sense now and I like that deltanet attention "learns a learning rate" according to one interpretation of beta from the text, if I'm understanding it correctly. Great article!by alf42red
- "Machine learning architecture is second nature to us AI researchers, so it's easy to forget that the average person probably only knows the formulas for gradient descent and ReLU."
"And softmax of course."
by alex-moon - Side note, before you ask: yes, bra-ket notation is called like that because of the brackets.by _Microft
- I could never get this about modern machine/deep learning or even the Transformers. Yes, it's not exactly rocket science, but when I see the data flow diagrams, it's not clear what is calculated in real time or multiple steps.
Is it really one big computation f(g(h(x)))?
- It's all vibes.by leonvoss
- What's your distinction between real time vs multiple steps? All computation is done in steps.
Is it all one big computation? Its turtles all the way down.
by choilive - Yes.
Each token prediction is one big function call. Then you just recursively generate more tokens until run out of context or the model predicts a next token indicating end of sequence. Technically the model outputs a matrix where the last row is a probability distribution, but I’m counting sampling from it as part of the chain. Hundreds of billions of dollars has gone into just making the function fatter and gradually changing pieces here and there.
by malwrar - There was a longform post on twitter which went through the same derivation at a bit higher level
https://x.com/waterloo_intern/article/2081762065392541951
and in particular this image which clarifies the key essential difference between liner attention and delta network by examining the case of two tokens with same key but different value
https://pbs.twimg.com/media/HOPCc7BaEAAQDtO.jpg?format=jpg&n...
I think for comparison it would also have been good to have how original quadratic attention handles it: since both keys are identical, the attention would be "evenly divided" between both values so the final output would be the average of both values, as opposed to the latest value
by krackers - When I see these types of articles and headlines, it just makes me supremely grateful for all the many people far smarter[1] than me. And humbles me, too, since I actually passed for a "very smart person" in places like high school and undergrad. In fact, I'm 'smart' for an average person, but there are definitely millions of people who make me look like a rube in comparison.
[1] I specifically mean those who are able to hold very big complex ideas and systems in their head, and reason about them, which seems to be an important talent for mathematicians.
by xp84 - Yes. I continue to believe that humans will still be the source of the vast majority of novel ideas, even as they increasingly use AI-related tools to accelerate their works.
One of the though experiments I ran with one of my friends during a recent conversation over drinks was this: raising a bunch of "control group" kids away from the screens and the algorithmic ocean of "normie-tier content," and in a very learner-friendly setting with hyper-strict control on the quality of media and source material they get access to, just like we've been doing it with frontier models. Think of it like a monastery but for kids, while teaching them all the latest advances in our understanding of reality through mathematics, engineering, computer science, deep learning, and whatnot.
What I'm getting at it is that we might still need super smart people to push the boundaries of knowledge while using super-advanced AI tools, and anyone who says AI will "completely replace" humans are just misguided. We will always need super smart people with largely unadulterated thinking.
by abixb - A visualized tutorial: https://snowchord.com/blog/linear-attention-visualized/by HonshinM
- The bra-ket notation makes this all very simple/intuitive for me. With "vectors" I always get confused which is horizontal/vertical, and then I just follow blobs, and get distracted, and leave. With bra-kets the whole thing was very intuitive! I'm now going to covert other articles to the notation as I must have missed a lot of good stuff!
(Side notes: I have physics PhD and mild dyslexia)
by dr_kretyn - > You Could Have Come Up with ...
Creating or combining to have something new, that does not already exist is actually freaking hard!
The moment its presented and people go "o, that is not that difficult", "i was able to also do that", or some nonsense like that. Everything looks simply the moment somebody did the hard work.
We have all been there was developers. Thinking we invented something new, and ... then you discover somebody already made it in the 70's and its everywhere. But because it never cross your path, you never realized it existed.
by benjiro29 - LLM written for sure:
> The identity [...] is the whole trick. The outer product is a matrix; the inner product is a number. We no longer store every past key and value. We store their summed outer products in the fixed-size state S_t.
by croemer - This is what you get when you prompt claude to avoid –by geraneum
- Ya, probably started with asking for a buzzy title.by robertclaus
- Machine learning could need, and probably has needed, some unified math notation for the past 15 years IMO. With that said, it was worse back in the day - when ML papers were the products of researchers from all over, you'd see some wild notation.
Many will likely disagree with me, but inconsistent notation (across papers!) is to me friction. At least in this article the author explicitly explains the notation at the very start...that is not always the case. Rarely, even.
EDIT: Didn't even notice the notation switch, much appreciated.
by TrackerFF - > At least in this article the author explicitly explains the notation at the very start
They explain one particular aspect of the notation but never define the variables used. What is k? q? S?
It's obvious if you've studied machine learning before, and for some of them you can make an educated guess, but it makes the article mostly opaque if you don't already have some domain-specific background knowledge.
by Asraelite - I used to think this, then I realized that the amount of time you spend with equations is so much more than code, and the terseness makes them much easier to read once you know what the symbols are.
Also, letters avoid having to name them, naming being a hard problem and all.
by whatsakandr - I never understood people who preferred traditional math notation (e.g. single letter symbols, weird characters like ∣q⟩ instead of writing down an explicit type, etc.). I guess the main advantage is terseness? To me, the mathematical expressions would be so much easier to understand if they were just written in pseudo code or an actual programming language like Python.by olalonde