Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- I could never get this about modern machine/deep learning or even the Transformers. Yes, it's not exactly rocket science, but when I see the data flow diagrams, it's not clear what is calculated in real time or multiple steps.
Is it really one big computation f(g(h(x)))?
- There was a longform post on twitter which went through the same derivation at a bit higher level
https://x.com/waterloo_intern/article/2081762065392541951
and in particular this image which clarifies the key essential difference between liner attention and delta network by examining the case of two tokens with same key but different value
https://pbs.twimg.com/media/HOPCc7BaEAAQDtO.jpg?format=jpg&n...
I think for comparison it would also have been good to have how original quadratic attention handles it: since both keys are identical, the attention would be "evenly divided" between both values so the final output would be the average of both values, as opposed to the latest value
by krackers - When I see these types of articles and headlines, it just makes me supremely grateful for all the many people far smarter[1] than me. And humbles me, too, since I actually passed for a "very smart person" in places like high school and undergrad. In fact, I'm 'smart' for an average person, but there are definitely millions of people who make me look like a rube in comparison.
[1] I specifically mean those who are able to hold very big complex ideas and systems in their head, and reason about them, which seems to be an important talent for mathematicians.
by xp84 - A visualized tutorial: https://snowchord.com/blog/linear-attention-visualized/by HonshinM
- The bra-ket notation makes this all very simple/intuitive for me. With "vectors" I always get confused which is horizontal/vertical, and then I just follow blobs, and get distracted, and leave. With bra-kets the whole thing was very intuitive! I'm now going to covert other articles to the notation as I must have missed a lot of good stuff!
(Side notes: I have physics PhD and mild dyslexia)
by dr_kretyn - > You Could Have Come Up with ...
Creating or combining to have something new, that does not already exist is actually freaking hard!
The moment its presented and people go "o, that is not that difficult", "i was able to also do that", or some nonsense like that. Everything looks simply the moment somebody did the hard work.
We have all been there was developers. Thinking we invented something new, and ... then you discover somebody already made it in the 70's and its everywhere. But because it never cross your path, you never realized it existed.
by benjiro29 - LLM written for sure:
> The identity [...] is the whole trick. The outer product is a matrix; the inner product is a number. We no longer store every past key and value. We store their summed outer products in the fixed-size state S_t.
by croemer - Machine learning could need, and probably has needed, some unified math notation for the past 15 years IMO. With that said, it was worse back in the day - when ML papers were the products of researchers from all over, you'd see some wild notation.
Many will likely disagree with me, but inconsistent notation (across papers!) is to me friction. At least in this article the author explicitly explains the notation at the very start...that is not always the case. Rarely, even.
EDIT: Didn't even notice the notation switch, much appreciated.
by TrackerFF