Comments
Hacker News
I really liked the ket notation. I was aprehensive at first, but it makes operations much more clear.
I would have liked some refresher on some variables though (like d_k in quadratic attention).
by juancn
Side note, before you ask: yes, bra-ket notation is called like that because of the brackets.
by _Microft
When I see these types of articles and headlines, it just makes me supremely grateful for all the many people far smarter[1] than me. And humbles me, too, since I actually passed for a "very smart person" in places like high school and undergrad. In fact, I'm 'smart' for an average person, but there are definitely millions of people who make me look like a rube in comparison.
[1] I specifically mean those who are able to hold very big complex ideas and systems in their head, and reason about them, which seems to be an important talent for mathematicians.
by xp84
A visualized tutorial: https://snowchord.com/blog/linear-attention-visualized/
by HonshinM
LLM written for sure:
> The identity [...] is the whole trick. The outer product is a matrix; the inner product is a number. We no longer store every past key and value. We store their summed outer products in the fixed-size state S_t.
by croemer
The bra-ket notation makes this all very simple/intuitive for me. With "vectors" I always get confused which is horizontal/vertical, and then I just follow blobs, and get distracted, and leave. With bra-kets the whole thing was very intuitive! I'm now going to covert other articles to the notation as I must have missed a lot of good stuff!
(Side notes: I have physics PhD and mild dyslexia)
by dr_kretyn
> You Could Have Come Up with ...
Creating or combining to have something new, that does not already exist is actually freaking hard!
The moment its presented and people go "o, that is not that difficult", "i was able to also do that", or some nonsense like that. Everything looks simply the moment somebody did the hard work.
We have all been there was developers. Thinking we invented something new, and ... then you discover somebody already made it in the 70's and its everywhere. But because it never cross your path, you never realized it existed.
by benjiro29
Machine learning could need, and probably has needed, some unified math notation for the past 15 years IMO. With that said, it was worse back in the day - when ML papers were the products of researchers from all over, you'd see some wild notation.
Many will likely disagree with me, but inconsistent notation (across papers!) is to me friction. At least in this article the author explicitly explains the notation at the very start...that is not always the case. Rarely, even.
EDIT: Didn't even notice the notation switch, much appreciated.
by TrackerFF
Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- I really liked the ket notation. I was aprehensive at first, but it makes operations much more clear.
I would have liked some refresher on some variables though (like d_k in quadratic attention).
by juancn - Side note, before you ask: yes, bra-ket notation is called like that because of the brackets.by _Microft
- When I see these types of articles and headlines, it just makes me supremely grateful for all the many people far smarter[1] than me. And humbles me, too, since I actually passed for a "very smart person" in places like high school and undergrad. In fact, I'm 'smart' for an average person, but there are definitely millions of people who make me look like a rube in comparison.
[1] I specifically mean those who are able to hold very big complex ideas and systems in their head, and reason about them, which seems to be an important talent for mathematicians.
by xp84 - A visualized tutorial: https://snowchord.com/blog/linear-attention-visualized/by HonshinM
- LLM written for sure:
> The identity [...] is the whole trick. The outer product is a matrix; the inner product is a number. We no longer store every past key and value. We store their summed outer products in the fixed-size state S_t.
by croemer - The bra-ket notation makes this all very simple/intuitive for me. With "vectors" I always get confused which is horizontal/vertical, and then I just follow blobs, and get distracted, and leave. With bra-kets the whole thing was very intuitive! I'm now going to covert other articles to the notation as I must have missed a lot of good stuff!
(Side notes: I have physics PhD and mild dyslexia)
by dr_kretyn - > You Could Have Come Up with ...
Creating or combining to have something new, that does not already exist is actually freaking hard!
The moment its presented and people go "o, that is not that difficult", "i was able to also do that", or some nonsense like that. Everything looks simply the moment somebody did the hard work.
We have all been there was developers. Thinking we invented something new, and ... then you discover somebody already made it in the 70's and its everywhere. But because it never cross your path, you never realized it existed.
by benjiro29 - Machine learning could need, and probably has needed, some unified math notation for the past 15 years IMO. With that said, it was worse back in the day - when ML papers were the products of researchers from all over, you'd see some wild notation.
Many will likely disagree with me, but inconsistent notation (across papers!) is to me friction. At least in this article the author explicitly explains the notation at the very start...that is not always the case. Rarely, even.
EDIT: Didn't even notice the notation switch, much appreciated.
by TrackerFF