Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • Love this! Not having spent too much time on understanding the architecture I always struggled to see how transformers get away with compressing all context in a flat vector (e.g. 768 numbers here) between attention and the MLP when computing the current token. But now I think I understand that since you often alternate between attention and MLPs it regularly mixes/queries the information of other tokens into the current computation. Probably common knowledge for everyone that learned about transformers but this made it so much quicker to see.
    by mhl47
  • Did something similar using a SimCity-like simulator for how LLMs work: https://laurentiugabriel.github.io/token-town/
    by laurentiurad
  • One thing that was not intuitive for me is that attention is per token generator and in theory unbounded for context size. Per token approach does not generate attention matrix as presented here but attention vector. It could provide different view and I find that aproach easier to reason about.
    by raluk
  • Visualization is definitely a good way to learn new things. And I also would like to recommend https://bbycroft.net/llm . It has beautiful graphs, clear animations and good introductions, explainng the LLM inference cores well
    by shinyoo
  • Great site, intuitive description. I also found [1] very useful in the past.

    [1] https://bbycroft.net/llm

    by utopcell
  • It seem that everyone is getting into details of how transformers work, but I am more interested in why other setups didn't work.

    Or is it?

    by est
  • Regarding the temperature explanation:

    > "Instead of picking the highest-probability token, we can use different selection strategies to balance safety and creativity in the generated text".

    Safety is definitely the wrong word here.

    Temperature 0 generated text actually has a weird "lack of surprise" character that makes it seem artificial. [1]

    > "high-probability texts can be dull or repetitive. Humans use language as a means of communicating information, aiming to do so in a simultaneously efficient and error-minimizing manner; in fact, psycholinguistics research suggests humans choose each word in a string with this subconscious goal in mind."

    I'd completely drop the dropout explanation. It's just not part of the modern recipe anymore, AFAICT.

    As for the ambitious goal of explaining transformers with a single interactive visualization, I just have a hard time imagining a person is going to newly understand both word embeddings (word2vec blew my mind in 2014) and also gain an understanding of attention.

    I am making my own visualizations for a presentation on "Full Bandwidth Transformers"[2] that I am giving tomorrow at the Deep Learning Study Group (SF) (on zoom for the non-locals)[3]. It's not meant to be stand alone/context free, but I'd love some feedback.

    https://rrenaud.github.io/fullbandwidth_transformer_viz/

    [1] https://arxiv.org/abs/2202.00666 [2] https://arxiv.org/abs/2608.08888 [3] https://www.meetup.com/deep-learning-sf/events/316601593/

    by robrenaud
  • Nicely done. For me the most fascinating thing about attention heads is the place where Attention matrix is already computed and is getting multiplied by Value vector. It behaves exactly like pushing Value vector through Dense layer of ordinary network where Attention matrix forms weights of that layer. So attention head is trained to construct this small single layer network dynamically during inference from Key and Query. And that's the point. That's rarely underlined in explanations of LLMs architecture and for me it's quite amazing that it works so well. This mechanism easy to observe in this particular visualization if you click through it.
    by andblac

Explore Birbla archives