Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • "interference-to-signal ratio", "capacity", ... These terms, and their context, are things a communications/electrical engineer would understand. I encourage comms engineers to start learning how AI works under the hood. Chances are, a lot of what you know about maximising channel capacity and throughput is applicable to AI (can be seen as a channel with memory?).
    by femto
  • it's great to read a blog post like this.

    I got to this part:

    > If the keys are random unit vectors

    And hesitated here. It seems then the article is predicated on us trying to store "white noise". However in the real world, data has much more structure than that. Every bit of structure is an opportunity to compress more. So all the conclusions here would be an upper/lower bound rather than a true estimate.

    I assume this is accounted for by:

    > In fact we haven’t actually considered the training data at all, which affects what actually is being represented in the keys and values

    but it understates the point: much of the compressability as well as intelligence in the models come explicitly from the structure in the data. So it's very interesting you can derive MLP and transformer architecture even absent that. But the real leverage comes from the interaction of those things WITH structure in the data.

    by zmmmmm
  • Would be nice if articles defined the requirements to be a “you”.
    by yoz-y
  • After reading the article - no, I could not have invented transformers.
    by piterrro
  • Obviously. The breakthrough was called “attention is all you need”! self-attention was being studied as an approach, like alternate schedulers, learning rates, various architecture connectivies, and so on. It turns out self-attention happened to include all of that; that’s the insight.
    by freeone3000
  • I think I can follow most of this, or probably will be able to do so with some help from Claude as we go through it with my questions, but I thought I'd ask humans first.

    I've watched most of Karpathy's videos but it's been a while so if I ask a dumb question it's because I'm at a very basic level from a while ago.

    To try to turn this into something concrete, let's say I want to store as facts the 50 state capitals. We'll do the most basic tokenization possible, so I have ['AL': 1, 'AK': 2, 'AZ': 3, ... 'WI': 49, 'WY': 50, 'Montgomery': 51, 'Juneau': 52, ... 'Cheyenne': 100]

    (Do I need an embedding matrix here? I feel like I need a 100x64 embedding matrix, one for each "token", even for those very first linear examples? Maybe I need two 50x64 matrices?)

    But then I'm learning randomly initialized V and K matrices by going through and dotting my 'AL' row by K and my 'Montgomery' row by V and then dotting them together, and going through all of the rest of the states similarly and then optimizing a loss function until I get a V and K that gives me good performance?

    I kind of wanted to go one step further and adding two more tokens, a 'StateHasCapital' token and a 'IsCapitalOf' token (say 101 and 102), and somehow use that to learn with compound keys ['StateHasCapital'+'AL': 'Montgomery'] - if I'm really learning facts, can I train on all 50 States->Capitals and maybe 25 Capitals->States and then as a test to see if I'm learning "facts", use the remaining capitals that I didn't train on to see if I can look up what state has the capital city of Olympia?

    I might be conflating a whole bunch of steps that don't make sense in that initial linear example and adding complexity, and maybe trying to fit too much into the later examples too (I don't see the 'Query' matrix in the any of the parts of the post that get into self-attention, and maybe this post is just about "here's how memory works, not how LLMs work"

    by tooDumbForML_th
  • Really cool perspective and a great write-up.

    Aside: I feel like this could be another intro example for students on how exponentials are a limit of a sequence of polynomials. We usually first see this in compound interest [1]. But I remember having a similar ah-ha moment when I first saw matrix exponentiation.

    [1]: https://en.wikipedia.org/wiki/Compound_interest#/media/File:...

    by scythmic_waves
  • Hey author here! I have a short summary here if people find it helpful: https://x.com/heyyalexwang/status/2111158818730397932?s=46
    by ketothekingdom

Explore Birbla archives