Birbla

Comments

Hacker News

Interesting that they went NoPE everywhere — everyone else hedges with RoPE in the local layers. Feels like the linear-attention stuff (Kimi Delta) is quietly doing the positional work so they can get away with it. Curious to see if it holds up at frontier scale.

by gokohl

Great breakdown. After using Kimi extensively, it's fascinating to see how architectural choices like KDA and NoPE translate into such strong real-world performance. Really impressive engineering.

by alealvarezarg

"Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead."

It just baffles me that this even works at all. Doesn't it just become a token soup? Is attention that precise that a second token can tell its the second token just because it learns to accumulate something in the embedding space without any sort of inductive bias?

by Ilaurens

Sabastian Raschka is one of the great LLM researchers/authors. I highly recommend his substack

by thatsgcasey

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • Interesting that they went NoPE everywhere — everyone else hedges with RoPE in the local layers. Feels like the linear-attention stuff (Kimi Delta) is quietly doing the positional work so they can get away with it. Curious to see if it holds up at frontier scale.
    by gokohl
  • Great breakdown. After using Kimi extensively, it's fascinating to see how architectural choices like KDA and NoPE translate into such strong real-world performance. Really impressive engineering.
    by alealvarezarg
  • i like your detailed breakdown. Thanks. <3
    by souravsspace
  • "Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead."

    It just baffles me that this even works at all. Doesn't it just become a token soup? Is attention that precise that a second token can tell its the second token just because it learns to accumulate something in the embedding space without any sort of inductive bias?

    by Ilaurens
  • Sabastian Raschka is one of the great LLM researchers/authors. I highly recommend his substack
    by thatsgcasey