Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • One thing i am hoping can make easier understanding how architectural changes trasnlate into actual implementation and performance
    by appyjain21
  • Really cool. I know the sauce is in the layers and weights, but its kind of mind blowing how short / simple the implementations are
    by gfrangakis
  • This is really cool, great way to reinforce our understanding of model architectures! But how is the author confirming that these model architecture implementations are correct though? I don't see any details in the README.md.
    by swimwiththebeat
  • This is really cool!

    Had a question on the MoE: in kimmi-K2/model.py, the router does torch.topk(..., k=self.num_experts) while every other MoE uses k=self.top_k. The ctor's top_k=8 is never stored either.. Is that intentional dense routing or should that be self.top_k?

    by tweedler290
  • This reminded me of lucidrains, perhaps one of the most prolific PyTorch users and paper implementers on the Internet. His implementations were crucial references to my own ability to boostrap my paper-implementing skills. Looks like he is still around and doing really awesome stuff. https://github.com/lucidrains
    by dwrodri
  • Hey anuj

    This is excellent for understanding. I'm having some trouble to get into understanding - pytorch is for me the RL which is used as gym/training. There I can chose ppo, dnq and other agents to perform some predefined actions in a predefined gym/world.

    The repo you are showing - I really have problems to get it into RL understanding of mine. What's the gym? What are the agents. Can it be used to train that models with pytorch?

    Sorry for the noob question. Papers are overwhelming my noob brain.

    by theGeatZhopa
  • I was yesterday years old when I learned that those open weight models need custom code to run.

    Somehow I expected inference engines are generic LLM runtimes that can execute any weight.

    So, to get this right.

    Someone trains a model.

    They release the weights and a reference implementation of the model architecture.

    Then a provider has to host this model either by running inference via the reference implementation, an open source implementation, or build their own.

    Does this mean, providers don't just differ in quantisation and configuration, but also in inference engine implementation?

    by k__
  • I have been studying modern LLM architectures and started implementing them from scratch in PyTorch to better understand the design choices behind each model.

    OpenArch is a collection of these implementations, including Llama, Qwen, DeepSeek, Gemma, Kimi, GPT-OSS and others.

    The goal is to keep the code readable and useful as a reference when going from the paper to an actual implementation.

    Would be interested in feedback from people working on model architecture and training.

    by anuj0456

Explore Birbla archives