Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • Is this architecture actually able to generalize or is it mostly based on memorization? Have you tried some basic tasks that require generalization? e.g. number addition etc?
    by cpldcpu
  • Interesting. I wonder how much could be gained from using tokenization, which makes the model work at a semantic level rather than a syntactic level? I think it’s a force multiplier, but idk if it works here.
    by K0balt
  • Seems interesting, I've been messing with a lot of continuous learning approaches lately and it's cool to see something that's built from the ground up for avoiding catastrophic forgetting. Worth a clone for sure
    by advael
  • This is the first thing I see in my life that really looks like proto-AGI, it deserves its name.
    by bananaflag
  • If you actually scroll through the transcript he links to, you will see that something that looks like it could be training is happening, but no coherent responses are coming out at any point. At least not that I saw skimming through.

    That might explain why there are no benchmarks of any kind.

    by ilaksh
  • Nobody will throw rocks, I think most people are curious/suspicious about the big players and wants more hands-on since we suspect that this all will come down in cost soon enough.
    by whizzter
  • Mini-AGI is a totally inappropriate name - it seems what this project is shooting for, but not delivering on, is being a language model with "continual learning".

    Where it seems to fail, by design, on this goal is in delivering continual learning that is more than just "memorization with LRU catastrophic forgetting".

    That said, props to the author for thinking different and actually implementing something. Maybe the project can grow into something more, or inspire different ideas, if they continue to work on it.

    by HarHarVeryFunny
  • I have not looked carefully but it seems like this is over-promising on avoiding catastrophic forgetting.

    The "trunk learning rate" is set at 0.1x the learning rate for the experts, so learning on different subjects disproportionately happens in the experts, and the trunk portion is comparatively more stable. But the population of experts can grow and shrink:

    > The pool grows when it is short of capacity and shrinks when parts of it stop being asked for.

    So:

    - doesn't the trunk then _eventually_ still undergo catastrophic forgetting, it just may take much longer?

    - and before that point, catastrophic forgetting happens in stepwise chunks whenever the expert pool shrinks?

    by abeppu

Explore Birbla archives