Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • Cool story. If it's true the company will be bought by open AI/Anthropic and Chinese labs will discover the trick and open source it by next quarter.
    by vatsachak
  • Imagine the sheer amount of power you could save by releasing the paper.
    by monneyboi
  • this is basically the only thing pre-training teams work on in labs. compute efficiency is the metric, the assumption that scaling = intelligence is considered a given.
    by brrrrrm
  • Super awesome. Wish they would release the paper about what they did to achieve this. I remember nous released the token superposition paper which improved pretraining FLOPs some, but not 50x: https://nousresearch.com/token-superposition. Wondering if they also found some cool tokenization strategiesa
    by ismael_rr
  • I don't know enough about the specific models they're comparing against to say this definitively, but it looks to me like they're comparing their pre-trained models with others' post-trained models.

    The metric upon which their 10x claim is based (bits-per-byte) is exactly the metric which is optimized during pre-training. Post-trained models are fine-tuned to optimize other metrics, which is known to be detrimental to performance on bits-per-byte evaluations. So bits-per-byte evaluations will always make a pre-trained model look favorable in comparison to a comparable model which has also undergone post-training.

    Can someone confirm whether the models they are comparing against (DeepSeek V4, Kimi K2, and Nemotron 3 Ultra) have been post-trained?

    by ansk
  • A lot of people are betting their money on infinite growth forever of AI performance, compute usage, user base, subscription price.

    I think cost will decrease forever.

    by pvillano
  • > We match DeepSeek V4 Pro Base using ~50x fewer FLOPs – that’s around half of GPT3’s pretraining compute, or ~$0.5M on GB200.

    If this holds up that's a really big deal.

    by simonw
  • They have a history of making grandiose claims like this[1] from 2024, with no visible products or research.

    [1]: https://magic.dev/blog/100m-token-context-windows (also linked to in their blogpost)

    by woadwarrior01

Explore Birbla archives