Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • more like 5.7 not 6
    by Readerium
  • Fascinating. This is the only benchmark I've seen so far with lack-luster results. I don't understand enough about AA's specific methodology to get the implications.
    by x3haloed
  • In the general Intelligence Index it scores exactly equal to Sol (61). In the Agentic Index it scores significantly lower than Sol (51 vs 58). In both it scores lower than Fable 5.1, Opus 5 and even Muse Spark 1.3.

    Am I missing something or is this not looking too... stellar?

    by eis
  • I don't understand how those labs are releasing models so close in performance to one another?

    Are they just scaling more? getting more data at the same rate? training against the same benchmarks? making the same breakthroughs?

    How can this be explained?

    by aogaili
  • Title: "major gains"

    First chart: from score 61 (GPT-5.6 Sol) to drumroll 61 (GPT-6 Astra)

    by NiekvdMaas
  • Well, seems like ECI [0] and the AA index is diverging quite a bit. Benchmarking LLM is tough and I think we are seeing the limitations of current benchmarks and applicability to real tasks.

    [0]: https://x.com/EpochAIResearch/status/2095602754282783108

    by ekojs
  • Are the Artificial Analysis benchmarks really worthwhile any more?

    They really don't seem to match my real-world experience, and based on the comments I see I don't think that match most other people's either.

    For example, Opus 5 was at the top for some time. My experience is that it's not noticeably better than Opus 4.8, and it definitely seems worse than Fable 5, which AA benchmarks put behind Opus 5. GPT 5.6-sol and Opus 5 seem pretty interchangeable, although Sol is noticeably better at finding problems in code, particularly edge cases.

    by bashtoni

Explore Birbla archives

GPT-6 Astra makes major gains in the Artificial Analysis Coding Agent Index · Birbla