Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • Feels like google has a different training set than the others?
    by steinvakt2
  • Qwen 3.8 Flash-Next is really missing here (because it's small enough to run it locally on a (formerly) affordable machine). This is what Qwen3.8-Flash-Next-UD-Q4_K_XL gives me for "an octopus operating a pipe organ": https://imgur.com/a/zHyHIqI Seems very similar to the output of Qwen 3.8 Max to me
    by meta-level
  • https://playcode.io/blog/macbook-svg-benchmark

    MacBook Pro 3D in SVG for me the most helpful one.

    by ianberdin
  • Would love to see Qwen3.8-27b here, since that is the model most people are running locally.
    by kennywinker
  • Anyone else surprised the generations look so remarkably similar? All of these models have “independently” generalized that the moose should roughly be standing at the same position (left) or that the giraffe should have a certain color palette.
    by outlore
  • All models are pretty good now at generating these images. Back in the day, I remember experimenting with the pelican images and most of the models couldn't align the legs with the wheels. Right now as well, GPT messed up an octopus leg by originating it through the instrument rather than the octopus itself.

    I think that intertwining two entities (living/non-living) is still challenging but overall they're pretty sound.

    by samayashar
  • Website looks very cool, Fable's octopus-organist looks very cute, but I feel like this benchmark (generate an SVG by a short and slightly ridiculous description) in general has been completely Goodharted [0].

    I think they all just added a bunch of similar tasks to their training sets, so we cannot judge true emergent capabilities of the models anymore.

    [0] https://en.wikipedia.org/wiki/Goodhart%27s_law

    by vova_hn2
  • I'd like to mention the Little Dorrit Benchmark [1] which I have been running for a couple of years now. It has a few nice features:

    1. It tests visual reasoning and structured output in a single task.

    2. It seems to sort correctly on advancing general intelligence. As a counterexample, if I'm not misremembering, artificialanalysis.ai made some changes to their benchmark recently after Astra ranked below several older models.

    3. While models have gotten significantly better in the past 2 years, the top model is still at 0.78 F1, so the test is not yet saturated. As a reference point, when I started, the top models were in the [0.1, 0.2] range.

    [1] https://dorrit.pairsys.ai/

    by svcrunch

Explore Birbla archives