Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- my personal experience with swe has been…suboptimal. I am not sure how much I buy these benchmarks and it has a very “just blurt it out even if it’s probably not right” style but hey it’s free.by captainregex
- Not really news, terminal bench 4 is the new metric. It's only a few points ahead in terminal bench 4 of some open weight models you can run on a 256GB system.by walrus01
- Which is pretty much a useless (i.e., saturated, contaminated) benchmark now.by samusiam
- These tests are pointless when often models get nerfed few days after release. Astra today is way dumber than just few days ago.by varispeed
- Related:
Cognition launches new SWE-2 model
- Terminal Bench 2/2.1 appears to be almost solved, so we probably shouldn't look too hard on that? Their Terminal Bench 4 score is soso.
Still quite impressive though.
by forgot-my-pw