Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

Join the discussion

• Hacker News
• Benchmarks are great, but I feel like there’s a better way this seems quite subjective.

  What you really need is an objective benchmark

  by purple-leafy
• > What you really need is an objective benchmark

  "When are all the software engineers unemployed?"

  by echelon
• I actually really like subjective benchmarks, so long as it's a human (ideally me) grading the results. LLM as judge never made much sense.
  by eli
• Interesting how close sonnet 5 comes to opus 4.8
  by noashavit
• > You are a senior SWE-Bench reviewer, make no mistakes.

  I don't know what a better approach would look like while still remaining feasible, however this approach of telling a LLM to make a subjective judgement seems fundamentally flawed.

  by LiamPowell
• This approach is effectively seeding the context with how you want the LLM to behave/operate ("senior reviewer", i.e. the style of the responses you want) and the context/domain in which the LLM is operating in ("SWE-Bench").

  This is common in system prompts and frames the responses.

  For example, you'd get different responses saying:

  1. you are a pirate writing sea shanties about programming;

  2. you are a news reporter writing an article on physics;

  3. you are a senior software engineer with complete knowledge of PostgreSQL.

  For 1 you could get responses along the lines of the Wellerman sea shanty -- "There once was a program that was set to C ...".

  The "make no mistakes" bit does look dubious. It would be interesting comparing the results with and without that bit and trying alternative ways of getting the same desired behavior.

  by rhdunn
• The “make no mistakes” admonition does seem pretty silly (it’s been skewered to death on yt), but… it’s easy to imagine how it might work. E.g. it could be interpreted as simply as “check your work”.

  Of course, no-one seems to be (publicly) doing the comparative measurements that might allow us to reach rational conclusions here.

  by antonvs
• More importantly, I suspect this actually hinders the work. If the LLM does make a mistake, it's now incentivized to downplay it instead of acknowledging and correcting.
  by FeepingCreature
• Using LLMs to judge this could be very hackable, shouldnt it be? Is that the best practice we expect? Would be very interested in see some ablations about failure modes where the LLMs tried to hack it somehow. Or failures from the llmaaj
  by ltononro
• If something like this works wouldn't that imply technical interviews can be automated?
  by apitman
• The value of a senior situation is to apply known solutions and strategies, to novel problems. I can not see how any benchmark, without ever changing, can provide a novel challenge for long.

  Any decent benchmark would use the whole of TRIZ to generate a giant ball of a problem first and watch a AI deduce a optimal solution.

  by 21asdffdsa12
• It's nice to see a new public benchmark from Snorkel. They're doing some pretty sophisticated stuff over there.
  by facorreia
• Top solve rate is currently 24% with Opus 4.8... What's a competent human supposed to score?
  by jonathanleane
• I mean these were all solved before I assume so 100% not the same human ofc but models are expected to be good at a variety of code bases while human can specialize in one and learn. I think it's fair to compare to an individual that is used to working on a product.

  I'm more interested in how fable would do

  by jascha_eng
• presumably whatever the top model uses and then some, since the human can use the model.

  I wonder if a model could score higher if it had a human at its disposal?

  by lacunary
• This makes so much sense as to why I've always felt that Opus 4.8 was leagues ahead of GPT 5.5. It's so good at taking underspecified requirements and filling in the gaps with sensible approaches for your project
  by _345
• Similarly, it explains to me why people found Claude so amazing, while I just thought "eh."

  Tool expectations

  by hypfer
• Better for vibe coders who always under specify. But at what point does it know you are under specifying but you have properly specified and it did it over your specification?
  by m3kw9
• Man I don't know if I'm living in a crazy bubble or something but GPT 5.5 is lightyears better than Opus 4.8 for me to the point where I'm honestly wondering how you're evaluating them or what kind of work you're doing.

  There's specific tasks that Opus does better on like Frontend Dev and Design but for anything else 5.5 just laps it.

  by CSMastermind
• > It's so good at taking underspecified requirements and filling in the gaps with sensible approaches for your project.

  At a high level. It misses low level or other non-functional requirements differently so I wouldn't say Opus is just strictly better.

  It's also possible that it's just a harness problem more than model.

  by re-thc
• same observation here opus 4.8 (and i dont understand the people defending gpt 5.5 constantly) was significantly mature, it would even push back against anything off putting where as GPT 5.5 will happily agree and do what is asked but I would note that it takes several tries.

  4.8 also requires more than one prompt but its output is significantly higher quality and offers more insight

  Fable 5 is a different beast however.

  by zuzululu
• The best benchmarks are the ones you create yourself.

  Its not my experience Opus is leagues ahead or even superior, but in any case, since GPT 5.5 has Instant, Medium, High, Extra High and Pro...Should the comparison be with GPT on Pro, instead of Extra High as it seems to be the case in the table?

  by root-parent
• Why supply underspecified requirements in the first place? Both models are good at challenging assumptions/edge cases and asking questions to clarify, but seemingly only when explicitly asked (i.e. something like a "brainstorm" skill).

  I don't think either harnesses do enough to encourage the model to challenge all assumptions and ask questions, maybe because users might find it annoying. That step is basically a requirement IMO.

  I've found all of the GPT-5 models to be very nit-picky, useful for code review and mathematics (important for my work), but seemingly gets in the way of "aesthetic" code, e.g. overly defensive code to cover all edge cases, even if unlikely.

  There is seemingly also a tradeoff between flexibility vs instruction following. In my experience Opus will sometimes ignore instructions but can "fill in the blanks" more, vs GPT-5.5 follows instructions better but perhaps at the cost of rigidity.

  by nsingh2
• I wonder how they're planning for the benchmark to stay relevant over time.

  If the benchmark is to implement features that are part of an open source project, and LLMs have those changes as part of their training dataset, it seems that they could just give a verbatim or slightly modified version of the change in their training data.

  And if one updates the benchmark to only incorporate code changes that are past the models knowledge cutoff, then the benchmark is less comparable over time, since the changes in the benchmark at time T and T+1 aren't the same.

  by jfim
• How far back in time do you need to give though? I imagine it will be limited by the time to think of new problems. Still seems valuable.
  by HaloZero
• Staff SWE Bench: LLM doubts whether we should do any of this, calls the entire project into question, refuses to merge code, but is happy to delete it.
  by jeffbee
• You jest but I indeed find rejection an integral part of the job. Not plainly saying "no, get away", but backing up, requesting big picture views and trying to see if the overall organization is in need of and capable of carrying said project long-term feels like the absolute minimum that needs doing before we even begin.

  I suspect LLMs can do this just fine and probably better than us, but they do need to be trained specifically for it and I have a hard time coming up with good sources of training data for it.

  by fsddfsdfssdf
• Principal version: similar, but also says the only acceptable approach is to do it like they did it at their last company.
  by organsnyder
• I saw on Twitter that in an ML course at Tsinghua University, one of the tests asks students to write quizzes that fail the most LLM models as possible.

  What if we create a benchmark that works like this and assigns ELO scores? Models fight head-to-head by writing a question, a bug, or an incomplete implementation, which the opponent has to answer, fix, or finish.

  by magnio
• That was Fudan I think
  by eunos