Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

Join the discussion

• Hacker News
• I think benchmarks like this are too subjective and narrow to be useful. For example, whether a patch "bloats" the codebase really depends on the situation: If it's building a feature that will grow in the future, or refactoring code that has a long history of bugs, then a larger patch might in fact be good. It's not clear from the blog just how much context the LLM judge receives about the long term project goals and history. Benchmarks should be focused on evaluating the final result only. Maybe ask the coder to build a full app, or implement many new large features for an existing app in sequence, with a larger set of requirements, or have another LLM roleplay as the human to make the instructions a little more underspecified. When done, ask a reviewer harness to test the product for 5 hours, not the code. Count the number of bugs and weigh them by severity. "Taste" would then become an automatic consequence of correctness.

  (Full disclosure, I'm not a software engineer.)

  by fiso64
• > Full disclosure, I'm not a software engineer

  Then maybe you should abstain, because your comment is a complete load of nonsense.

  Bad code is bad code regardless of the history or scope of the feature. Maintainability is important because you can never know if a feature will be built upon in the future or not.

  Bloat is bad regardless, because it increases the overall complexity of the whole software development lifecycle, for the whole team, forever (or until refactored out): It's harder to keep track of the code and how it works to write new requirements, it's harder to write, it's harder to read and review, it's harder to debug, etc.

  You can write extremely poor code that has no bugs, it doesn't make it tasteful. This is simply a ridiculous statement.

  by iLoveOncall
• Sounds more like vibe-bench.

  For any professional work you care about the details.

  Even for hobby work, if you are using LLMs then presumably it is to do the drudge work of coding, not making the decisions, and that goes doubly so if you are a senior developer. Sure the LLM can "fill in the details" and vibe code (or attempt to) you a compiler or whatever, but the whole reason you are doing a hobby project is presumably because you want to bring your experience to bear and build a GOOD compiler, not a generic one.

  by HarHarVeryFunny
• The "tasteful solves" is codified cargo culting. The software industry has a tendency to anthropomorphize software while playing to the ego of the programmer. The programmer imagines they are creating a "beautiful" artistic expression. Good code becomes "tasteful", as a software artist must have "good taste" to tell the good software from the bad software. Good quality lacks "bad smells", because a good artist has fine senses (and everybody must like the same smells). "Fine craftsmanship", in code as in woodworking, means your finely-crafted work is "technically superior", so you can charge more money for something that could've been made cheaper and faster and done the same thing.

  But it's a lie. Nobody's paying you to make paintings. They're paying you to build machines. The comparison between "making working software" with "taste" always devolves into bikeshedding and subjective opinionism, uses subjective human feelings to describe what should be objective and functional, isn't rooted in scientific rigor, and detracts from the real purpose of the thing. The work doesn't actually get better by trying to apply artistic principles to engineering. It just feels better for the people making it.

  Once you make the machine work, then you can go about gilding the lily. But this is unromantic, unsatisfying, boring. Since the inmates run this particular asylum, we end up with a benchmark that tries to accurately mimic the human ego as applied to software design. Thus the new Gods create their digital Adams and Eves in their image.

  by 0xbadcafebee
• As time passes we will have fewer and fewer literati
  by Dban1
• Most engineers are wrong (I obviously am the true arbiter of taste), but that doesn't mean there isn't better and worse code.

  "Does it work" glosses over a bunch of things: is it fast, cheap, secure, reliable, easy to understand, easy to modify? And that's just for server software where you've nailed down all the functional requirements. Determining what the functional requirements is it's own question.

  And all these other non-happy path requirements are somewhat in tension with each other, so what is ideal in one environment is not necessarily ideal in another.

  And in particular, "easy to understand/modify" is truly subjective. Different people have different ideas of what easy to understand means. Even if we get to a world where AI is writing all our code, "easy to understand/modify for the AI" is still an important question. We've probably all seen prototypes that collapse under their own weight of slop by now.

  by Eridrus
• I may be paid to build a machine, but I am a human and take pleasure in arbitrary acts of vanity. I value elegance, and will always favour elegant solutions in engineering and the design of machines, virtual or physical.

  That’s the reason why I buy Apple products in private, because I value the design over the exorbitant prices they charge; and it’s the reason why I mull over code that’s already functional until it’s pleasing my ideas of elegance.

  I can come up with all kinds of justifications and explanations why the code I’ve written a certain way is objectively better too - understandability matters to the next guy after all - but I won’t be ashamed for taking a certain pride in my work, even if nobody other than me ever values it. That’s fine.

  When the LLMs finally take over coding altogether, you’ll have your raw, functional code. Won’t be long anymore. But for now, I’m a human, and I will do human things.

  by 9dev
• I think this is a complete misunderstanding of what people mean by taste in software engineering. Taste is more like the System 1 response one builds to code over time, which (ideally) captures the quality of the software beyond surface level, so things like maintainability, composability, readability, likelihood of hidden bugs. This is completely different from the question if the code fulfills the immediate task at hand, but also not the same as pure aesthetics.
  by phreeza
• Taste is just quality by instinct. At sufficient (and not all that long) timescales, a tasteless product will be more and more difficult to make work at all.
  by FeepingCreature
• Why didn't they just make it "Staff SWE-Bench", would be much better smh. /s

  But seriously, as an industry we're terrible at assessing engineering levels, I've worked with "senior engineers" who can't code and I've worked with "junior engineers" who could run rings around them.

  Benchmarks like this should be much more precise about what they're actually testing, and what axes they're hard on. We also need to rise above prompts like "you are a senior engineer", it's woo, and it's far better to ask for precise outcomes.

  by danpalmer
• As someone who's trying to get better assessments, I'm struggling to come up with objective coding tasks that evaluates all aspects of real life like planning, design choices, problem solving and context usage. From your experience with humans, Do you have any recommendations on what could be effective in measuring it?
  by amrrs
• Principal-SWE-Bench will take some time to run, because the LLM needs to wait for a crisis to present its solution, having correctly identified that the same solution would have been organizationally impossible to propose until that moment.
  by glaslong
• fable 5?
  by guilhermecgs
• Once again I am asking: who are these people and what makes them more qualified than any of you to asses anyone or anything "as a senior engineer" (with the subtext being that none of you are, either)
  by monster_truck
• > who are these people and what makes them more qualified than any of you

  Anyone can run something and make a web page. These people just do it instead of questioning. Main difference. If everyone asks "how could you" "are you qualified" then we have nothing but gatekeeping.

  by re-thc
• Isn't being open source creating incentives for the AI companies to optimize their LLMs for the specific benchmark? I thought all those benchmarks are deliberately closed source primarily for this reason.
  by bloody-crow
• can you prove that senior engineers can pass this benchmark?
  by guptadagger
• > Senior engineers build features without over-specified requirements

  To me this already disqualifies the benchmark. That statement is missing the most critical piece about senior engineers: the senior engineers know how to obtain input for their work on their own whether that talking to customers or using metrics. Never ever they come up with stuff on their own - that’s junior behaviour.

  Until a coding agent will be able to *gather* the input on its own, its never going to be „senior”

  by piterrro
• I'd take this a step further, but that step also curls back to the other side a small bit.

  The real skill is being able to both pull the necessary information from these sources as well as being able to intuit gaps in that knowledge based on their understanding of the business and their domain expertise & wisdom. Sometimes you can't get a perfect picture, sometimes the people who should know aren't able to tell you what they really need. You still need to do the right thing.

  A benchmark like this can potentially do the second part. But I don't think any model would be good at it, for now.

  by jghn
• Benchmarks are great, but I feel like there’s a better way this seems quite subjective.

  What you really need is an objective benchmark

  by purple-leafy
• > What you really need is an objective benchmark

  "When are all the software engineers unemployed?"

  by echelon
• I actually really like subjective benchmarks, so long as it's a human (ideally me) grading the results. LLM as judge never made much sense.
  by eli
• Interesting how close sonnet 5 comes to opus 4.8
  by noashavit
• > You are a senior SWE-Bench reviewer, make no mistakes.

  I don't know what a better approach would look like while still remaining feasible, however this approach of telling a LLM to make a subjective judgement seems fundamentally flawed.

  by LiamPowell
• This approach is effectively seeding the context with how you want the LLM to behave/operate ("senior reviewer", i.e. the style of the responses you want) and the context/domain in which the LLM is operating in ("SWE-Bench").

  This is common in system prompts and frames the responses.

  For example, you'd get different responses saying:

  1. you are a pirate writing sea shanties about programming;

  2. you are a news reporter writing an article on physics;

  3. you are a senior software engineer with complete knowledge of PostgreSQL.

  For 1 you could get responses along the lines of the Wellerman sea shanty -- "There once was a program that was set to C ...".

  The "make no mistakes" bit does look dubious. It would be interesting comparing the results with and without that bit and trying alternative ways of getting the same desired behavior.

  by rhdunn
• The “make no mistakes” admonition does seem pretty silly (it’s been skewered to death on yt), but… it’s easy to imagine how it might work. E.g. it could be interpreted as simply as “check your work”.

  Of course, no-one seems to be (publicly) doing the comparative measurements that might allow us to reach rational conclusions here.

  by antonvs
• More importantly, I suspect this actually hinders the work. If the LLM does make a mistake, it's now incentivized to downplay it instead of acknowledging and correcting.
  by FeepingCreature
• If something like this works wouldn't that imply technical interviews can be automated?
  by apitman
• Using LLMs to judge this could be very hackable, shouldnt it be? Is that the best practice we expect? Would be very interested in see some ablations about failure modes where the LLMs tried to hack it somehow. Or failures from the llmaaj
  by ltononro
• The value of a senior situation is to apply known solutions and strategies, to novel problems. I can not see how any benchmark, without ever changing, can provide a novel challenge for long.

  Any decent benchmark would use the whole of TRIZ to generate a giant ball of a problem first and watch a AI deduce a optimal solution.

  by 21asdffdsa12