CursorBench 3.1

Join the discussion

• Hacker News
• Very skeptical about the composer accuracy. I have been using it for 6 months now and it is very fast, especially compared to anthropic models, but the result it produces, especially with more difficult tasks is very shallow. It feels like it just finds the cheapest way to deliver the task.
  by Losenok
• The most interesting part is costs . Gpt 5.5 and sonnet 5 cost same amount of money as GLM 5.2 but are more capable models
  by maxdo
• I've used both Composer 2.5 and GPT 5.5 (both in Cursor and in Codex) extensively, and their claim that Composer 2.5 is anywhere close in performance to GPT 5.5 is absolutely farcical. It's faster, but it's nowhere near as good.

  And given that you can only use Composer with a Cursor monthly subscription, cost comparisons are pointless since an equivalently priced OpenAI subscription gets you just as much usage of the better model.

  by BugsJustFindMe
• Cursor’s model excels at Cursor’s benchmark; news at 11.

  The other models however are reasonably where I’d expect them to be from experience piloting all of them. Fable is outclassing everything at most things at 10x the cost, but sometimes it isn’t a choice between cheap and expensive, but expensive and possible; I’ll need to learn where that boundary is just as it was the case with other models.

  by baq
• If I understand the graph correctly;

  Fable is using less tokens to achive that same tasks compared to sonet and opus. If so that is a good thing. It feels like we for a while there was spitting out tokens to get a better result. If the model themselves are getting better without generating more tokens that feels like a real win.

  Q1: Why is number of steps relevant in this graph? What does it tell us?

  Q2: and why have they flipped the horizontal graph so that 0 is to the right and not at origo? Is that some kind of new smart thing? can't say i have seen it before

  by sisve
• I wish all these sites would show pareto frontier graphs of cost/performance. That's the main 2 things that matter (I guess you could make it 3D with a speed param as well). https://paraplouis.github.io/llm-pareto-frontier/ is the best of these graphs I've seen but it doesn't update as frequently as I'd like.
  by xyzsparetimexyz
• That site is useless though because thinking tokens (and caching) and the efficiency thereof aren't accounted for. GLM5.2 is promoted by every 50 Cent Party the PLA can muster on the internet but it falls short because of its extremely verbose thinking. Anthropic models have the same problem but starting from a much higher base of real intelligence.

  Which is exactly why every credible comparison now represents cost associated with completing a task, not arbitrary input and output token costs.

  by mrngld
• Composer 2.5 is really effective at some tasks, but doesn't do as well on higher complexity tasks from my own experience.

  With that being said, Cursor Bench is nonetheless a fantastic gauge of LLM quality and performance. The most interesting outcome of Cursor Bench is Fable 5 to GPT 5.5, its almost a perfect continuous line. As of the report, Fable to GPT might be the new standard of agentic programming/building

  I did my own rounds of tests for a lot of models.

  I made multiple rounds of comparisons for issues in my open source project using Fable 5, GPT 5.5, Opus 4.8, and Composer 2.5

  Made them work on various complexity issues

  Here are the reports, i recommend using a Sonnet 5 or better model to summarize them because its quite a bit of information to consume across all the various tests: https://www.richkuo.com/#llm-battles

  My takeaway from Composer 2.5 is that its better to use Opus or GPT for planning, and then Composer to build, use Opus/GPT to pr review, and have Composer fix findings, and loop that process. Incredibly token efficient, fast, and gets the same quality as if you worked the whole thing with Fable/Opus/GPT.

  by richkuo
• everytime a new benchmark appears, Chinese models are far lower than the level where they are supposed to be according to existing benchmarks. then after a while they recover :)
  by nok22kon
• The magic of distillation!
  by generalpf
• Interesting that Opus 4.7 does better than 4.8. Too bad they didn't test 4.6, too. I witnessed a man here mocked yesterday for insisting it was better than its successors!

  Although, the benchies are always tricksy ... On DeepSWE, GPT-5.5 beats Opus-4.8, by a fair margin, but on FrontierCode, the situation is the other way around.

  The only benchmark you can trust is your actual workload!

  by andai
• It's hard to believe Composer 2.5 is that good. I tried to compare it with GLM 5.2 or Opus 4.6 and it lacked thinking about the problem and critical reasoning. It's great for executing plans made by other models, but even then it does some weird code manipulation that is far from how other files around actually work.
  by __natty__
• I read these and think it is just the jagged edge. I do not doubt your personal experience, I have used Composer 2.5 (via Grok and the credits I get with my X premium account) the past month.

  I am not building rockets, but have been quite impressed. All the models do dumb things sometimes, it has done the work I have asked it to pretty well though and has done to me some impressive work.

  It is fast on Grok, for other models I have worked extensively with I think it is better than gemini 3.1 (3.5 and antigravity for me is worse than the prior gemini cli). And is comparable to Opus 4.6. (Have not used the more recent models in Claude Code.)

  by apwheele
• I'm not using Cursor at the moment, but when I did (not too long ago) my experience was similar. Plan with Opus, implement with Composer, clean up with Opus.

  Composer did a competent but not amazing job with a good plan. What I really liked though is it was fast! Opus could take 30 minutes to do something Composer would get done in 5-10 minutes. Of course the output wasn't perfect, but that's why I'd do a cleanup pass using Opus or Codex.

  It's all a balance though, constantly changing and completely dependent on the problem you're solving. I just remain flexible and adapt my process to what's working best in the moment.

  by bmurphy1976
• I'm pretty baffled by their choice of axes. I would have thought that the left was the cheapest, not the most expensive. I appreciate that this layout means that top right can be best, but it's still unintuitive to have this backwards cost axis IMO.

  Putting that aside, I spend all day every day implementing very, very hard things right on the edge of what agents are (barely, sometimes) capable of, and I have had to keep Opus on max for things that need 'real validation' for a while now. And that has felt like 'the only way' to get Opus to perform even close to 5.5 xhigh. I'm only using Opus at all because GPT-5.5 in the subscriptions only has a small (400k, but 258k effective) context window.

  The difference is that 5.5 xhigh is extremely fast in most practical cases, both efficiently implementing _overall_, and responding very quickly with great adaptive thinking if you ask it something that it doesn't have to think about. Opus 4.8 Max will needlessly chew on everything and can take hours to implement even simple things, so I can mostly only use it for planning/review.

  Fable is much much better at adaptive thinking / responding quickly (although probably still worse than 5.5 xhigh), and... I think folks have said enough elsewhere about its strengths and weaknesses. Sadly still not a reliable implementor for my hard tasks though (that's still GPT's domain) – it tends to leave big, dangerous holes hiding inside implementations unless babied.

  by tekacs
• > I'm pretty baffled by their choice of axes

  To put their own model out in front?

  by rc1
• You can set GPT 5.5 to 1M context mode in Cursor but it costs more after the default 272k.
  by cherryteastain
• I agree why they reverse the x axis makes this graph very hard to understand for the casual observer.
  by daft_pink
• > I'm only using Opus at all because GPT-5.5 in the subscriptions only has a small (400k, but 258k effective) context window.

  Do you find that makes a difference in your work? I've been using 5.5 high/xhigh to optimize and benchmark a C codebase, and just reading the initial code virtually fills the first context window. A session will auto-compact 5-15 times, but it seems to do okay in spite of that because the task is mainly focused on the latest window each time.

  I think for programming the strength of GPT over Opus is winning here over the context window.

  by pbowyer
• > it tends to leave big, dangerous holes hiding inside implementations unless babied

  it's fascinating that I used these same exact words to express my distaste for Composer and my preference for Opus. I suspect, the domains and problems we are trying to solve need to be shared. I wrote about it here: https://news.ycombinator.com/item?id=48766275

  Would love to reach out to discuss more, if you're ok with it, or absolutely feel free to do the same as my email's in the profile like yours!

  by subhobroto
• It’s Gartner. Top-right is where you want to be.
  by mklarmann
• >it tends to leave big, dangerous holes hiding inside implementations unless babied.

  A brainwave: perhaps GLM or DeepSeek could be integrated into the mix for the purposes of red-teaming the code. Fable has been blinded to security by design[0], and the open models are pretty decent at it.

  [0] It's not clear what the situation with GPT-5.6 will be but the blog suggests similarly over-cautious safety filters.

  Amusingly the posts for recent Opus releases brag that they successfully made it worse at security! "during its [Opus 4.7] training we experimented with efforts to differentially reduce these ["cyber"] capabilities"

  by andai
• >Putting that aside, I spend all day every day implementing very, very hard things right on the edge of what agents are (barely, sometimes) capable of

  Is a single thing in your post demonstrable, or are we just supposed to take your word for it? Because all of this stuff sounds laughably subjective.

  by budsniffer952
• I'm a bit skeptical.

  Cursor's benchmark finds that Cursor's model (Composer 2.5) is basically as good as Opus 4.8 max and GPT-5.5 xhigh, but at a fraction of the price.

  Artificial Analysis' testing shows Composer 2.5 to be pretty far behind: https://artificialanalysis.ai/agents/coding-agents. You look at the DeepSWE benchmark (which is probably the hardest to game at this point) and GPT-5.5 xhigh gets a 64, Opus 4.8 max gets 56, and Cursor 2.5 gets 16.

  I don't doubt that Cursor works well for some people. It's beating DeepSeek v4 Pro in the DeepSWE benchmark and that's a very capable model. But I'm skeptical of the claims that it's a competitor for Opus 4.8 and GPT-5.5. It just seems convenient that their model does so well on their own benchmark while third party benchmarks have it far behind. Maybe it's a really great benchmark and a better measure than third party ones - I'd love for a cheap model to do as well as the expensive ones.

  by mdasen
• I can't speak to benchmarks, but I have used Composer 2.5 extensively and it's performed quite well in my real world tasks.
  by jmcqk6
• For lighter interactive agentic coding, where you type stuff into an IDE and a minute or three later get results back for review, composer 2.5 is honestly pretty great. The results get notably worse for larger tasks though.
  by datadrivenangel
• Naturally, given it’s their benchmark they have overfitted their model somewhat to it.
  by justachillguy
• Anecdotally, I find Composer 2.5 to be useless. I do use light LLMs like Claude Haiku and some of Cursor's older free models, but Composer is negative productivity for me.
  by muzani
• Cursor sessions are pretty much what composer models are RL'd on. This bench and the training data are/should be basically the same distribution.
  by famouswaffles
• DeepSWE is slightly flawed in the sense that is uses only its own harness and that causes issues on models that are not correctly supported by it. There's huge amount of evidence that the harness plays a big role in how these models work and yet DeepSWE entirely removes that (and has probably only tested that it works fine with some favourite model of them).

  There's also issues with cost calculation (as that harness doesn't use caches) and so on as reported on their github issues.

  None of the benchmarks are perfect, but that does explain a lot of the variations between benchmarks.

  by burmanm